Note: Descriptions are shown in the official language in which they were submitted.
CA 02439737 2003-09-04
PRIORITY TO PROVISIONAL APPLICATION
2 This application claims priority to provisional application serial number
60/416,976 filed October 7, 2002.
4
TEC~INICAL FIELD
The systems and methods described herein relate to associating time
references to multimedia presentations and specifically to systems and methods
that associate unique time-code values to audio video elements and
interspersed
elements.
io
BACKGROUND
~2 Audio video (A/V) presentations are made up of discrete elements (A/V
i3 elements) that include video frames, pictures, video clips, and audio
clips. A
is grouping or sequencing of A!V elements (i.e., video and audio) provides a
;s particular scene or chapter in an A/V presentation. Examples of A/V
presentations
~s include video tape movies, television broadcast, and multiple titles in a
digital
versatile disc (DVD). DVD disc titles may include a trailer title, a
promotional
~a title, and a movie title. In certain DVD discs, there may be multiple movie
titles.
To provide the proper viewing experience, specific video and audio
2o elements are sequenced to play at a particular time. To accomplish this,
various
2~ elements are associated with time values which are in turn compared with a
time
22 reference or other linearly increasing clock value to determine when to
initiate the
23 elements. For example, audio and video elements for a first scene of a
movie
2a might be played beginning at a time 141 and audio and video elements for a
~eet~hayeswk sos.na..,us I G:IMSI-111444usIMSl-1444USpal.app.doc
CA 02439737 2003-09-04
1 second scene of the movie might be played beginning at a subsequent time
160,
2 wherein both times are specified in relation to the common time reference.
s Prior methods of providing time values for audiovisual elements include a
a scheme known as "control track." Videotape presentations make particular use
of
s the control track scheme. The control track scheme attaches nondescript
pulses to
s each frame of a videotape. As the videotape is played, the pulses trigger a
counter,
which in turn generates a monotonically increasing time value corresponding to
s the currently playing point in the presentation. This time value is compared
to the
s time reference to govern playback speed, location, and synchronization. A
shortcoming in the control track scheme is that different time values are
generated
1 i depending on the point in the presentation at which the counter is
initiated-the
1z point at which playback begins. This can be problematic when searching for
and
t3 accurately identifying particular frames, since there is no unique value
that can be
is guaranteed to be consistently associated with any particular frame.
15 "Time-code" schemes are alternatives to the control track scheme. A time-
isll code scheme associates a mufti-bit time-code number to each A/V element
or
scene. A time-code number can be structured as "HH:MM:SS:FF" where "HH"
represents hours, "MM" represents minutes, "SS" represents seconds, and "FF"
1 s represents frames within each second. Corresponding video and audio
elements
zo have the same time-code number, so that the particular video and
corresponding
audio begin at the same time. Again, the time-code values are compared with
the
22 time reference to determine when the elements are to be initiated.
Zs The Society of Motion Picture and Television Engineers (SMPTE), a
z4 professional organization chartered to developed standards for film and
video,
2s standardized the use of time-code. Evolving SMTPE time-code standards
include
lee~lhayesp~e sosam.sxs' 2 G:IMSl-1I1444urIMSI-1444US.pat.app.doc
CA 02439737 2003-09-04
1 SMTPE 12M and SMTPE 328M. Since SMTPE time-code standards are widely
2 implemented in industry, it is desirable that AN presentations support or
are
backwards compatible with SMPTE time-code.
In many cases, an A/V element is made up of a computer or machine-
s readable data file. In this case, a time-code number may be placed in a
header of
s each AN element or file. Each element might correspond to a particular scene
in
a movie, and a movie therefore might comprise a number of scenes or elements.
Movies and their corresponding AlV elements are often stored and
distributed on a DVD. A DVD contains audiovisual data that is organized as
video object units (VOBUs). A VOBU is a physical unit of DVD storage, and is
» the smallest access unit of a DVD disc. A contiguous (physically sequential)
~2 group of VOBUs may make up a unit known as a "cell". A cell is the smallest
13 portion of video addressable by an index. Each cell corresponds to some
portion
is of A!V content, for instance a scene. Cells may vary in temporal length
from
_fr_actions of a second to hours.
A group of cells may logically make up a video object (VOB). A VOB is
17 single file that may include multiple multiplexed audio and video streams.
A
1s collection of VOBs may make up a video object set (VOBS).
A DVD might have numerous "titles". Each title is a potentially different
zo arrangement or sequencing of cells (and VOBUs). In many cases, these titles
2i might correspond to different versions of a movie. For example, a movie
might
22 have versions with different ratings, in different languages, and so on.
When
2s viewing a DVD, the user typically selects which version is to be rendered.
In
2a many cases, different titles will share or make use of many of the same
cells or
leel~hayes w~ sosam.emss 3 G: lMS!-lll444usIMS!-1444US.pal.app.doc
CA 02439737 2003-09-04
VOBUs [from the same VOBS, Video Object Set; possibly from the same VOB
file in the set].
s Data on a DVD is physically formatted as a sequence of logical sectors
a identified by logical sector numbers (LSNs). Each sector contains a sequence
of
s 2048 bytes of data. It is conventional for cells and VOBs on the DVD to be
s specified in terms of their sectors' LSN and byte offset into a file or
sector.
Each Video Title Set on a DVD has an information or IFO file that
s describes the playback behavior and user control options for "Titles" on the
DVD.
In particular an IFO file provides pointers to cells (i.e., LSN) on the DVD.
The
1o IFO file specifically is used in the playing of titles on the DVD providing
11 navigation and presentation information.
12 Because of the DVD's organization, its content can be played in different
13 ways to result in the different DVD titles. Titles in a DVD-Video disc may
be
1a made up of up to 999 program chains (PGC). A program chain contains up to
99
is programs (PG). A PGC is a list of cells. Each PGC has a program control
block
is (PCB) that includes an ordered set of pointers to cells indicating a
sequence in
which programs and cells are to be played. Different PGCs allow different
sequences of cells. IFO files provide the indexes to resolve the cells listed
in a
is PGC to VOBUs and LSNs that allow playing of different titles with different
zo video sequences.
Fig. 1 illustrates two types of DVD titles and how they might be played.
z2 DVD titles start at a "Relative Zero Time" on a timeline 102, where
timeline 102
z3 represents a linear time reference. A DVD title that is to be played
straight
24 through is referred to as an "one sequential PGC title". A DVD title that
has
25 potential variations in its sequencing is referred to as a "multi~PGC
title", a
lee~hayes pk sosa=.azss 4 G: IMSI-111444usIMSl-1444US.pal.app.doc
CA 02439737 2003-09-04
1 "multi~angle title", or a "parental block title". A PGC 1 105 is shown that
provides for a title to be played straight through or provides for linear
play. Non-
s linear play exists when a "multi~PGC title", "mufti angle title", or a
a "parental block title" exists. A particular title may start with a PGC 2 110
then
s branches off to either a PGC 3 115 or a PGC 4 120. PGC 3 115 branches to a
s PGC 5 125. PGC 4 120 branches to a PGC 6 130. PGC 5 125 branches to PGC 7
135. PGC 6 130 may branch to either a PGC 8 140 or a PGC 9 145. Finally, all
s titles in this example end by playing PGC 10 146 in locations 150, 152, or
154.
s The version of the title that branches to PGC 7 135 might be a NC-17 rated
1 o version of the movie. The version of the title that branches to PGC 8 140
might be
> > an R rated version of the movie. The version of the title that branches to
PGC 9
12 145 might be a G rated version of the movie.
i3 Different variations (i.e. titles having "mufti PGC title" or "parental
block
is title") will have scenes that start at different times. In other words
different
1s variations will have corresponding scenes that may start at different times
on
timeline 102, which is the duration of time from the start of PGC 2 110. This
is
because titles defined by "mufti PGC title" may have different videos for the
same
scene, and/or different languages for the same video. Based only on a time-
code
~s number, there is no way to differentiate which video and/or which audio of
the
Zo different variations to play for a particular scene. In this example, PGC
10 146
could begin playing at three different times on timeline 102, depending on the
title
22 playing and the path taken to get there.
Zs Time-code numbers may also apply to broadcasts or streaming of AN
za presentations. Fig. 2 illustrates a prior art streaming of scenes
interspersed with
25 elements, such as a hypertext markup language (HTML) document that
represents
IeeQlhayeso.a ws.axau' 5 GrIMSI-111444us1MS!-1444US.pat.app.doc
CA 02439737 2003-09-04
a commercial. An element is distinguished from a scene that is composed of
audio
z and video components. The scenes and elements may come from different
sources
or broadcast points. A user receives a broadcast 200 that includes a streams)
of
a scenes and elements. The scenes may be made up of a number of discrete A/V
components that include audio and video clips. In this example broadcast 200
s broadcasts in the following order: scene 205, element 210, scene 21 S, scene
220,
element 225 and scene 230. Each A/V component (clip) of each scene is
identified by a particular linear time-code (LTC) number. The particular LTC
s number is placed in the file header of and identifies each scene. The
example
shown illustrates a LTC header 205-A that is associated with audio clip 205-B
and
> > video clip 205-C of scene 205. LTC header 215-A is associated with audio
clip
1z 215-B and video clip 215-C of scene 215. LTC header 220-A is associated
with
audio clip 220-B and video clip 220-C of scene 220. LTC header 230-A is
is associated to audio clip 230-B and video clip 230-C of scene 230.
15 Elements 210 and 225 may be computer (machine) readable files such as
1s files written in HTML or a similar language. In this example elements 210
and
225 do not have particular LTC numbers associated with them. Broadcast and
is received streams of scenes have LTC numbers describing when the scenes are
to
~s be played; however, elements without LTC numbers do not have information to
zo indicate when they are to be played. This is particularly problematic when
scenes
2~ and interspersed elements are received from different sources and there is
no way
z2 to determine their sequence of play.
23 An evolving area is web based multimedia playback that incorporates DVD
2a video from a local player, interspersed with elements from a remote web
site.
zs Such elements may include scene updates, scene changes (e.g., new plot
twists in
lee~hayesure soo-nzvaxss 6 G:IMS7-I1I444usIMSl-l444USpat.app.doc
CA 02439737 2003-09-04
1 a movie), different language (audio) tracks, overlays of text and graphics
(e.g.,
z subtitles), and/or promotional content. Since a DVD relies on an IFO file,
new or
s interspersed web based elements without time-code can not be sequenced with
a existing A/V elements or randomly accessed based on the existing IFO file.
s S LrMMARY
The systems and methods described herein include identifying multimedia
s elements that have linear time-code numbers associated with them and adding
s prefix and suffix values to the linear time-code numbers to create unique
extended
~o time-code numbers that differentiate the particular multimedia elements.
i i In certain embodiments linear time-code numbers with prefix and suffix
1z values are added to multimedia elements without a linear time-code.
13 In particular embodiments, a time map table may be used to correlate
is extended time-code numbers with particular multimedia elements.
1 s BRIEF DESCRIPTION OF THE DRAWINGS
1~ Fig. 1 is a block diagram illustrating titles defined by a linear play
"one sequential PGC title" and a non-linear play "mufti PGC tile".
7s Fig. 2 is a block diagram illustrating prior art streaming of scenes and
zo interspersed elements from a broadcast point or broadcast points.
zi Fig. 3A is a block diagram illustrating receiving a stream of interspersed
zz elements from one source and a stream of primary elements from another
source at
z3 a player.
za Fig. 3B is a block diagram illustrating an extended time-code header that
is
z5 part of interspersed and primary elements.
lee~hayesmk sosaz..szss 7 G.'IMSI-II1444ut'IMSI-1444US.paf.app.doc
CA 02439737 2003-09-04
1 Fig. 4 is a block diagram illustrating an IFX file that includes an IFO file
z and an appended time map table, and interrelationship to a DVD.
Fig. 5 is a block diagram illustrating a system of multimedia devices and
a broadcasters to provide multimedia presentations.
s Fig. 6 is a block diagram illustrating an architecture of a multimedia
device.
Fig. 7 is a block diagram illustrating an architecture of a multimedia player.
Fig. 8 is a flowchart illustrating a process to create or add extended time
code values to A/V elements and/or elements.
s Fig. 9 is a flowchart illustrating a process which searches for an element
1o based on an extended time-code number.
11
1z DETAILED DESCRIPTION
13 Unigue Time-code Identifier Number
14 The techniques described utilize prefixes and suffixes in combination with
1s lir_ear time-code numbers in order to provide a unique identification of
any
1s particular multimedia element. In some embodiments, the prefixes and
suffixes
1~ are appended to the SMPTE time-code structure "HH.MM.SS.FF" in order to
1s preserve backwards compatibility with relevant precursor standards, such as
1s SMPTE 12M and 328M. The SMPTE time-code structure HH:MM:SS:FF" is
zo defined as follows: "HH" represents hours, "MM" represents minutes, "SS"
z1 represents seconds, and "FF" represents frames within each second.
zz Fig. 3A shows receiving interspersed elements and primary elements.
zs Interspersed elements 300(1), 300(2), ..., 300(N) include for each
particular
2a interspersed element 300 a header 305(1), 305(2), ..., 305(N), and actual
content
is 310(1), 310(2), ..., 310(N). Interspersed elements 300 are received at a
player
iee~nayes wk so.a,..szs' 8 G: IMSI-111444uslMSl-1444US.pat.app.doc
CA 02439737 2003-09-04
1 315. Interspersed elements 300 may be received from a source such as an
Internet
z source (e.g., a website).
3 Player 315 further receives primary elements 320(1), 320(2), ..., 320(N)
a that includes for each particular primary element 320 a header 325(1),
325(2), ...,
s 325(N), and actual content 330(1), 330(2), ..., 330(N). Primary elements 320
may
s be received from a separate source, such as an Internet source (e.g., a
second
website). When actual contents 310 and 330, are authored, the relative timings
of
s actual contents 310 and 320 relative to one another may not be known. By
s attaching a time-code or timestamps to headers 305 and 325, each of the
1o interspersed elements 300 may be properly sequenced amongst each of the
11 primary elements 320.
i2 For example, the player 315 may be playing primary elements 320 from a
i3 DVD disc, while interspersed elements 300 are received from (i.e., streamed
from)
~a a website where the interspersed elements 300 may be director's comments
related
to particular primary elements 320. With a time-code attached to each
1s interspersed element 300, a proper sequencing is performed with the primary
i~ elements 320. In other words, the director's comments of interspersed
elements
300 match particular primary elements 320, which may be scenes of a movie.
1s Other examples include properly sequencing promotional elements such as
Zo commercials as interspersed elements 300 with primary elements 320 which
may
2i be a movie. This allows commercials to be appropriately placed between
scenes
z2 of a movie. Presentation of "interspersed elements" may alternate with the
2s primary audiovisual elements, or be simultaneous (e.g., foreign language
dialog
2a synchronized to actors' lips, and graphical or text subtitles presented
z5 superimposed on the video image or underneath it).
lee~lhayesauc sos~sx~ezss 9 G:IMSI-111444usIMSI-1444USpot.app.doc
CA 02439737 2003-09-04
Fig. 3B illustrates pertinent details of a header 335 associated with a
particular piece of content 300 and 320. The header 335 comprises an LTC
number 340, a prefix 345, and suffixes 350, 355, and 360. LTC number field 340
a specifies a presentation time in the format "HH.MM.SS.FF", to indicate when
content 300 and 320 should begin. LTC number field 340 may be a pre-existing
s SMPTE number that has previously been associated with a particular A/V
element, or may be a derived value that is attached to a multimedia element
that
s previously did not have an SMPTE number. Sequencing of interspersed elements
s with corresponding audio/video (A/V) elements may be performed as a
production
1o process by when the audio and video elements are created or edited for a
particular
> > A/V presentation. Alternatively, corresponding time-code numbers may be
~z attached to interspersed elements separately delivered (i.e. combining web
v3 delivered content with audiovisual content fram a local DVD-Video disc),
and the
1a separately delivered elements linked to existing A/V elements with mutually
,5 consistent time-code numbers. This process is referred to as "late-bound"
or "late-
is binding" synchronization. Time-code itself may be "late-bound" to existing
» audiovisual content by downloading a time-code index that references
audiovisual
content that was previously without time-code (e.g.., a time map table that
1s synchronizes time-code to audiovisual content using an LSN index).
zo Prefix 345 precedes LTC number field 340 and comprises an epoch or title
z~ field. Title field 345 may describe and differentiate particular titles in
a DVD
2z disc, or may describe or differentiate between a broadcast program or
streamed
zs movie and an interspersed element such as a commercial. For example, on a
za particular DVD disc, there may be a title that contains a special video
feature
25 about how the movie was made, and a title that contains the movie. It may
be
IeeQ~hayes wk sosaz~axss 10 G:IMS!-l1I444us1MSl-I444USpat.app.doc
CA 02439737 2003-09-04
1 desirable to sequence particular multimedia elements from the special
feature with
2 scenes from the movie title. In accordance with the scheme described herein,
a
particular special feature video segment will be sequenced with a particular
movie
a scene, and may have the same time-code value in its respective LTC number
field
s 340. The prefix title field 345 is used to differentiate between the comment
and
s movie scene. For example, the respective headers might contain "making the
7 movie.00:53:33:13" and "movie.00:53:33:13" where a LTC value of
"00:53:33:13" is used to synchronize a particular comment with a particular
movie
s scene.
~o Prefix title field 345 itself may be formatted in accordance with one of
several standards including International Standard Audiovisual Number (ISAN).
12 ISAN is intended to provide a global database that uniquely labels each
video
13 work, such as a movie or television show. The ISAN numbering convention
is defines an identifier of 16 hexadecimal digits divided into two segments: a
12-
~s digit root segment followed by a 4-digit segment for the identification of
episodes
is or parts when applicable.
17 As discussed there may be different variations that include different
is combinations of video and audio describing particular scenes. Variations
include,
19 but are not limited to, language, camera angle, and rating (i.e., parental
block).
2o Header 335 contains three suffix value fields: language fteld 350, angle
field 355,
21 and parental block (rating) field 360. Language field 350 differentiates a
2z particular title by the language used in audio, video, and subpictures
(e.g, Spanish,
zs English, and French). Angle field 355 differentiates further differentiates
a title by
2a camera perspective (e.g., front, back, first person). Parental block field
360
25 differentiates a particular title by rating (e.g., "G", "R", "PG").
IeE'~hayesWC sosaz~etss 11 G:IMSI-III444us1MSl-1444US.pat.app.doc
CA 02439737 2003-09-04
Different A/V elements (i.e. different scenes that are played for different
variations) may make use of the same value for LTC number 340 (i.e., different
scenes played at the same time or sequence); however, they are differentiated
by
particular values for language field 350, angle field 355, and parental block
field
360. For example, if a particular combination of prefixes and suffixes allow,
there
s may be a scene described by "movie.1:23:02.15.span.front.NC-17" where a
movie
scene starts at a relative time value of 1 hour, 23 minutes, 2 seconds, and 15
s frames, content in Spanish, front view angle, and having an NC-17 rating.
s Another element may have the same time value of 1 hour, 23 minutes, 2
seconds,
io and 15 frames; however, this element may be an interview with the director.
The
i 1 particular element may be identified as "interview. )
:23:02.15.english.null.null"
1z Exemplary applications making use of the extended time-code naming
convention include playing a particular movie based on a selection of
particular
is suffix values; skipping over particular elements that are defined by a
,s "commercial" prefix title field 345; and searching for a particular scene
based on a
particular suffix(s) played at a particular LTC number.
m An extended time-code number such as contained in header 335 may be
described as "TT.HH.MM.SS:FF.LL.AA.PB", where "TT" is title, "LL" is
language, "AA" is angle, and "PP" is parental block.
2o Multimedia elements are described by particular file formats. For example,
21 an audio file may be described by "wave" audio file format as defined by
the
22 Microsoft Corporation. A wave audio file is defined by a ".wav" extension.
An
z3 example of a particular video file format is the audio/video interleaved
format or
AVI. An AVI file is defined by an ".avi" extension. It is contemplated that
lee~hayeswk soamx~ozss 12 G:IMSI-I1I444usIMS!-I444US.pal.app.doc
CA 02439737 2003-09-04
regardless of the file format, each particular A/V element may include a
header
that describes a particular extended time-code number.
In certain cases, particular fields may not apply. In other words, there may
a only be one or a limited number of variations for a particular scene, and
for other
scenes there may be a greater number of variations. A "NULL" value may be set
s for fields that do not apply. Different delimiters may be used (e.g.,
period,
comma, colon, semicolon, slash, etc.), and delimiters may be omitted in the
case
s where leading or trailing nulls can be represented without ambiguity.
s Information Extension (IFX) file
~o A DVD disc can be described as a sequence of data structures referenced by
11 logical sector numbers (LSN) and byte offsets which are identified in index
data
1z structures. Index data structures occupy a defined order of sequential and
,3 contiguous LSNs. Video objects (VOB) are arranged sequentially and
is contiguously in order to provide known locations by LSN for access by
program
1s chains (PGC) and other video index data structures. An information (IFO)
file on
1s a DVD contains the index data structures and provides pointers to physical
disc
» locations (i.e. LSN) which contain the video referenced by a particular PGC.
For
i8 a title that is mufti PGC title, multiple PGCs in the IFO file are linked
by title
is information, selected by user actions, or selected by programming commands,
zo resulting in playback variations of the title that is selected.
z, Data structures in the IFO file include Video Manager Information (VMGI)
zz that provides information as to arrangement of the disc; Video Title Set
23 Information (VTSI) that provides information as to navigation and
presentation of
za a particular title set on the disc; and Video Object Sets (YOBS) that
contain a
z5 grouping of video cells in VOBs. One of the functions of the IFO file is to
~ee6~nayeswk so..m..~~ I3 G:IMSl-111444us1MS1.1444USpal.app.doc
CA 02439737 2003-09-04
i describe the location of VOBUs, cells, and VOBs of particular titles by
providing
the appropriate pointers to particular LSNs that contain the data.
Fig. 4 shows an IFX file 400 that includes an IFO file 405, a time map
(TMAP) data structure or TMAP 410, and an extended TMAP data structure or
s extended TMAP 415.
s TMAP 410 associates LTC values to VOBUs in its Video Title Set. In this
example, TMAP 410 is made up of entry 1 415(1), entry 2 415(2) to entry N
s 415(N). Each entry of entries 415 has a linear time code value 420 and a
pointer
s 425. Pointers 425 of entries 41~ point to a VOBU table 430 and in particular
to
1o entries 435(1) to 435(N) that provide video content and are defined by VOBU
440
> > and LSN 442. TMAP 410 defines defines one sequential PGC titles. VOBU
12 table 430 may be included in IFO file 405 and links each VOBUs to storage
locations on the disc expressed and indexed by the first LSN 442 of each VOBU
i4 440.
15 I Typically IFO file 405 includes only TMAP 410. Extended TMAP 415 is
appended to TMAP 410 and IFO file 405 to create IFX file 400. Extended TMAP
415 provides an implementation of extended time-code format that adds extended
i8 time-code for mufti-PGC titles while retaining normal DVD-Video IFO file
19 functionality on legacy playback devices that only read the IFO file 405.
In this
2o example, the extended TMAP 415 is shown providing pointers 445 to some
entries
z7 435 of VOBU table 430. In certain cases the same entries of entries 435 as
z2 referenced by TMAP 410 are pointed to; however extended TMAP 41 S may not
Zs necessarily reference (point to) all or the same entries 435 referenced by
TMAP
2a 410.
Zs
lee~lhayesqk sosameZSe 14 G:IMSI-I11444usIMSI-I444US.pat.app.doc
CA 02439737 2003-09-04
1 Extended TMAP 415 is used by players that are equipped to read extended
z time-code numbers. In particular, extended TMAP 41 S associates VOBUs with
extended time-code numbers. Legacy players that are not equipped to read
a extended time-code numbers, use IFO file 425 (TMAP 410) and ignore extended
s TMAP 415.
s Extended TMAP 415 includes entry 1 450(1), entry 2 450(2), ..., entry N
450(N). Entries 450 associate particular elements with particular linear time-
code
s numbers as provided by column linear time-code 455. Each of the entries 450
are
s further identified by an element column 460. Entries 450 may be
differentiated
~o based on a prefix or title value provided by title (prefix) 465 describing
the
applicable DVD title that the particular entry supports. Further a suffix 470
~z differentiates entries based on suffix values such as "language, camera
angle, and
i3 parental block". Title (prefix) 465, linear time-code 455, and suffix 470
make up
1a the extended time-code structure. For each entry 450 a pointer is provided
by
~s pointer 445. Pointers of entries 450 point to particular VOBUs in table
430.
Physical access to the video data corresponding to the extended time-code is
completed by accessing the LSN 442 associated with the VOBU 440 in table 430
is and its linked extended time-code in table 415.
1s IFX file 405 is structured so that IFO 405 (TMAP 410) is made available to
zo legacy players that are not capable of reading extended time-code numbers.
21 Extended TMAP 415 is appended to IFO file 405 to provide extended time-code
zz numbers, providing additional functionality. The extended time-code numbers
za provide functionality to present and search for (i.e., navigate) DVD
content based
za on extended time-code numbers. Extended time-code numbers allow DVD titles
25 to be particularly identified by the title prefix of the structure of the
extended time-
~ee~hayes wk somz.ezss I5 G: IMSI-III444usIMSl-1444US.pat.app.doc
CA 02439737 2003-09-04
1 code number, and refer to elements of particular titles by the linear time-
code that
z is part of the extended time-code number. Further, additional elements may
be
s included (i.e., interspersed with) with DVD content of a particular title
described
a by extended time-code, where the additional elements include extended time-
code
numbers that identify where they are to be played in relation to elements of a
s particular title.
Exemnlary System
s Fig. 5 shows a system 500 of multimedia devices and broadcasters to
s provide multimedia presentations. A network 505 includes local area, wide
area,
1o private, and public/private networks, including the Internet. Hardwired
(e.g.,
1i cable) and/or wireless networks may make up network 505. Network SOS is
12 designed to provide multimedia communications from and to various entities.
i 3 A television broadcast point S 10 provides "multimedia presentation
i4 content" in system 500. A television broadcast point 510 plays multimedia
elements from an A/V source 515. A number of multimedia elements such as A/V
1s elements make up a total A/V presentation from A/V source 515. A/V elements
may be associated with a particular time-code number. Television broadcast
point
is 510 may intersperse A/V elements from A/V source 515 with elements such as
is commercials or promotions (i.e., interspersed elements) from an added
content
2o source 520. The interspersed elements may not have time-code numbers
2~ associated with them. Collectively the A!V elements and the added elements
are
22 referred to as multimedia presentation content or MPC. A/V source 515 and
zs added content source 520 may stream at different times, therefore may be
received
2a at different times at various multimedia devices. An extended time-code
number
lee~hayescue sosomans 16 G:VMSI-J11444usIMSI-1444US.pat.app.doc
CA 02439737 2003-09-04
i as defined in Fig. 3B may be appended to each AlV element and interspersed
z element as they are streamed from television broadcast point 510.
Television broadcast point 510 is connected to a satellite dish 525 which
a uplinks to a satellite 530. Satellite dish 525 transmits signals carrying
MPC to
s satellite 530. A satellite dish 535 downlinks the signals carrying MPC from
s satellite 530. Network 505 is directly connected to satellite dish 535 to
receive the
MPC. Alternatively network 505 may be directly connected to television
8 broadcast point 510. Television broadcast point 510 may also transmit radio
s frequency (RF) signals carrying the MPC through an antenna 540. Such RF
yo signals may be received by various multimedia devices that include
televisions.
i ~ A multimedia device 545 is a standalone unit not coupled to network 505,
~z but is able receive the MPC from television broadcast point 510. Multimedia
is device 545 includes an A/V display unit 545-A which may be a television set
or
1a monitor with a tuner and A!V inputs and outputs. Multimedia device 545 may
1s further include a multimedia player 545-B such as a DVD player, and a
s multimedia recorder 545-C. Multimedia device 545 is connected to an antenna
550 that receives RF signals from sources such as antenna 540, where the RF
~a signals carry MPC. MPC may further be received by multimedia device 545
1s through a satellite dish 555, where satellite dish 555 receives signals
from satellite
zo 530.
zi In certain cases, multimedia device 545 receives A/V elements and
zz interspersed elements without any associated time-code number. An extended
z3 time-code number may be attached to each A/V element and interspersed
element
za as they are received at multimedia device 545. The A/V elements and
interspersed
leefs~hayes wa soa.m..szss 17 G:IMSI-lll444ttsIMSl-1444US.pat.app.doc
CA 02439737 2003-09-04
1 elements may be recorded by multimedia recorder 545-C, and accessed by
z reference to unique extended time-code numbers.
Broadcast of MPC may also be performed through the Internet. Typically,
a this is known as a "web broadcast" and provides digital transmission of MPC.
A
s web broadcast point 560 may include an interactive web site accessible
through
s network 505. Web broadcast 560 receives A/V presentation from an AIV source
565 with interspersed elements that can include text, graphics, audio/video
clips,
s and/or HTML or SMIL files from an added content source 570. Alternatively
s such elements may come from other sources.
io Web broadcast point 560 delivers MPC (i.e., A/V elements and interspersed
1 ~ elements) to network 505 for access by various multimedia devices
connected to
1z network 505. Alternatively, web broadcast point 560 may deliver MPC through
a
1s satellite as described above.
1a A multimedia device 575 is coupled to network 505 and receives MPC
is from various sources including television broadcast point 510 and web
broadcast
~s point 560. Multimedia device 575 includes an A/V display unit 575-A which
may
i~ be a television set or monitor with a tuner and A/V inputs and outputs.
is Multimedia device 575 may further include a multimedia player 575-B such as
a
~s DVD player, and a multimedia recorder 575-C.
zo In certain cases, multimedia device 575, similar to multimedia device 545,
z~ may receive A/V elements and interspersed elements without any time-code
zz number associated. An extended time-code number is attached to each A/V
z3 element and interspersed element as they are received at multimedia device
575.
za The AlV elements and interspersed elements may be recorded by multimedia
zs recorder 575-C, and accessed by reference to unique extended time-code
numbers.
~eephayes wk sosaa'sus 1 8 G:IMSI-III444usIMS!-I444USpa(.app.doc
CA 02439737 2003-09-04
1 In other cases, multimedia devices 545 and 575 receive A/V elements and/or
z interspersed elements that have extended time-code numbers. Sequencing of
and
3 playing the presentation at multimedia devices 545 and 575 are based on the
a extended time-code numbers.
s Multimedia devices and standalone units include WebTV units, and
s personal video recorder (PVR) units. In this example a PC 580 is coupled to
network 505 and receives MPC. Like multimedia devices 545 and 575, PC 580
a may attach extended time-code numbers to A/V elements and interspersed
s elements that are received without extended time-code numbers.
1o Various sources may provide other content, including updates to files and
i1 programs, used by multimedia devices. This content may not necessarily be
1z multimedia elements or A/V elements and can include IFX files as described
in
is Fig. 4. In this example, a web site 585 illustrates a source from which a
~a multimedia device accesses additional content such as updated files.
Further,
i5 broadcast points such as television broadcast point 510 and web broadcast
point
is 560 may stream new A/V elements to be sequenced with pre-existing content
at
~ various multimedia devices.
is Exemplary Multimedia Device
is Fig. 6 shows an architecture for an exemplary multimedia device 600.
zo Multimedia devices 545, 575, and 580 of Fig. 5 are representative of the
z1 architecture of multimedia device 600. Multimedia device 600 may be a
distinct
zz self contained unit such as PC 580, or may be comprised of various units
such as
zs multimedia devices 545 and 575. As will be evident below, multimedia 600
za provides functionality to receive MPC, process MPC, and display MPC. In
certain
z5 cases, multimedia device 600 also records MPC.
tee~nayes w~ sosaz,ezss 19 G: IMSI-1I1444uslMSl-1444USpat.app.doc
CA 02439737 2003-09-04
Multimedia device 600 may include a receiver module 605 that is
z configured to receive RF signals from a television source. Other signals and
data
3 may also be received from various sources, including devices and networks,
a through an audio video input module 610.
Multimedia device 600 includes one or more processors 615 (e.g.,
s microprocessors) which process various instructions to control the operation
of
multimedia device 600 and to communicate with other electronic and computing
a devices.
When A/V elements and interspersed elements are received without time
code numbers, such numbers may be attached to each through the use of an
11 extended time-code module 620. Extended time-code module 620 is configured
to
1z generate and attach a unique extended time-code number to each A/V element
and
1s each interspersed element as they are received by multimedia device 600.
is Multimedia device 600 can be implemented with one or more memory
i5 components, examples of which include a non-volatile memory 630 (e.g., ROM,
is Flash, EPROM, EEPROM, etc.). Memory 630 is configured to store an operating
system module 635 that contains an operating system to be used on multimedia
is device 600. Application programs may be stored in an applications program
module 640 of memory 630. Random access memory (RAM) may also be
zo implemented in multimedia device 600 for temporary storage of files and
2i programs used by multimedia device 600. A RAM module 645 is illustrated.
zz MPC may be stored in a multimedia storage module 650. Multimedia
zs storage module 650 may make use of various media such as optical disks,
za magnetic tape, and magnetic disk. Multimedia storage module 650 may be
zs accessed by other modules and devices that are part of multimedia device
600, and
lee6°~shayes ~s~ sos.u.azss 20 G:IMSl-I11444uslMSl-1444US.pal.app.doc
CA 02439737 2003-09-04
~ devices external to multimedia device 600. Multimedia recorder module 655
z provides recording capability to multimedia storage 650. A multimedia player
3 module 660 may either directly receive MPC from processors) 615 or access
4 MPC from multimedia storage 650 for MPC. Processors) 615 may access AlV
elements and elements in multimedia storage module 650 by identifying
particular
s extended time-code numbers.
An audio/video output module 665 provides direct output to a display or to
s an intermediate device. Output module 665 may split audio and video
information
s and stream them to individual channels. Processors) 615 controls output
module
~0 665 and the A/V streams sent from output module 665.
1 ~ Exemplary Multimedia Player
Fig. 7 shows an architecture for an exemplary multimedia player 700.
is Multimedia player 700 includes various devices, but in particular includes
DVD
i4 players that utilize industry standards, in particular standards set by the
National
i5 Televisions Systems Committee (NTSC). The architecture of multimedia player
is 700 may be used by multimedia player 660 of Fig. 6. Multimedia player 700
may
~ read from prerecorded media such as a commercially produced DVD, or from
is media that may be written to such as recordable DVDs, or receive A/V
element
is and interspersed element streams.
2o One or more processors 705 may be used to process signals received from a
z1 medium (e.g. DVD), and to communicate with various modules of multimedia
22 player 700. In cases where multimedia player 700 is part of multimedia
device
23 700, processors) 615 illustrated in Fig. 6 may be used for processors) 705.
24
IeeNylhayes y sosamexss 21 G: IMS!-lll444uslMS!-l444US.pat.app.doc
CA 02439737 2003-09-04
i Processor(s) 705 process and sequence A/V elements appended extended
z time-code numbers. In the case of DVD, IFX files are used by processors) 705
to
sequence A/V elements.
Multimedia player 700 may include a memory module 710 which stores an
s operating system module 715 that contains an operating system to be used on
s multimedia player 700. Memory module 710 may further include a
fileslprograms
module 720 that contains applicable files and/or programs for multimedia
player
a 700.
s An AIV element or element synchronizer module 725 is used to
synchronize separate streams of audio, video, and/or other data based on
extended
time-code numbers. Synchronizer module 725 is particular used to perform late
1z binding of streams of A/V elements and/or elements that were previously not
13 synchronized. For example, multimedia player may be receiving a stream
is containing a new language track that is to be synchronized with video that
is either
15 resident at multimedia player 700 or received from another source.
Synchronizer
1s module 725 synchronizes the audio and video based on the extended time-code
» number. In particular, a linear time-code number of the extended time-code
1s number is used to synchronize the audio and video.
1s Multimedia player 700 may further include a presentation engine module
z~ 730 that uses presentation data from the media to control what is displayed
from
z1 multimedia player 300 as derived from information in an IFO or IFX file. A
zz navigation manager module 735 uses navigation data to provide an interface,
z3 create menus, and control any branching of A/V elements or scenes which are
also
za derived from an IFO or IFX file. Further, an audiolvideo output module 740
is
zs
IeeQ~hayes one sos.msezss 22 G: IMSI-111444usIMSI-1444US.pat.app.doc
CA 02439737 2003-09-04
provided in multimedia player 700 to output signals to a display device, or an
z intermediate device prior to the display device.
3 Adding extended time-code
a Fig. 8 shows a process 800 which creates or adds extended time-code
s values. Process 800 may be performed at a broadcast point such as television
s broadcast point 510 or web broadcast point 560 of Fig. 5. In certain
embodiments,
process 800 is performed at a multimedia device such as multimedia devices
545,
s 575 and 580 as shown in Fig. 5, and described by the architecture of
multimedia
s device 600 as shown in Fig. 6. In other cases, process 800 is performed when
a
1o DVD or other media is created.
At block 805, a decision is made if an AN element or interspersed element
has a linear time-code number appended to it. The linear time-code number may
13 be defined by a SMPTE standard. If the A/V element or interspersed element
has
is a linear time-code number (following the YES branch of block 805), the
linear
15 time-code number is kept. If the A/V element or interspersed element does
not
is have a linear time-code number (following the NO branch of 805), a linear
time-
code is attached at block 810. At block 810, the linear time-code that is
attached
js describes when the particular A!V element or interspersed element is to be
played.
~s At block 815, a title value is added to the linear time-code of a
particular
zo A/V element or interspersed element. An extended time-code number is
created.
zi The title value differentiates among different MPC presentations.
zz At block 820, a decision is made if applicable suffixes (i.e., suffix
fields)
23 exists that further define an A/V element or interspersed element. Such
suffixes
include angle, language, and parental block (i.e. rating). If a suffix applies
(the
zs YES branch of block 820), block 825 is performed, and a suffix value is
added to
lee~hayesyk sosazsezss 23 G:IAfSl-l1I444us1MSl-I444USpat.app.doc
CA 02439737 2003-09-04
i the extended time-code number. Block 820 is performed until all applicable
suffixes are determined. Each applicable suffix is concatenated to the
extended
time-code number, until all suffix values have been determined (the NO branch
of
d block 820). In certain cases, particular suffix fields will not apply to
certain A/V
elements or interspersed elements.
s Searching based on extended time-code
Fig. 9 shows a process 900 which searches for an element based on an
a extended time-code number. Process 900 may be performed at a multimedia
s device such as multimedia devices 54~, 575 and 580 as shown in Fig. 5, and
io described by the architecture of multimedia device 600 as shown in Fig. 6.
11 Multimedia player ?00 of Fig. 7 may also use process 900 to search for
particular
~2 AIV elements andlor interspersed elements based on title prefix and/or
suffix
13 values.
is I At block 905, the title value of the extended time-code number is
1511 determined. The title value distinguishes A/V elements and interspersed
elements
that are part of a particular MPC from other MPCs. For example, a search may
be
i7 made as to a particular movie title, or to particular commercials.
1B Several suffixes may apply in the selection of a particular A/V element
19 and/or an interspersed element. For example, it may be desirable to search
based
20 on a particular language, a particular angle, a particular parental block,
or a
2i combination of the three.
22 At block 910, a decision is made as whether a particular suffix applies.
23 Although suffixes may proceed or follow other suffixes in an extended time-
code
2a number structure, suffixes have the same hierarchy. Therefore searches may
be
25 made for any particular suffix value.
lee~hayesou~ sasaz,9zss 24 G:IMSl-111444usIMSI-1444USpar.app.doc
CA 02439737 2003-09-04
1 If the search is based on a particular suffix value (the YES branch of block
2 910), block 915 is performed. At block 915 the particular suffix value is
3 determined. If all suffix field search values have been presented (the NO
branch
4 of block 910), then block 920 is performed. At block 920, all A/V elements
and
s interspersed elements are selected based on the determined title and suffix
values.
s It may be desirable to complete process 900 at this point and provide all
A/V
elements and interspersed elements that meet the value determinations.
A further step may be taken to select a particular AN element or
s interspersed element, which can include a particular scene, based on a
linear time-
1o code number. At block 925, the searching is narrowed down to a particular
linear
11 time-code number. If further search is performed based on the linear time-
code
1z (following the YES branch of block 925), then block 930 is performed
identifying
13 the particular element. For example, process 900 may be used to perform a
search
14 for a particular element based on a particular scene of a particular
variation of a
15 particular movie title.
is Although the invention has been described in language specific to
structural
1~ features andlor methodological acts, it is to be understood that the
invention
18 defined in the appended claims is not necessarily limited to the specific
features or
1s acts described. Rather, the specific features and acts are disclosed as
exemplary
2o forms of implementing the claimed invention.
21
22
23
24
lee(~hayeso~ sosamsus 25 G:IMSI-Ii1444usIMSl-I444USpat.app.doc
