Note: Descriptions are shown in the official language in which they were submitted.
CA 02492870 2006-03-20
77787-26D(S)
-1-
NATIIRAL COLOR MATCHING IN A
VIDEO EDITING SYSTEM
This application is a divisional of Canadian
Patent Application S.N. 2,368,487 filed April 14, 2000.
BACKGROUND
Digital non-linear editing (DNLE) is a process by
which digital media may be edited. DNLE, as the name
implies, is performed on digital media stored as data in
digital media files on a digital random access medium. DNLE
may be conducted in a non-linear fashion because the digital
media files in which the digital media is stored can be
randomly accessed. Thus an editor may access a piece of the
digital media without having to proceed sequentially through
other pieces of the digital media stored in the same or
other digital media files. More than one editor also may be
able to access different pieces of the same digital media
contemporaneously. The digital media may be a digitized
version of a film or videotape or digital media produced
through live capture onto a disk of a graphics or animation
software application. Example commercial DNLE systems
include the Media Composer or the Avid SymphonyTM video
production system or NewsCutter news editing system
available from Avid Technology, Inc. For a more detailed
description of DNLE, see Digital Nonlinear Editing, New
Approaches to Editing Film and Video, Second Edition, 1998
by Thomas Ohanian.
CA 02492870 2000-04-14
77787-26D
-la-
Basic components of digital media are segments, which are described in more
detail below. For audio segments, each segment includes a plurality of
sequential audio
samples. For a video, a segment comprises at least one digital image, where
each digital
image comprises an array of picture elements called pixels. A video segment
has several
properties or attributes associated with digital images of the video segment.
The
attributes include, inter alia, image formatting attributes, color attributes,
storage location
attributes, and source attributes. Source attributes define the source data
structures from
which a segment originates. Segments and their source data structures are
described
below in more detail.
Formatting attributes may include: the color format of the pixels; the pixel
depth
of the pixels; the number of rows and columns of pixels, commonly referred to
as the
image size; the shape of the pixels, for example, square and non-square; the
type of video
signal on which the digital image is, was or will be transmitted; the video
rate at which
the signal is, was or will be transmitted; the sampling rate by which the
video signal is,
was or will be transmitted, for example, 4:2:2 or 4:4:4; and the pixel clock
rate used to
transfer the digital pixel data within a DNLE system.
CA 02492870 2000-04-14
77787-26D
-2-
Many of these formatting attributes are defined by standards. For example, in
a
standard square pixel NTSC image format, there are 640 columns and 480 rows of
pixels. Further, in both non-square pixel PAL format and non-square NTSC
format,
pixel information is transmitted at a rate of 27 MHZ.
Video signals may be classified into two major types: composite signals and
component signals. Component signals represent a pixel with multiple color
components, each component defming a value along a dimension of the color
space in
which the color being represented is defined. A composite video signal is an
analog
signal that uses a high frequency subcarrier to encode color information. The
subcarrier
is a sinewave of which the amplitude is modulated by the saturatioir of the
color represented by the signal, and the hue of the color is encoded as a
phase difference from
a color burst. Analog composite signals generally are used to broadcast
television video
signals.
The pixel depth of an image defines the number of binary digits necessary to
store the color of a pixel on a computer=readable medium. For example, a
common pixel
depth for a component color fonsnat is 24 bits, where 8 bits are used to store
each color
component.
Video and audio segments may also have modification attributes associated with
them. Modification attributes of a segment are attributes associated with
modifications
made to the segment by a system user such as an editor or a colorist. These
modifications can be applied to digital images and audio samples on a DNLE.
Video or
image modifications include, inter alia, resizing an image, keying, panning
and
scanning, blending, box wipes, dissolves, color effects, color correction, and
other
various modifications and effects known to those skilled in DNLE. Audio
modification
include, inter alia, mixing, sweetening, manipulating audio waveforms, and
other various
modifications and effects known to those skilled in the art.
A sub-class of video modification is color modification. Color modification
may
be performed both to correct color errors due to process errors and to adjust
the colors
used in the video for artistic expression. For a given image, color
modifications may be
applied to all pixels in the image or pixels of a portion of the image. Color
modifications
may include enhancing contrasts or color in an image to give a program an
overall
"look," or applying special effects to selected segments. Other color
modifications may
be made by an editor during an editing session to correct problems with color
or lighting
CA 02492870 2000-04-14
77787-26D
-~-
resulting from the source of the media. Such corrections may include color
balancing for
camera and lighting differerices, correcting for film processing differences,
matching
colors and tones from shot to shot, or adjusting video levels for differences
in source
tapes, source decks, etc.
There are a variety of component color formats used to represent color.
Component color formats are used in component video signals as well as in
digital pixel
streams. An digital image storage RGB (Red, Green, Blue) format represents a
color
with a red component, a green component and a blue component. CMY (Cyan,
Magenta, Yellow, Black) format represents a color with a cyan component, a
magenta
component, and a yellow component. CMY is a format commonly used by printeFS:--
The
CMY components are color opposites of RGB components. In a three-dimensional
coordinate system, each component of either the RGB or the CMY format
represents a
value along an axis, the combination of the values defining a cubic color
space.
'I'he color formats HSL (Hue, Saturation, Lightness or Luminance) and HSV
(Hue, Saturation, Value) represent a color with a hue component, a saturation
component, and a luma component. In a three-dimensional coordinate system, the
luma
component represents a value along a luma axis, the hue component represents
the angle
of a chroma vector with respect to the luma axis and the saturation component
represents
the magnitude of the chroma vector. The combination of the values defines a
hexagonal
cone-shaped color space around the luma axis.
YCrCb, YUV, and YIQ are three color formats that represent a color with a luma
component Y, and two chroma components, Cr and Cb, U and V, or I and Q,
respectively, that define a chroma vector. In a three-dimensional coordinate
system,
each component of either the YCrCb, YUV, and YIQ format represents a value
along an
axis, the combination of the values-defining a cylindrical color space around
the luma
axis. The chroma components define the chroma vector. In data formats with a
luma
component, the luma component can be used independently to represent a pixel
in a
black and white image to be displayed, for example, with a black and white
monitor.
A typical color modification in HSL color space may include increasing a color
component or a combination of color components for all pixels in each digital
image of a
section of digital media. Typically, an editor accesses a segment of a
composition that
represents the section of media through an editor interface and inputs desired
color
modifications through the editor interface. Some systems permit an editor to
apply color
~
CA 02492870 2000-04-14
77787-26D
4
modifications to only portions of a digital image. Portions
of a digital image can also be described as one or more
pixels. For example, an editor may select with a mouse,
keyboard, or some other editor input device a portion of the
image and define color modifications for the selected
portion. A suitable commercial system for color modification is Avid Media
l1lusionT"' available from Avid
Technology, Inc. Other commercial software applications may
be used.
SUMMARY
One problem with current techniques for modifying
digital media is that a single modification generally cannot
be specified for several segments in a composition sharing a
common attribute. An editor generally has to access every
segment sharing the common attribute to make the
modification. This process can be time-consuming and prone
to error. An editor may intend on making the same
modification to a first and second segment sharing a common
attribute, but the modifications may be inconsistent.
Further, different editors may be working on the different
segments sharing a common attribute. Although copying a
modification from one segment to the other is one solution
to this inconsistency, the other segment is not always
available, and having to copy adds more time to the'editing
process. As the number of segments to be modified
increases, time is added to the editing process.
The system and method described herein permits a
modification to be specified for several segments sharing a
common attribute. Such shared modification may be combined
with other separately specified modifications made to a
composition or part of a composition. Such modification may
CA 02492870 2006-03-20
77787-26D(S)
-5-
be audio modifications specified for audio segments or a
video modification specified for video segments.
Accordingly, in one aspect the invention provides
a computer-implemented method for performing natural color
matching, comprising: receiving an indication of components
of a selected destination color having a first luminance and
components of a source color having a second luminance;
determining a product of a ratio of the second luminance to
the first luminance and the components of the selected
destination color; and adjusting values of components of
pixels in an image according to the determined product.
In another aspect the invention provides a method
for modifying colors in an image such that a first color in
the image will naturally match another color, comprising:
determining a ratio of luminance of the first color in the
image to luminance of the other color; determining a second
color to which to change the first color in the image
according to the determined ratio and the other color; and
processing the colors in the image according to a
relationship defined by the first color and the second
color.
In yet another aspect the invention provides a
computer program product, comprising: a computer readable
medium; computer program instructions stored on the computer
readable medium that, when executed by a computer instruct
the computer to perform a method for modifying colors in an
image such that a first color in the image will naturally
match another color, comprising: determining a ratio of
luminance of the first color in the image to luminance of
the other color; determining a second color to which to
change the first color in the image according to the
determined ratio and the other color; and processing the
CA 02492870 2000-04-14
77787-26D
-6-
colors in the image according to a relationship defined by
the first color and the second color.
In another aspect the invention provides an
apparatus for modifying colors in an image such that a first
color in the image will naturally match another color,
comprising: means for determining a ratio of luminance of
the first color in the image to luminance of the other
color; means for determining a second color to which to
change the first color in the image according to the
determined ratio and the other color; and means for
processing the colors in the image according to a
relationship defined by the first color and the second
color.
In another aspect the invention provides a
computer program product, comprising: a computer readable
medium; computer program instructions stored on the computer
readable medium that, when executed by a computer instruct
the computer to perform a method for performing natural
color matching, comprising: receiving an indication of
components of a selected destination color having a first
luminance and components of a source color having a second
luminance; determining a product of a ratio of the second
luminance to the first luminance and the components of the
selected destination color; and adjusting values of
components of pixels in an image according to the determined
product.
In another aspect the invention provides an
apparatus for performing natural color matching, comprising:
means for receiving an indication of components of a
selected destination color having a first luminance and
components of a source color having a second luminance;
means for computing a product of a ratio of the second
CA 02492870 2000-04-14
77787-26D
-7-
luminance to the first luminance and the components of the
selected destination color; and means for modifying values
of components of pixels in an image according to the
determined product.
In another aspect the invention provides a
computer program product, comprising: a computer readable
medium; computer program instructions stored on the computer
readable medium that, when executed by a computer instruct
the computer to perform a method for performing natural
color matching on an image comprising pixels defined by
components, wherein each component of a pixel has a value,
comprising: receiving an indication of components of a
selected destination color having a first luminance and
components of a source color having a second luminance;
determining a ratio of the second luminance to the first
luminance; for each of the components of the selected
destination color, determining a product of the component of
the selected destination color and the determined ratio; and
for each component, defining a graph that maps input values
for the component on a first axis to output values for the
component on a second axis according to the value of the
component for the source color as an input and the
determined product for the component as an output; and
adjusting the value of each component of each pixel in the
image according to the determined mapping for the component.
BRIEF DESCRIPTION OF THE DRAWINGS
In the drawings,
FIG. 1 is a block diagram illustrating
relationships between source data structures and source
media;
CA 02492870 2000-04-14
77787-26D
-8-
FIG. 2a is a block diagram illustrating
relationships between composition data structures in a
composition;
FIG. 2b is a block diagram illustrating parallel
and serial relationships between the composition data
structures of FIG. 2b;
FIG. 3a is a diagram illustrating a user interface
for a digital nonlinear editing system;
FIG. 3b is a diagram illustrating a section of
media;
FIG. 4 is a block diagram illustrating
relationships between the source data structures of FIG. 2
and the composition data structures of FIG. 3a;
FIG. 5 is a table illustrating an embodiment of a
relationship;
FIG. 6 is a block diagram illustrating an
embodiment of effects of source color modification and
composition color modification on segments within a
sequence;
FIG. 7 is a block diagram illustrating an
embodiment of source data structures and composition data
structures;
FIG. 8a is a data flow diagram illustrating an
embodiment of a source color modifier;
FIG. 8b is a data flow diagram illustrating an
embodiment of a composition color modifier;
CA 02492870 2000-04-14
77787-26D
-9-
FIG. 9 is a flow chart illustrating an embodiment
of an editing process implementing composition color
modification and a source color modification;
FIG. 10 is a flow chart illustrating an embodiment
of a process of identifying a source data structure defined
by a source relationship attribute;
FIG. 11 is a flow chart illustrating an embodiment
of a process of identifying a composition data structure
defined by a composition relationship attribute;
FIG. 12 is a data flow diagram illustrating an
embodiment of a media player implementing source color
modification and composition color modification;
FIG. 13 is a flow chart illustrating an embodiment
of a process of implementing source color modification and
composition color modification during playback;
FIG. 14 is an example embodiment of a user
interface for color modification;
FIG. 15 is an illustration describing operation in
the HSL color space;
FIG. 16 is a block diagram illustrating an
embodiment of the storage of source and composition color
modifications; and
FIG. 17 is a table illustrating an embodiment of a
composition source table.
DETAILED DESCRIPTION
The following detailed description should be read
in conjunction with the attached drawings in which similar
reference numbers indicate similar structures.
CA 02492870 2000-04-14
77787-26D
-10-
A common attribute that segments may share is a
common source. This common source attribute is also
referred to herein as a source relationship. Modification
for segments having a source relationship will now be
described.
A DNLE may track or have access to information
that indicates how segments of media in a composition may be
related by the source of the media data that they represent.
This information may be used to apply a modification to
multiple segments at the same time. Such modification is
referred to herein as source modification.
The relationships between segments of a source
media in a composition are represented by references to
source data structures representing media that can be
combined or composed into a multi-media composition. Video
and audio modifications, including source video and audio
modifications, are typically defined within the context of a
composition. Compositions are represented by composition
data structures or components. Data structures for source
media and compositions are described in more detail below.
Relationships between segments of media in a
composition, the data structures that represent these
relationships, and the relationships between these data
structures themselves are described in one or more U.S.
patents, including U.S. Patent No. 5,752,029 by
Michael J. Wissner and issued May 12, 1998, entitled Method
And Apparatus For Representing And Editing Multimedia
Compositions Using References To Tracks In The Composition
To Define Components Of The Composition (The Wissner
patent), U.S. Patent No. 5,267,351, issued on
November 30, 1993 by Stephen J. Reber et al. entitled MEDIA
STORAGE AND RETRIEVAL SYSTEM (the Reber patent), U.S. Patent
CA 02492870 2000-04-14
77787-26D
-11-
No. 6,374,336 and issued April 16, 2002, by Eric C. Peters
entitled Computer System And Process For Transferring
Multiple High Bandwidth Streams Of Data Between Multiple
Storage Units And Multiple Applications In A Scalable And
Reliable Manner (the Peters patent). Source relationships
and data structures also are described in the OMF
Interchange Specification (OMF), version 2.1, 1997,
available from the OMF Developers' Desk Of Avid
Technologies, Inc., and available on the Internet at the
URL: http://www.avid.com/3rdparty/omfi, and in the Advanced
Authoring Format specification (AAF).
A general summary of such source relationships and
data structures that may be used will now be described.
A DNLE system typically permits an editor to
create a multimedia composition. A multimedia composition
is collection of relationships between time-varying media
data, representing how the data should be synchronized and
combined over time. Time-varying data, may be, for example,
video or audio data, but is not limited to such data.
Static data that does not vary with time, for example, still
pictures and text, is a subset of time-varying data, and may
also be included in a multimedia composition. The data are
related by grouping them into different types of components,
the combination of which forms a composition. A method and
apparatus for representing such a media composition is
described in one or more U.S. patents, including the Wissner
patent.
Media data used in a composition includes
digitized versions of physical source media such as video or
audio tape, compact disk, computer generated images, etc.
Physical source media are also referred to herein as
physical media. Digitized versions of physical media
CA 02492870 2000-04-14
77787-26D
-12-
available for use are referred to herein as digital media
sources or digital sources. Digital sources may include
digital samples of physical media or may have been created
from application software for graphics, animation, or word
processing, etc. A digital source created by application
software is referred to herein as a digitally created
source. A digital source digitally sampled directly from a
physical media is herein referred to as an original digital
media source or an original digital source. A digital
source may represent a single image or a single sample, and
may be a copy of or a portion of another digital source.
Digital sources are stored in digital media files.
Representations of digital sources and
representations of physical media are referred to herein as
source data structures. Source data structures may be
stored in data files in a data base or any other format. A
source data structure representing a digital source may be
stored in the same media data file as the digital source it
represents or in a separate data file.
A source data structure includes information
describing the media, whether digital or physical, that it
represents. The information may include: how the digital
source was created; an identification of the corresponding
physical media; the sample rate (and therefore the duration
of a sample), and the length of each sample in bytes; an
indication of the section of the physical media that it
represents and the time offset from the source physical
media of its first sample. The units of this offset is the
sample duration for the digital source.
Multiple digital sources of the same physical
media and their corresponding source data structures also
may be stored if desired. Storing multiple digital sources
CA 02492870 2000-04-14
77787-26D
-13-
and source data structures allows the composition to support
the interchange of media at different levels of visual or
audio quality for different purposes. For example, one
digital source might have a level of quality which is
suitable for output to video tape, whereas an alternative
digital source might be useful for displaying in a small
window on a computer screen. Examples of such a system for
storing and accessing multiple digital sources of a single
physical media are described in the Reber patent and the
Peters patent. A commercial storage system suitable for
storing media data files and composition data includes the
MediaShare storage system available from Avid Technologies,
Inc. Other commercial systems may be used.
A source data structure called a source clip
represents a single time-contiguous section of media. As
with other source data structures, a source clip does not
include the actual media data of the digital source, but
only references it, for example by referring to a data file.
A source clip represents a digital source, which could be an
original digital source or a digitally created digital
source. A source clip that represents an original digital
source or a digitally created source is referred to herein
as a master source clip or a master clip. A source clip
that represents a digital source that is a section or a copy
of an original digital source or a digitally created source
is herein referred to as a subclip. A source data structure
that represents a physical media is herein referred to as a
physical source media object or physical media object.
Source clips are described in OMF and AAF.
A source clip may include a source identifier that
identifies another source data structure that represents
either a digital source or physical media that includes the
section of media represented by the source clip. If the
CA 02492870 2000-04-14
77787-26D
-13a-
source clip represents a digitally created digital source,
the source identifier is a null value because the digitally
created digital source does not originate from another
source. A source clip that does not represent a digitally
created digital source also includes a source offset. The
source offset identifies a starting position relative to the
section of media represented by the source data structure
from which the source clip originates.
If a first source data structure is referred to as
originating from a second source data structure herein, the
second source data structure is the source of the first
source data structure. If a first source data structure is
referred to as indirectly originating from a second source
data structure herein, the second source data structure is
the source of at least a third source data structure that is
the source of the first source data structure. "At least" a
third data structure means that there may be multiple source
data structures between the first and second source data
structures, each data structure originating from the other
in a source chain between the first and second source data
structures. For example, a subclip may indirectly originate
from a physical media object, as described below in
connection with Fig. 1.
An example illustration of offset is a source clip
that has an offset of 40 units and represents an original
digital source. The source clip thus originates from a data
structure that represents a physical media, as discussed in
more detail below. If the data structure from which the
source clip originates includes an offset of 100 and
represents units 100-200 of a physical source, the 40 units
defined by the source clip are offset from unit 100 of the
physical source. The source thus represents a section of
CA 02492870 2000-04-14
77787-26D
-13b-
media beginning at unit 40 of the digital source, and which
ends at unit 140 of the physical media.
A source data structure to which a source clip
refers also may refer to another source data structure,
which also may refer to yet another source data structure,
etc. This type of multiple layering is described in the
Wissner patent and by OMF.
An example embodiment of such a multilayered
representation is illustrated in Fig. 1. The physical media
object 48 is a source data structure that presents the
physical source media 54 from which original digital media
sources 50 are created. Each original digital source 50 may
be stored in digital media file as described in the Reber
patent. Each original digital source 50 is a digitized
portion of the physical media 54, where any of the original
digital sources 55 could be a digitization of the entire
physical media 54. For each original digital source 50
created from the physical media 54, a master source clip 46
that represents the original digital source is also created.
Other digital sources such as 52 may be specified
as sections of the original digital sources 50.
Alternatively, other digital sources may be created as
copies of an original digital source 50. For each other
digital source 52 created, subclips 44 are created to
represent the digital source 52. Each subclip 44 originates
from a master clip 46 which may originate from a physical
media object 48. Consequently, each subclip 44 may
indirectly originate from a physical media object 48.
In order to support the editing of compositions of
a variety of media, composition data structures are used for
organizing and storing information concerning a composition
and operations manipulating those composition data
CA 02492870 2000-04-14
77787-26D
-13c-
structures. The basic building blocks of a composition are
called components. A composition is structured as a tree of
components including a root component. A component may or
may not have subcomponents, depending on its type. A
component may be considered a function over time because it
includes information for producing the state of its portion
of the composition at any time within its range. A
component thus represents a time-dependent sequence of media
data or sources called a media stream.
The composition data structures used for
representing the components of a composition exclude the
media data itself. The composition data structures include
indications of or references to the media data and
representations of the relationships between and
combinations of the media data which form the composition.
Thus, compositions are stored separately from the media data
to which they refer, and allow many compositions to use the
same media data without duplicating it.
Several types of data structures may be combined
to form a composition. Fig. 2a illustrates one possible
combination of components. Each segment 6 represents a
section of media as a component of a composition. Although
each segment also may represent more complex structures such
as a sequence or a track group, which is discussed further
below, a segment as referred to herein represents a single
time-contiguous section of media in the context of a
composition, unless otherwise specified. A segment includes
a source identifier that identifies a source clip.
Transitions 8 are components which are located
between two segments in a sequence of components, and
indicate how a presentation should transition from
displaying one segment to displaying the next segment.
CA 02492870 2000-04-14
77787-26D
-13d-
A sequence 4 represents the serialization or
concatenation in time of a collection of components. A
sequence may define an ordered list of segments separated by
transitions, and may itself be a segment within another
sequence. The order of segments in a sequence defines the
order of interpretation or "playback". Each sequence 4 may
include a list of subcomponents and includes the
subcomponent identifier of each subcomponent. An example of
a playback system suitable for playing a composition is
described in the Wissner patent and the Peters patent and
U.S. Patent No. 5,045,940 issued September 3, 1991 by
Eric C. Peters entitled VIDEO/AUDIO TRANSMISSION SYSTEM AND
METHOD (the second Peters patent). Also, a commercial
playback system may be used for playing compositions that
implements the Media Engine video playback system available
from Avid Technology, Inc. that is incorporated in the Avid
AirPlay MP playback server system. Media Composer from
Avid Technology, Inc. is a suitable commercial system for
playback as well as editing. Other commercial systems may
be used.
A track group defines a parallel relationship
between sequences or segments, which are defined by tracks
within the track group. For example, one track within a
track group may be an audio sequence or segment to be
synchronized with a video sequence. Another track within
the track group may be the video segment or sequence, and a
third track within a track group may be a background video
effect segment or sequence to be combined with the video
segment or sequence. A composition is essentially a track
group wherein the parallel tracks begin at a same point in
time and end at a same point in time
CA 02492870 2000-04-14
77787-26D
-14-
Fig. 2b illustrates a representation of a composition 2 from which a still
image
may be shown in Fig. 3a, wherein sequences 4 and 5 are represented by parallel
tracks
14 and 15, respectively. Each track 14 and 15 may be considered a logical
player
channel, and therefore is of a single media type. Because tracks 14 and 15 are
subcomponents of a composition 2, the sequences 4 and 5 start at the same
point in time
and end at the same point in time.
Fig. 4 illustrates the relationship between source data structures and a media
composition. The composition 2 includes 5 tracks, where each track is a
sequence,
including a sequence 4 that includes two segments and a transition. Sequence 4
includes
a segment 6 that originates from a subclip 44. The subclip 44 originates from
master clip
46 that originates from the physical media object 48. Source clip 44 is the
source of
segment 7 as well as segment 6. Segment 7 is a subcomponent of sequence S.
Thus, the
subclip 44 is the source of two segments 6 and 7 which belong to two different
sequences 4 and 5, respectively. Although not shown in Fig. 4, the subclip 44
could be
the source of a segment used in an entirely different composition. The same
kind of
relationship between source data structures and composition data structures
can be true at
the master clip level also. A master source clip or a physical media object
can be the
source of multiple subclips linked to segments of multiple compositions.
Having now described the various kinds of source relationships and an
ernbodiment of the data structures representing these relationships and how
they are used
to define a composition, an embodiment of source media modification will now
be
described.
Source media modification provides a simple, time-saving method of applying a
single modification to several audio or video segments originating from a
common
source. The common source of a segment may be determined by applying a source
relationship parameter (attribute), which may be defined by an editor, to the
segment, as
described in more detail below. .
Fig. 5 illustrates a table 60 that lists possible source relationships for
source
media modification and possible composition relationships for composition
media
modification. The table 60 includes a source relationship column 62 and a
composition
relationship column 64.
The source relationship parameter can be defined as a segment relationship 61,
a
source clip relationship 63, a master clip relationship 65, physical media
relationship 67,
CA 02492870 2000-04-14
77787-26D
-15-
or a source name relationship 59. If the source relationship is defined as the
segment
relationship 61, the modification is applied only to the active segment during
playback.
An active segment, as used herein, means either the segment currently being
edited on an
editing system or a segment currently being played on a playback system. If
the source
relationship is defined as the source clip relationship 61, during playback
the
modification is applied to any segment, used within any composition on the
system, that
originates from the same source clip as the active segment. If the source
relationship
attribute is defined as a master clip relationship, during playback the
modification is
applied to any segment that directly or indirectly originates from the same
master clip
from which the active segment indirectly originates. If the source
relationship.attribute
is defined to be the physical media relationship, the modification is applied
to any
segment that indirectly originates from the same physical media object from
which the
active segment indirectly originates.
For example, an editor may define the source relationship to be a source clip
relationship. If an editor notices that a video segment representing film
footage is too
dark, the editor can increase the luminance of the segment to brighten the
image. During
playback, the increase in luminance may be applied to any video segment
originating
from the source clip.
If the source relationship attribute is defined to be the source name
relationship,
the modification is applied to any segment that shares the same name and same
media
type (i.e., audio or video) as the active segment. The source name
relationship has
particular application in media editing processes that incorporate an off-line
and an on-
line process. For example, typically, in an off-line digital video process, an
editor makes
rough edits to low resolution images of a sequence. Next, the editor batch
digitizes the
edited off-line sequence to produce an on-line sequence having high resolution
images.
In an embodiment of this off-line to on-line digital video process, when the
off-
line sequence is made into an on-line sequence, a new master clip is created
for each
segment of the on-line sequence, from which each segment of the new on-line
sequence
directly originates, respectively. The source name relationship 59 allows a
common
attribute, i.e., name, to still exist between segments of the on-line sequence
that
otherwise may not still have a common source relationship due to the batch
digitize.
This is described below in more detail in connection to Figure 6.
CA 02492870 2000-04-14
77787-26D
-16-
In an embodiment of source media modification, the scope of the effect of the
modification may be restricted to only the sequence or composition to which
the active
segment belongs. Such a restriction may be an option for an editor or colorist
selected
through a user interface or may be a system option. Such a restriction may be
considered
as a combination of source media modification and composition media
modification,
which is described below. For this restricted source media modification
embodiment, the modification
specified by a user will be applied to all segments that meet the requirement
of being a
part of the composition or sequence currently being edited, in addition to
satisfying the
source relationship. Whether the requirement is to be part of the current
compositiQU or
part of the current sequence may be a user or system option. Restricting the
effect of a
source image modification ensures that a colorist or editor does not
inadvertently modify
a composition or sequence being edited or colorized by another user.
Composition media modification provides a method for an editor to apply a
single modification to several segments within a composition. A composition
relationship parameter (attribute) defines the ancestor of the active segment
that is used
to determine a common ancestor. The common ancestor is used to determine all
descendants in the composition to which the modification is to be applied
during
playback.
Referring to the table 60 of Fig. 5, the composition relationship parameter
may be
defined to be a segment relationship 69, a selected segments relationship 71,
a sequence
relationship 73, or a composition relationship 75. If the composition
relationship
parameter is defmed to be the segment relationship 69, only the active segment
has the
modification applied during playback. If the composition relationship
parameter is
defined to be the selected segments relationship 71, only the selected
segments have the
modification applied during playback. If the composition relationship
parameter is
defined to be the sequence relationship 73, all segments descended from or
included in
that sequence have the modification applied during playback. If the
composition
relationship parameter is the composition relationship 75, all segments of the
composition have the modification applied during playback.
The combination of composition media modification and source media
modification applied to a single segment will now be described with reference
to Fig. 6.
CA 02492870 2000-04-14
77787-26D
-17-
Although color modification is used in the following example, this example
analogously
applies to other types of audio and video modifications, for example, audio
mixing.
Suppose that various footage has been shot by a film crew to be used in a
movie.
Included in this footage is some footage from scene A, footage from another
scene B.
footage from another scene C, and footage from yet another scene D. The film
editors
decide to use digital non-linear editing to combine the footage, make color
modifications,'and add special effects, etc. to produce the finished movie.
First, the editors create original digital sources from the film footage.
Thus, an
original digital source A, an original digital source B, an original digital
source C, and an
1 o original digital source D are created. As discussed above, for each=
digital source =ated,
a master source clip that represents the digital source is created also. Thus,
a master
source clip A, a master source clip B, a master source clip C and a master
source r-lip D
are created.
The editors then decide that the footage of scene B can be divided into two
separate shots: shot 1 and shot 2. Thus, from the original digital source B, a
digital
source of shot 1 and a digital source of shot 2 are created. In response to
the creation of
the two new digital sources, a subclip of shot 2 is created to represent the
digital source
of shot 2, and a subclip of shot 1 is created to represent the digital source
of shot 1.
The editors then decide to combine the footage of the various scenes together.
Using a digital non-linear editor, the editors create a composition that
includes a
sequence 66 that includes a segment 74 originating from the master clip A, a
segment 76
that originates from the subclip of shot 2, a segment 78 that originates from
the master
clip C, a segment 80 that originates from the subclip of shot 1, and a segment
82 that
originates from the master clip D.
While viewing the various segments in the video display, an editors may notice
that the segment 76 of shot 2 is too dark. The editors may decide to make a
color
modification to increase the luminance of this segment.
Using source color modification, the editors define the master clip
relationship as
the source relationship attribute. The editors then increase the luminance of
the segment
3o 76 of shot 2 by 5 IRE. Because the source relationship attribute was set to
master clip
relationship, upon playback both the segment 76 of shot 2 and segment 80 of
shot 1,
which both indirectly originate from the master clip B, have luminance
increased by 5
IRE as shown in item 68 of Fig. 6.
CA 02492870 2000-04-14
77787-26D
.
-18-
The editor may then realize that all the segments of the sequence 66 are too
bright. The editor may set the composition relationship attribute to a
sequence
relationship. The editor may then decrease the luminance of the active segment
by 2
IRE. Because the coniposition relationsliip attribute is defined as a sequence
relationship, ai1 segments within the sequence 66 have their liuninance
decreased by two
IRE during playback, as shown by row 70 in Fig. 7. Row 72 shows the combined
color
modification of both the composition color modification and the source color
modification that is applied to each segment within the sequence 66 during
playback.
In the above example, the source name relationsliip 59 would be usefid if the
sequence 66 is an off-line sequence and the above edits were= made as part of
an off-line
process. If'the off-line sequence 66 were then batch digitized, new master
clips would be
created for each of the segments 74, 76, 78, 80, 81 of the new oa Iine
sequence.
Consequently, an editor could no longer use the master clip relationship to
specify a
single edit to affect both segments 76 and segment 80. In an embodimeni,
assuming the
name of the original master clip of scene b is Sceneb, the new master clips
for segmeats
76 and 80 may be entitled Sceneb.new 01 and Sceneb.new.02, respectively. In
this
embodiment, the source name relationship 59 allows a sysoem user to specify a
single
edit for both segmeats 76 a.ud 80. Thus, if an editor notices that both the
segmarts in the
on-line sequence originating from scene b are too bright, the editor may
specify a
modification of -1 IRE for either segment 76 or 80 the modification of
-1 IRE is applied to both segments.
For example, the editor first accesses segment 76, wbich originates from
master
source clip Sceneb.new.01, and specifies a color modification of -1 iRE. If
the editor has
the source name relationship attn'bute 59 selected, the system deoermines that
the source
name is Sceneb (where the .new.0l extension is ignored). The system identifies
other
segments originating from source data struciure having a name Sceneb, and
applies the
image modification to the segments that satisfy the name requirem=t.' In the
above
example, segment 80 satisfies the requirement, and the image modific.ation is
applied to
it.
Fig. 7 illusQates an embodiment of source and composition data stnucmm that
can be used for source and composition media modification. Although this
embodiment
descnbes image modification, embodiments of audio sample modification one
within the
scope of one of ordinary skiIl in the art of digital media editing. A segment
130 includes
CA 02492870 2000-04-14
77787-26D
-19-
information 132 about the section of media that the segment 130 represents.
This
information 132 may include a source identifier 134, a parent identifier 136,
a source
image modification attribute 138, and a composition image modification
attribute 140.
The source and composition image modification attributes 138 and 140 may
themselves
be partitioned into a plurality of image modification attributes.
The source identifier 134 identifies the source data structure from which the
segment 130 originates. The parent identifier 136 identifies the parent of the
segment in
a composition 158. The source image modification attribute 138 represents a
source
image modification to be applied to the segment 130 during playback. The
composition
to color modification attribute 140 defines a composition image modification
to -be applied
to the section of media represented by the segment 130 during playback. It
should be
noted that if a source or composition image modification is not defined for a
data -
structure, an image modification attribute may be a null value, or the image
modification
attribute may not be included in a data structure.
In Fig. 7, the source identifier 134 points to a subclip data structure 122.
The
subclip data structure 122 includes information 124 about the section of media
that the
subclip represents. The information 124 includes a source identifier 126 and
may
include a source image modification attribute 128, depending upon whether the
source
image modification was defined for the. subclip 122 during an editing session.
The
source identifier 126 of the subclip 122 refers to a master clip data
structure 114. The
master clip 114 includes information 116 about the digital media that the
master clip 114
represents. The information 116 may include a source identifier 118 and may
include an
image modification attribute 120. If the master clip 114 represents a
digitally created
digital source, the source identifier 118 may be a null value indicating that
the master
clip does not originate from another source.
The source identifier 118 of the master clip 114 may refer to a physical media
object 106. The physical media object 106 includes information 108 that
describes the
physical media that the physical media object 106 represents. The information
108 may
include a source identifier 112 and may include an image modification
attribute 110.
The information 108 includes a source identifier 112 if the physical media
that the
physical media object represents was created from another physical media. For
example,'
the physical media may be a video tape that was created from a film through a
telecine
CA 02492870 2000-04-14
77787-26D
-20-
process. In sucfi case, the source identifier 112 refers to the physical media
object that
represents the film.
In the example of Fig. 7, the parent identifier 136 of the segment data
structure
130 refers to a first sequence data structure 142. The first sequence 142
includes
information 144 about the first sequence 142. The first sequence information
144 may
. ~
include a parent identifier 146 and a subcomponent identifier 148. The first
sequence
information 144 would not include a parent identifier 146 if the sequence 142
is itself a
composition, which is not the case in Fig. 8. The first sequence information
144 may
include an ordered list of subcomponents of the first sequence 142, where the
order of
the subcomponents determines the order in which the subcomponents are played
during
playback. For each subcomponent, the information 144 may include a
subcomponent
identifier 148.
In the iAustration of Fig. 7, the subcomponent identifier 148 identifies
segment
130. The parent identifier 146 identifies a second sequence 150 of which the
first
sequence 142 is a subcomponent. The second sequence 150 includes information
152
about the sequence 150. The second sequence information 152 includes
information
analogous to that described for first sequence 142, including a parent
identifier 154 that
refers to the composition data structure 158. The second sequence information
152
would not include a parent identifier 154 if the second sequence 150 is itself
a
composition which is not the case in Fig. S. The information 152 also includes
an
ordered list of subcomponents, the order of the subcomponents determining the
order of
playback of the subcomponents during playback. For each subcomponent, the
information 152 includes a subcomponent identifier 156. The subcomponent
identifier
156 identifies first sequence 142 as a subcomponent of the second sequence
150. The
parent identifier 154 identifies the composition 158 as the parent of sequence
150.
Composition 158 includes information 160 about the composition 158. The
composition information 160 includes the sequences of each track of the
composition,
wherein the sequences are subcomponents of the composition 158. For each
subcomponent of the composition 158, the composition information 160 includes
a
subcomponent identifier 162. The subcomponent identifier 162 identifies
sequence 150
as a subcomponent of composition 158.
Fig. 8A is a dataflow diagram that illustrates how the source data structures
illustrated in Fig. 7 can be used to implement source image modification
during an
CA 02492870 2000-04-14
77787-26D
-21-
editing session. A source image modifier 82 receives user-defined image
modifications
for an active segment 84 and a source relationship parameter 86. Using the
user defined
image modification 84 and the attribute 86, the source image modifier 82
accesses the
source data structures 88 and modifies the source data structures 88 in
accordance with
the image modification 84 and the source relationship parameter 86.
Fig. 9 is a flowchart illustrating an editing process implementing source
image
modification 186 and composition image modification 188. With reference to
Fig. 9, the
implementation of source image modification 186 by the source image modifier
82
during an editing session will now be described. First, in step_ 164, an
indication of an
t o image modification is received. Next, in step 166, an editor may detennine
wltether the
image modification is a source image modification or a composition image
modification.
Such a determination is made, for example, by accessing an indication of the
image
modification mode selected by an editor. If it is determined that source image
modification shall be applied, at the next step 178, the source image modifier
determines
the source relationship to be used to determine a common source of the active
segment
and other segments on the system. Determining the source relationship to be
used can be
accomplished by either accessing a default source relationship parameter or by
receiving
a source relationship parameter defined by an editor.
Next, in step 180, the source image modifier identifies a source data
structure of
2o the active segment defined by the source relationship. Fig. 10 is a
flowchart that defines
in more detail the step 180 of identifying the source data structure. First,
in step 190, the
source image modifier 82 determines whether the current source data structure
is the
source data structure defined by the source relationship. The first time
through the loop
defined by steps 190-194, the current source data structure is the active
segment. If the
current source data structure is the defined source data structure, then the
step of
identifying the source data structure of the active segment is done, as
illustrated in step
196. If the current source data structure is not the defined source data
structure, then in
the next step 192, the source image modifier 82 reads the source identifier of
the current
source data structure. In step 194, the source image modifier then accesses
the source
3o data structure identified by the source identifier. Steps 190-194 are
repeated until the
source data structure defined by the source relationship attribute is
identified.
CA 02492870 2000-04-14
77787-26D
-22- =
Returning to the flowchart of Fig. 9, in the next.step 182, the source image
modifier stores an indication of the source image modification in the source
data
structure identified in step 180.
In an altemative embodiment of source media modification, modifications may
be stored as an entry of a composition source table, as opposed to being
stored in a
source data structure as described above. Figure 17 is an example illustration
of a
composition source table 500. Each entry 502 in the composition source table
include
two pieces of data: the identification of a source data structure 504 for
which a source
modification of a segment within the composition is defined;.and an attribute
506
representing the source modification. If a source modification is defined for
a segment
within the composition, an entry may be stored in the composition source table
500
identifying the source data structure from which the segment originates in
accordance
with the source relationship parameter of the segment, and the attribute
representing the
modification.
The composition source table allows a source media modification to be shielded
from other compositions on the system that use material from the defmed
source. The
source media modification may be defined specifically for segments of the same
composition that originate from a common source, but not for all segments on
the system
originating from the common source. Thus, other editors are not forced to
implement
modifications on a composition that they are editing where the modification
was defined
by a different editor working on a different composition.
Another benefit of using the source composition table 500 is that it provides
an
editor with the option of not applying a source media modification to the
composition.
An editor may select whether the composition source table 500 is to be
activated. For
example, if a colorist defines a series of source color modifications for a
composition,
the source data structures and the color modifications are stored in entries
of the
composition source table 500 as described above. At a later point in time, an
editor may
access the composition, but not activate the table, and thus not apply the
modifications
defined therein.
In this alternative embodiment, for an active segment during playback, the
modification system may first determine if there is a source modification
defined for the
active segment data structure. If a modification is defined for a segment data
structure,
the modification may be applied. If no modification is defined in the active
segment data
CA 02492870 2000-04-14
77787-26D
-23-
structiur, the composition source table 500 may be accessed. If it is
determined that
there is an entry for the subclip from which the active segment originates,
the source
modification defined for the subclip is applied to the active segment. If it
is determined
that no entry for the subclip is present in the table 500, it is determined
whether the table
500 includes an entry for the master clip from which the segment originates.
If it is
determined that there is an entry for the master clip from which the active
segment
originates, the source modification defined for the master clip is applied to
the active
segment. If it is determined that no entry for the master clip is present in
the table, it is
determined whether the table includes an entry for the physical media object
from which
l 0 the segment originates. If it is detennined that there is an entry for the
physical iaedia
object from which the active segment originates, the source modification
defined for the
physical media object is applied to the active segment. Although the
altemative
embodiment is described above with respect to the source relationslup common
attribute,
this embodiment also applies to other common attributes, such as the
formatting
ts attributes and color attcibutes described above.
For example, instead of a composition source table, a composition format table
may be maintained. An entry in the format table includes two pieces of data:
the
identification of a common format attribute; and an attribute representing the
modification defined for segments of the composite having the format
attribute. When a
20 modification is defined for a segment within the composition, an entry may
be stored in
the composition format table identifying the common format attribute for which
the
modification was specified. The common fonaat attribute may be determined from
a
default common attribute parameter or a user-defined common attribute
parameter.
In another altemative embodiment of source modification, the source modifier
82
25 accesses every segment on the system that originates from a source data
sttuctime 88
determined from the source relationship parameter 86 of the active segment 84.
The
source image modifier 82 then modifies each of accessed segments,
respectively, with
the modification.
In an aspect of this alternative embodiment, the source modifier 82 only
accesses
30 and modifies segments in the current sequence or composition of the active
segment 84
that satisfy the source relationship parameter 86. A system option or user
option may be
used to determine whether the modification is restricted to the sequence or
the
composition. This alternative embodiment of source image modification
analogously
CA 02492870 2000-04-14
77787-26D
-24-
applies to other types of common attributes image modification, where a modif
er
accesses every segment on the system that shares the common attribute and
modifies
each segment with the modification. The common attribute may be determined
from a
default or user-defined common attribute parameter.
Fig. 8B is a dataflow diagram illusirating how composition image modification
can be implemented by a composition image modifier 90. The composition color
modifier 90 receives image modifications 92 to be applied to the active
segment. The
composition image modifier 90 then accesses a default relationsliip attrt'bute
or receives a
composition relationship attribute 94 defined by an editor. The composition
color
modifier then accesses the composition data sftuetures 96 and modi.fies the
composition
data stractures 96 in accordance with the image modification 92 and the
relationship
attabute.
Referring again to Fig. 9, an implementation of composition image modification
implemented by the composition image modifier 90 will now be desmt'bed As
des=13ed
above with respect to source image modification, the first step is step 164 of
receiving an
indication of an image modification. The digital media editor then selects the
editing
system then determines whether source image modification or composition image
modification is to be applied in step 166.
If composition image modification is to be applied, in the next step 168, the
composition image modifier detetmines the composition relationship to be used.
The
composition reiationship is defined by either a default composition
relationship
parameter or a composition relationship parameter defined by an editor.
In the next step 170, the composition image modifier identifies the ancestor
data
structtue of the active segment defined by the composition relationship
parameter.
Fig. 11 provides a more detailed flowchart of step 170. First, in step 190,
the
composition image modifier determines whether the current composition data
structure is
the ancestor data structure defined by the composition reistionship
atttibafie. The first
time through the loop defined by steps 198-202, the current composition data
strncture is
the active segment. If the caaent composition data strncture is the defined
ance,stor data
structure, then the step 170 of identifying the ancestor data sttucture is
done, as
iIlustrated in step 204. If the current composition data structtnt is not the
detined
ancestor, the next step 200 reads the parent identifier of the current
composition data
structure. In the next step 202, the composition image modifier accesses the
composition
CA 02492870 2000-04-14
77787-26D
-25-
data structure identified by the parent identifier. Steps 198-202 are then
repeated until
the ancestor data structure defined by the composition reiationship attribute
is identified.
Returning to Fig. 9, in the step 172, the composition image modifier
determines
the descendants of the identified ancestor data structure, for example, by
accessing the
subcomponent list of the identified data structure and accessing the
subcomponent
identifiers for each subcomponent, and traversing downward in the composition
from the
identified ancestor data structure. In step 174, for each identified
descendant or
subcomponent of the identified ancestor data strucwre, an indication of the
image
modification is stored in a descendant's data sftucttme. Traversing a
composition to
to gerform editing operations on componeats is descn'bed in the Wissner patent
A.nother aspect of source and composition media modification is the
appiicatzon
of the modifications to segments within a composition during playback. Fig. 12
is a
dataflow diagram tbat illustrates an implementation of source and composition
image
modifieation during playback. Although this embodiment descxibes image
modification,
embodiments pertaining to audio sample modification are within the scope of
those
siciiled in the art of digitat media editing. A digital media ptayer 98
receives an
indication of a starting point 100 within a composition. The starting point
100 may be
the beginning of the composition by defanlt or may be defined by input from an
editor on
a user interface. The digital media player 98 accesses composition data
structtues 96 to
properly synchronize the playing of sections of a digital media.
When a segment is identified within the composition, the digital media player
identifies source data structures 88 from which the active segment originates,
and
determines whether any source image modification attabutes are defined for
these source
data structures 88. The digital media player also uses the source data
stiucture.s 88 to
2s determine the digital sources from which the digital media 104 should be
accessed. The
digital media player also determines whether the active segment has a
composition
image modification. 'lhe digital media player then applies the source image
modification
and the composition image modification to the active segment This process is
repeated
for every segment of the composition, or every segment included in a portion
of the
composition selected by the user, such that a sequence of digital images 102
is produced
as defined by the composition. A playback system may apply image modification
sequeatially on a segment, one digital image at a time. More specifically, a
playback
system may apply image modification sequentially on an digital image. Such a
system is
CA 02492870 2000-04-14
77787-26D
26
described in U.S. Patent No. 6,417,891 issued July 9, 2002
and entitled "Color Modification on a Digital Nonlinear
Editing System" (the Cacciatore patent) by Ray Cacciatore
and Rob Gonsalves.
Fig. 13 is a flowchart that illustrates an
implementation of source and composition image modification
during playback. In step 206, the digital media player 98
accesses the next segment in a composition. Next, in step
208, the digital media player 98 determines whether a source
image modification attribute is defined for this data
structure. If there is a source image modification
attribute defined for this data structure, the digital media
player determines whether there is a temporarily stored
source image modification attribute for this active segment.
It should be noted that the first time through the
loop defined by steps 208 - 216, the source data structure
is the active segment. For this reason, the first time
through the loop defined by steps 208 - 216, at step 210
there is not a temporarily stored image modification
attribute for the active segment. Alternatively, for the
first time through the loop defined by steps 208 - 216, step
210 may be skipped. The data structure that holds the
temporarily stored source image modification attribute for
the active segment also may be initialized between steps 206
and 208.
Returning to the flowchart of Fig. 13, if there is
no temporarily stored source image modification attribute,
in step 212 the digital media player temporarily stores the
source image modification attribute for the active segment.
If the digital media player determines in step 210 that
there is already a temporarily stored source image
CA 02492870 2000-04-14
77787-26D
26a
modification attribute for the active segment or determines
in step 208 that there is no image modification attribute
defined for this data structure, the digital media player
proceeds to step 214. In step 214, the digital media player
determines whether there is a source data structure from
which the current source data structure originates.
If the media player determines that there is a
source data structure from which the current source data
structure originates, the media player proceeds to step 216.
In step 216, the media player accesses the source data
structure from which the current source data structure
originates. Steps 208 - 216 are repeated until there is no
source data structure from which the current source data
structure originates.
If there is no source data structure from which
the current source data structure originates, the digital
media player proceeds to step 218. In step 218, the digital
media player determines whether there is a temporarily
stored source image modification
CA 02492870 2000-04-14
77787-26D
-27-
attribute for the active segment. If there is a temporarily stored source
image
modification for this segnent, the digital media player in step 220 applies
the
temporarily stored source image modification attribute to the active segment.
After applying the temporarily stored source image modification attnt'bute to
the
s active segment, or after determining that there is no temporarily stored
source image
modification attribute for this segment, the digital media player proceeds to
step 222. In
step 222, the digital media player determines whether there is a composition
image
modification defined for the active segment. If there is a composition color
modification
defined' for this segment, the digital media player proceeds to step 224 where
it applies
1o tht composition image modification to the active segmeat.
After applying the composition image modification to the active segment craRer
determining that there is no composition image modification to find for this
active
se}nent, the digitai media player then proceeds to step 206 where it accesses
the next
segment in the composition. Steps 206-224 are repeated until the entire
composition or
is the portion oftiu composition defined by a user has been played.
Fig. 14 is an example embodiment of a user interface 400 for digital media
modification. Although not shown ia Fig.14, various audio editing tools and
controls
may be part of user interface 400. A three-image display 270 includes a
current image
displav 229, a previous image display 228, and a next image dispiay 230. The
cu:ient
2o image display 229 represents an active segment of a sequence, and the
previous image
display 228 and the next image display 230 represent the previous and next
segmeats of
the sequence, respectively. A control panel, for example the control pane1272
of the
csurent image 229, may be associated with each image display. For the current
image
display, for example,.the control panel may include a previous segment
button?a"2, a
25 next segment button 234, a previous unmodified segment bimon 236, a next
unmodified
segment button 238, a play button 240, and a stop button 242. Each image may
also
include a scroll bar, for example scroll bar 263 of the current image 229.
Each scroll bar
mav include a position indicator 262 which indicates the temporal position of
the current
image 229 within the segment that it represents.
30 The three image display 270 allows the editor to see the effects of image
modification. If selected, the previous segment button 232 replaces the active
segnnent
with the previous segment in the current image display 229. In the example of
FIG 14, if
the previous segment button 232 is pressed, a digital image from the previous
segment
CA 02492870 2000-04-14
77787-26D
-28-
appears in the current image display 229. Analogously, if the next segment
button 234 is
selected, a digital image from the next segment appears in the current image
display 229.
During an editing session, various image modifications may be defined.
Depending on the conunon attribute parameter chosen during the editing
session, various
segments of the sequence being represented by the three image display 270 may
be
affected by the image modification. The previous unmodified segment button 236
and
the next unznodified segment button 238 allow an editor to view the effects of
image
modification in an efficient manner.
If the previous unmodified segment button 236 is selected, a previous segment
closest in time to the active segment that has not yet had an image
modification applied
replaces the active segment in the current image display 229. The unmodified
segment
buttons allow an editor to quickly determine which segments of a sequence have
not
been modified. Rather than having to use the next and previous buttons 232 and
234
repeatedly to traverse all the segments in a sequence, even when the segments
have
already been modified, an editor can simply press buttons 236 and 238 to view
the
closest unmodified segments.
In an embodiment of the user interface 400, the user interface 400 includes a
second display window, for example, a second 3-image display 700. An editor or
colorist may use images from both 3-image display 270 and 3-image display 700
to edit
or colorize, images from each display. For example, an editor may color match
a color
from image display 712 into current image display 229. Color matching is
described
below in more detail. Another example would be to add the image of previous
image
display 228 to the end of the sequence being displayed in 3-image display 700,
and
possibly define a blend between the last image of the sequence of 700 and the
image of
the previous image display 228.
In another embodiment of the user interface 400, a common attribute tool panel
702 is included. The tool panel 702 may include a remove button 704, an update
button
706, a flatten button 710, and a merge button 708. The remove button 704
allows a
system user to remove all previous modifications specified for a particular
common
attribute from the current sequence or composition, the modifications having
been
applied while editing the current sequence or composition, respectively. A
user may
select the remove button 704, and then be presented with a screen, pull-down
menu,
dialog box, etc., that provides a list of common attributes for which the
remove may be
CA 02492870 2000-04-14
77787-26D
-29-
specified. The user may then select the common attribute resulting in all
previously
applied modifications that were specified for the selected common attribute
being
removed from the current sequence or composition.
For example, suppose a user previously made modifications while editing the
current sequence, specifying the physical media source relationship as the
common
attribute, and the modifications modified several segments of the current
sequence. If
the user selects the remove button, and then selects the physical media source
relationship as the common attribute, all the previous modifications specified
for the
physical media relationship are removed or reversed from the segments of the
current
sequence modified above. - - -
The update button 706 allows a user to apply previous modifications made
during
an editing session to new segments that have been added to a sequence or
composition
since the modifications were made. For example, earlier in an editing session
a user
may have lowered the luminance of all segments in a video sequence that share
a
common source name by 10 IRE. A user may have then added a new segment to the
sequence that also shared the common source name for which the luminance
increase
was previously applied. Selecting the update button 706 applies the previously
defined
luminance increase to the newly added segment sharing the common source name.
The flatten button 710 may be used to return a specified common attribute
parameter to a default common attribute parameter. For example, a user may
want to be
sure that certain edits only apply to the active segment. Commonly, the
default source
attribute parameter and default composition attribute parameter are defined as
the
segment relationship. Thus, selecting the flatten button 710, the source
relationship
attribute parameter and a composition relationship parameter returns to the
segment
relationship.
The merge button 708 can be used when combining segments from one sequence
with segments from another sequence. A typical situation in which this
function would
be used is when an editor, who has trimmed and combined several video segments
in a
sequence, wants to incorporate the color modifications made by a colorist to
at least one
of the video segments of the editor's sequence. In such an example, the
colorist may
have copied the original sequence that the editor had started with, and made
several color
modifications to each of the segments. The merge button allows several changes
made
by both the editor and colorist to be combined in a single step. However, when
these
CA 02492870 2000-04-14
77787-26D
several changes are combined, conflicts may occur. For
example, the editor may have specified a color change that
increases the hue of all images in the sequence, while on
the other hand the colorist may have specified a color
5 modification that decreases the hue of all segments within
the sequence. I
There are several ways by which these conflicts
may be resolved. One of these ways may be assigned as a
default, selected as a system option, or selected by a user.
10 Thus, the ways in which the merge button may resolve
conflicts includes: automatically selecting the image
modification specified by the destination sequence to which
the source sequence is being applied; automatically
selecting the image modification specified by the source
15 sequence, which is being applied to the destination
sequence; automatically selecting the image modification
which came first or last in time, in accordance with a time
stamp associated with the image modification; or prompting
the user to select between the conflicting image
20 modifications.
In an example embodiment using color modification
and source and composition relationships as common attribute
parameters, color function buttons allow an editor to select
function screens for both source and composition color
25 modification. For example, tab 244 allows a user to select
color functions for source color modification, and tab 246
allows a user to select functions for composition color
modification. The color function screens allow an editor to
perform specific color modification functions. Color
30 function buttons may include an HSL button 254, a channels
button 252, a levels button 250, and a curves button 248.
The HSL button 254 brings up an HSL function screen that
CA 02492870 2000-04-14
77787-26D
30a
allows a user to define changes to pixel colors in HSL color
space, including defining color modifications to pixel
colors as a function of the luma of a pixel. Channels
button 252 allows a user to access a channels screen, where
channels in this context refers to the red, green, and blue
components of an RGB component format. The RGB screen
allows a user to modify values of the RGB components as a
function of a component of the RGB components or
combinations of the RGB components.
The levels button 250 allows a user to access a
levels screen in which a user can determine the effects of
RGB color modifications on the luma of the pixel, and toalter the luma of the
pixel as a function of the luma of the
pixel. Example embodiments of the color modifications
available on a levels screen is described in published PCT
application WO/0064141.
CA 02492870 2000-04-14
77787-26D
-31-
The curves button 248 allows a user to access a curves screen. An example
embodiment of a curves screen is illustrated in user interface 400, in which
the curves
button has been selected. The curves screen allows a user to define color
modifications
for a red, green, or blue component of a color, or for all three components of
the color.
The curve screen includes a red graph 280, a green graph 282, and a blue graph
284 for
defining the functions of the individual components, and a master graph 286
for defining
a function that is applied to all three color components. In each graph, the
horizontal
axis represents an input value of the function, while the vertical axis
represents the
output value of the function. Each graph may include control points that if
added, moved
or deleted alter a curve representing the function, thereby altering the
function.-
For example, for the green graph 282,, by altering any of the control points
292,
294, 296, and 298, the green curve 300 is altered, thereby redefining a
function for the
green component. The new values for the functions are determined using
interpolation,
for example, linear interpolation, cubic splines, or Bezier curves.
In an example embodiment, the curve screen of the user interface 400 allows a
user to color match using colors from images of the three-image display 270,
text input
from a text entry field such as 288, or other color sources. For example, a
user may
select a color from the image displayed in the next image display screen 230
and apply it
to the image in the current image display screen 229. The curve screen allows
the user to
preview the effect of adding a color to an image, using the graphs 280, 282,
284, and
286. The curve screen also provides a color matching gadget 257 to assist an
editor in
color matching. The color matching gadget 257 includes an input swatch 261 and
an
output swatch 260. The RGB values of the selected colors for color matching
are
displayed in the swatches. The RGB graphs 280, 282, 284, and 286 allow the
user to
preview these colors without committing to a change in the functions
represented by
these graphs.
Typically, an editor picks from the current image display 229 for the source
color, the value of which is displayed in the input color swatch 261, and
picks from a
known good color, the value of which is displayed in the output color swatch
260. A
known good color may be provided by a color palette, or from another image,
such as the
image displayed in the previous image display 228 or the next image display
230. The
user may select how they would like to match the color. There may be eight
choices,
including: master, R+G+B, R+G, G+B, B+G, Red, Green, Blue. If an editor picks
CA 02492870 2000-04-14
77787-26D
32-
R+Cr+B as the match type, as displayed in selection box 258 of the user
interface 400,
and hits the match color button 256, then a new control point is added to each
of the red,
green, and blue graphs 280, 282, and 284, respectively. Hash marks or some
other
indicator on the graphs represent the changes to those graphs resulting from
the color
match- For example, the hash mark 294' on the green graph 282 represents a
change in
the value of the control point 294, which alters the green component fimction.
In an embodiinent of color matching, the hue and satiuation of the ontpiut
color
are automatic:ally adjusted to match how objects behave under natiual lighting
environments. For example, if an object is illuminated from natural light, the
ratios
1o between the RGB values of the object remain proportional from areas of the
object
highly illuminated by the light to darker areas of the object not illaminated
by as mn-hc
light
Natural color matching operates in the following maaner. Given a source color,
Rs, Gs, Bs, for example, the color represented in the input color swatch 261,
and a
destination color RD, GD, BD, for example, the output color represeated by the
output
color swatch 260, an adjusted destination color, R D, G D, B D is detemiined.
The luminance of the source color, Ys may be defined as:
Equation 1: Ys = 0.299 = Rs + 0.587 = Gs + 0.114 = Bs
Input red/inminancr_ ratio, PRs, input ~uminance ratio, Pvs, and inpd
2o blue/Iumi*+A+s-e ratio, pB& may be defined as:
Equatton 2: pxs -PIS "
Ys
Equation 3: pGs -Gs
Ys
Equation 4 : pBs -Bs
Ys
The luminancx of the destination color, Yp, may be defined as:
Equation 5: Yn = 0.299 = RD + 0.587 = GD + 0.114 = BD
CA 02492870 2000-04-14
77787-26D
-33-
Output red/luminance ratio, ppj), output green/luminance ratio, pop, and
output
blue/luminance ratio, pBp, may be defined as:
Equation 6: Pw = Rn
Yn
_-Gn
Equation 7: Pr;n
Yn
Equation 8: Pav - Bn
YD
Adjusted red/luminance ratio, p' RD, adjusted green/luminance ratio, p'cD,aad
io adjusted blue/luminance ratio, p'Bn, may be defined as:
Equation9: R'11= p'u)Rs= Y' Rn
Px; Yn
Equation 10 : G'õ = Pc;n G r Y~ Gn
Pas YD
Equation 11: B'n=PB"Bn-Y-v
-Bn
pB,) Yn
In a color difference space, such as YCbCr, the Y value represents the
luminance
and the two color components, CbCr, represent the chrominance. Chrominance may
be
defined by two values: hue, which is defined by the angle of the vector (0,0) -
> (Cb.Cr),
and saturation which is the magnitude of this vector and may be defined as ~+
C; .
Fig. 15 illustrates the effect of natural color matching in HSL space. By
matching
the ratios of RGB to luminance, a resulting modified image adopts the hue of
the
specified destination color, Hn, but maintains the luminance Ys of the source
color. If a
destination vector is drawn from the black point to the destination color, the
adjusted
color, C'o, is located at the intersection of this destination vector and a
plane defined by
the source luma Ys. The saturation of the adjusted color, C'D, is the
magnitude of an
adjusted vector that may be de5ned as C'a + C'~ .
Natural color matching may be used to match any combination of the red, green,
and blue components. R+G+B matching was described above. Natural matching with
CA 02492870 2000-04-14
77787-26D
.
-34-
the, master curve affects luminance only. Six other combinations that can be
used are:
R+G, G+B, B+G, Red, Green, and Blue.
An embodiment of natural color matching allows a user to select the other
color
match combinations, R+G, G+B, B+G, Red, Green, Blue. The process of naturally
matching colors for these selections is similar to matching with all three RGB
components as described above. Ratios are detemiined by dividing the selected
components of a color by the weighted sum of the components not selected. One
may
consider two classes of these selections: single eomponent selections
including Red,
Green, and Blue, and two component selections including R+G, G+B, and B+G.
For single component selections, given the source color components, Rs, Gsrand
Bs, and the destination color components RD, GD, BD, an adjusted destination
component
may be determined as follows, using an adjusted red component, R'n; as an
example. A
weighted sum of the green and blue source components, Gp and BD approximates
source
luminance, Ys. This weighted sum may be defined by the following equation:
0.587=Gti+0.1140Bs
Equation 12 : Ys = 0. S87 + 0.114
The source red/luminance ratio, pits, of the source color is determined. The
source
red/luminance ratio pRs may be defined by the following equation:
Equation 13 : pR, = R'
Ys
A weighted sum of the green and red destination components, Os and Bs , may
approximate the destination luminance, Yp. This weighted sum may be defincd by
the
equation:
Equation 14 : YD = 0. 587 = Gõ + 0.114 = B D
0.587 + 0.114
The destination red/luminance ratio, pRs, is determined. The source
red/luminance
ratio pRs may be defined by the following equation:
Equation 15: pjw = Rn
Yn
The adjusted red component, R'D, may be determined by combining Equations 12
- 15 to produce the following equation:
CA 02492870 2000-04-14
77787-26D
-35-
Equation 16 : R'n = PRn R.,= Ys RD For two component selections,
Pes Yn
given the source color components, Rs, Gs, and Bs, and the destination color
components
RD, GD, BD, an adjusted destination component may be determined as follows,
using
adjusted red and blue components, R'p and B'D as examples. The green source
component approximates source luminance, Ys.
Equation 17: Gs = Ys
A source red/luminance ratio pRs and a source blue/luminance ratio pBs is
i0 determined. These ratios may be defmed by the following equations:
Equation 18 : P,a = R.r
Ys
Equation 19 : Pmr = Bs
Ys
The green component of the source color, GD, may approximate luminance, YD
Equation 20: YD = GD
The destination red/luminance ratio pRn and the destination blue/luminance
ratio
PBD are determined. These ratios may be defined by the following equations:
Equation 21: PRD = Ra
Yn
Equation 22: PBa = Bn
Yn
d
29
Equations 17 - 22 may be combined to determine the adjusted red and blue
components Rn and BD as defined by the following equations:
_Pw~Rs=Y,Rn
Equation 23: R'n -
PRS Yv
Equation 24: B'D= ~BD Bs= Yn Bn
BS
CA 02492870 2000-04-14
77787-26D
36
Thus, natural color matching adjusts the values of
t,he selected destination color components as a product of
the ratio of the source and destination luminance of the
selected components and the,value of destination component.
FIG. 16 is a block diagram illustrating an
embodiment data structures used in storing modifications
specified for source media and modification specified for
composition media. In particular, FIG. 16 illustrates
example source and composition color modification data
structures, 138 and 140, respectively. Both the source and
composition modification structures 138 and 140 may include
color modifications specific to a HSL, channels, levels, and
curves color functions and parameters. Parameters may be
considered a subclass of functions as parameters are
essentially simple linear functions or constants. Both the
data structures 138 and 140 may include references to other
data structures specific to the color functions, for
example, HSL modifications data structure 404, channels
modifications data structure 406, levels modifications data
structure 408, and curves modifications data structure 410.
These functions may have been defined through the user
interface 400 described above in connection with Fig. 14,
using the color function buttons 254, 252, 250, and 248 to
access the HSL, channel, levels, and curves screen,
respectively.
In FIG. 16, if there are multiple sub-structures
that are identical in form, then only one of the sub-
structures are illustrated in full detail. Items in a data
structure followed by a"" character indicates a sub-
structure indicated by an arrow to the sub-structure. For
example, in the source color modification data structure
138, the next to "HSL" indicates that there is a sub-
CA 02492870 2000-04-14
77787-26D
37
structure 404 defining HSL color modifications. Items
followed by a"=" character indicates a parameter, with a
category in parentheses. For example, in the channels color
modification data structure 406, the = next to "preview mode
(num)" indicates that there is a numeric parameter defining
the preview mode selected by a user. Items followed by a
"f" indicate'a function. Functions may be stored as fixed
length array of input/output pairs of values. These values
are used to calculate a lookup table that define the
function. For example, in the curves color modification
data structure 410, the f next to "red(rgb)" indicates a
function for RGB component curves. Using functions and
lookup tables to define color modifications is described in
U.S. Patent No. 6,552,731 issued April 22, 2003 and entitled
"Multi-tone Representation of a Digital Image on a Digital
Non-Linear Editing System," by Robert Gonsalves and the
Cacciatore patent. The data structures of FIG. 16 may be
used to define coefficients for a matrix or values of a
lookup table as described in the-Cacciatore patent.
A multimedia composition may be represented and
edited with a typical computer system. The invention is not
limited to any specific computer described herein. Many
other different machines may be used to implement source
color modification. Such a suitable computer system
includes a processing unit which performs a variety of
functions and a manner well-known in the art in response to
instructions provided from an application program. The
processing unit functions according to a program known as
the operating system, of which many types are known in the
art. The steps of an application program are typically
provided in random access memory (RAM) in machine-readable
form because programs are typically stored on a non-volatile
CA 02492870 2000-04-14
77787-26D
37a
memory, such as a hard disk or floppy disk. After a user
selects an application program, it is loaded from the hard
disk to the RAM, and the processing unit proceeds through
the sequence of instructions of the application program.
The computer system also includes a user
input/output (I/O) interface. The user interface typically
includes a display apparatus (not shown), such as a cathode-
ray-tube (CRT) display in an input device (not shown), such
as a keyboard or mouse. A variety of other known input and
output devices may be used, such as speech generation and
recognition units, audio output devices, etc.
The computer system also includes a video and
audio data I/O subsystem. Such a subsystem is well-known in
the art and the present invention is not limited to the
specific subsystem described herein.. The audio portion of
subsystem includes an analog-to-digital (A/D) converter (not
shown), which receives analog audio information and converts
it to digital information. The digital information may be
compressed using known compression systems, for storage on
the hard disk to use at another time. a typical video
portion of subsystem includes a video image
compressor/decompressor (not shown) of which many are known
in the art. Such compressor/decompressors convert analog
video information into compressed digital information. The
compressed digital information may be stored on hard disk
for use at a later time. An example of such a
compressor/decompressor is described in U.S. Patent
No. 5,355,450.
One or more output devices may be connected to a
playback system or editing system. Example output devices
include a cathode ray tube (CRT) display, liquid crystal
CA 02492870 2000-04-14
77787-26D
37b
displays (LCD) and other video outputdevices, printers,
communication devices such as
CA 02492870 2000-04-14
77787-26D
-38-
a =
modem, storage devices such as disk or tape, and audio output. One or more
input
devices may be connected to the editing or playback system. Example input
devices
include a keyboard, keypad, track ball, mouse, pen and tablet, communication
devicz,
and data input devices such as audio and video capture devices and sensors.
The editing
and playback systems are not limited to the particular input or output devices
used in
combination with the computer system or to those described herein.
The editing or playback system may be a general purpose computer system which
is programmable using a computer programming language, such as "C++," JAVA or
other language, such as a scripting language or even assembly language. The
computer
system may also be specially programmed, special purpose hardware. In a
gsnera~-=--
purpose computer system, the processor is typically a commercially available
proeessor,
such as the series x86 and Pentium processors, available from Intel, similar
devices from
AIviD and Cyrix, the 680X0 series microprocessors available from Motorola, and
the
PowerPC microprocessor from IBM. Many other processors are available. Such a
microprocessor executes a program called an operating system, of which
VmdowsNT,
Windows95 or 98, UNIX, Linux, DOS, VMS, MacOS and OS8 are examples, which
controls the execution of other computer programs and provides scheduling,
debugging,
input/output control, accounting, compilation, storage assignment, data
management and
memory management, and communication control and related services. The
processor
and operating system define a computer platform for which application programs
in
high-level prograrnming languages are written.
A memory system typically includes a computer readable and writeable
nonvolatile recording medium, of which a magnetic disk, a flash memory and
tape are
examples. The disk may be removable, known as a floppy disk, or permanent,
known as
a hard drive. A disk has a number of tracks in which signals are stored,
typically in
binary form, i.e., a form interpreted as a sequence of one and zeros. Such
signals may
define an application program to be executed by the microprocessor, or
information
stored on the disk to be processed by the application program. Typically, in
operation,
the processor causes data to be read from the nonvolatile recording medium
into an
integrated circuit memory element, which is typically a volatile, random
acsress memory
such as a dynamic random access memory (DRAM) or static memory (SRAM). The
integrated circuit memory element allows for faster access to the information
by the
processor than does the disk. The processor generally manipulates the data
within the
CA 02492870 2000-04-14
77787-26D
-39-
integrated circuit memory and then copies the data to the disk after
processing is
completed. A variety of mechanisms are known for managing data movement
between
the disk and the integrated circuit memory element, and the invention is not
limited
thereto. The invention is not limited to a particular memory system.
Such a system may be implemented in software or hardware or firmware, or a
combination of the'three. The various elements of the system, either
individually or in
combination may be implemented as a computer program product tangibly embodied
in
a machine-readable storage device for execution by a computer processor.
Various steps
of the process may be performed by a computer processor executing a program
tangibly
1 o embodied on a computer-readable medium to perform functions by operatingon
iIlp[it
and generating output Computer programming languages suitable for implementing
such a system include procedural programrming languages, object-oriented
programming
languages, and combinations of the two.
The playback system or editing system used to implement source or composition
tS modification is not limitai to a partiaular computer platform, particular
processor, or
particular programming language. Additionally, the computer system may be a
multi
proce.4sor computer system or may include multiple computers connected over a
computer network. Each step of FIGS. 9-11, and 13 may be separate modules of a
computer program, or may be separate computer progrmns. Such modules may be
20 operable on separate computcrs.
Having now described some embodiments, it should be apparent to those skilled
in
the art that the foregoing is merely illustrative and not limiting, having
been presented by
way of example only. Numerous modifications and other embodiments are within
the
scope of one of ordinary skill in the art and are contemplated as falling
within the scope
25 of the invention. In particular, although many of the examples presented
herein describe
modifications to video media, numerous audio embodiments are within the scope
of one
of ordinary skill in the art of digital media editing.
