Note: Descriptions are shown in the official language in which they were submitted.
~5~9~3
BAC~GROUND OF THE INVENTION
Field o_the Inven~ti-on
The present invention relates to video production
systems and methods and particularly to such systems and
methods in which the video characte~istics of the video
frame~ may be interactively con~rolled during video production.
DESCRIPTION VF THE PRIOR ART
_ _ .
Video production systems are well known in which
video information stored on conven~ional storage devices,
such as analog video tape recorders, is pro~e~sed in order
to edit or treat the information so as to enhance the pre-
recorded inEormation in the manner desired by the user.
Such prior art video production systems known to applicants,
however, are limited in that the p~erec~rded information ~ :
15 can o~ly be treated on a frame by ~rame basis in which the :~
entire video ~rame ~f infoxmation mu~t be treated. Thus,
for example, i~ it is desired to change the coloration of
a given video frame, the coloration of the entire frame must
be modified. Thus, if it is desired ~o increase the red
con~ained in a particular portion of an image of the frame,
: ~he e~tire frame must experience the re~ color change which
increa~es the c~loration of ea~h of the portions o~ the entire
video image by the inclusion of the additional red color
change.
This has generally prQven unsatisfactory and accor-
dingIy, matte ~echniques have been applied so as to attempt to
limit the coloration to only the portions o the image desired.
These matte t~chniques, however, are also limited, such as
where they are employed to modify ~ideo ima~es in real time ~;
in which a plurality of changes are desired in the frame.
~ 1 --
'~
r~ ,
1~953
In addition, in applying such prior art matte techniques,
as the portion o~ the image which is being treated moves
in real time the matte or mattes employed must correspondingly
move. This creates limitations on this technique when it
~ 5 is desired to process a rapidly changing scene as well as
under other conditions. Accordingly, although such matte
techniques are satisfactory for certain applications, they
have limitations in other areas which have prevented the
wide spread acceptance of such an approach as a viable method
o~ specific color correction or portions of a video frame.
In addition, another lirnitation on such prior art matte
techniques is that i~ the video image being treated has a
complex shape, considerable difficulty occurs in generating
the matte to be used in treating such an image. For the
above reasons, as well as others, prior art matte techniques
have generally not enjoyed wide scale popularity in video
production but rather have been limited to film production
and optical printing.
It has also been known in the prior art to digitize
video information. However, generally these techniques have
not been applied to video production on a wide scale. An
example o~ a digital video production system is the Ampex
SS-2; however, such a prior art digital production system
~ aoes not employ r~al time interaction and, moreover, treats
; 25 the composite video signal as an entity rather than separately
processing its various color video components. This prior
art digital video production system, however, is strictly
a recording and playhack device and does not enable pixel
by pixel interactlon with the video images. Thus ! the limita-
-tions previously described in the prior art are not resolved
.
9 5 3
by this type o~ systemO
Another type of prior art di~ital image processing
system is the type commercially available ~rom Comtal Corpora-
tion of Pasadena, California and distrlb~ted under tha designa-
~ion Vision ONE/20. This system is a real time interac~ivemodular system which provides stand alone ull feature image
processing as well as the option of interfacing to numerous
host computers. However, the prior art Comtal system is
an image analysis system as well as an image enhancement
system and has not been employed in video production. Accor-
dingly such system is not designed to handla real time inter~
active video production. For example, the information processed
is processed in a 1:1 aspect ratio o~ previously digitiæed
images, such as provided ~rom a host computer, although the
Comtal Vision ONE/20 does have a video digitizer option in
which bla~k and white video information may be provided to
the system for p~ocessing in a 1:1 aspect ratio format, such
as for yraphic image treatment. In addition, the Comtal
processor, since it provides processin~ on a 1:1 aspect ratio, .~:
20 treats the digital image on a 256 by 256, 512 by 512, or 1024
by 1024 active pixel basis. Thus, although the commercially
available Comtal ~y~tem is satisfactory for many applications
it is not satisfactory for r~al time interactive video produc
: tion at the video frame ra~e.
In addition to the above, it is well known in the
prior art to modify color video information on a frame by
frame basis, such-as disclosed in U.S. Patent Nos. 4,096,523
or Re 29,787, although these prior art systems do not accomplish
such modification on a pixel by pixel basis. There are also
priox art image analysis systems in which a color video signal
9 ~ 3 .
is digi-tized for image analysis such as disclosed in U.S.
Patent Nos. 3,835,245 and 3,739/078. Other prior art color
video treatment systems known to applicants whlch enable
treatment of a color video signal, although not on an active
_ pixel by pixel basis and not enabling real time interactive
control of such information treatment on a pixel by pixel
basis, are disclosed, by way of example, in U.S. Patent Nos.
4,017,680; 3,617,626; 3,904,816i 4,026,555; ~,000,510; 3,512,094;
4,060,829; 3,997,912; 3,8~9,662; 4,122,489; 3,949,416 and
10 3,969,757.
Thus, although the digitizing of video data for video
image treatment is known in the prior art, as is the color
modification of video information on a frame by frame basis,
and the dot by dot control of static color separation images,
there are no prior art systems known to applicanks in which
real time interactive video production treatment of video
images provided at a real time video frame rate is accomplished
on a active pixel by pixel basis, nor are there any prior
art systems known to applicants in which a single frame of
video may be interacted with in real time on a pixel by pixel
basis. These disadvantages of the prior art are overcome
by the system and method of the present invention.
SUMMARY OF THE INVENTION
An interactive video production system and method
is provided in which an analog video storage means, such
as an analog video tape recorder, capable o~ retrievably
storing a composite analog color video signal r such as a
composite analog color video signal having red, blue and
green analog video components and having an associaked video
information content, retrievably provides this composite
.
~ ~ 3
lalog color video signal as a video output signal at a real
time video frame rate for digital video image processing
in a 4:3 aspect ratio configuration of active pixels on a
real time interactive basis so as to enable the video charac-
- 5 teristics of the active pixels forming each video frame to
be interactively controlled during vidco production. Decoding
means are provided for decoding the composite color video
output of the analog video storage device into red, blue
and green analog video components before provision to the
digital video irnage processor. These decoded red, blue and
green analog video components are digitized and are separately
digitally processed in the previously mentioned 4:3 aspect
: ~ /7~ ratio configuration of active ~pixels with the red, blue
~q and green video components being simultaneously processed.
The real time interactive control of the digital video image
processing may be accomplished by a keyboard, by a data tablet
or by a track ball or by any other conventional interactive
device or combination thereo. The digital video image proces~
sor thereafter provides the digitally processed red, blue
and green video component signals as separate processed analog
signals which are subsequently encoded and combined into
a processed composite color video analo~g signal. It is this
processed composite color video analog signal which is the
end product of the video production process and system.
Tile encoding of the processed signal can occur in a separate
encoding device. Preferably, the decoding is such as to
provide an NTSC decoded output and the encoding is such as
to provide an NTSC encoded output. Moreover, the NTSC encoded
output is preferably a compatible viaeo output signal having
an associated 4:3 aspect ratio picture, such as one comprising
Il 1 11 ~9~3
525 total scan lines at a video frame rate of 30 frames per
second.
The presently preferred system may also include a
digital video storage device, such as an Ampex ESS-2, for
~ 5 retrievably storing single video frames of a composite video
signal for subsequent single processing in the manner described
above with respect to multi-frame processing. The system
also preferably includes a video switcher connected between
the analog and digital storage devices and the encoding and
decoding devices for enabling control of the routing of the
various video signals in the system and, moreover, may provide
a sync reference for the video production system. As used
throughout the description and claims, the term video production
is used in its broadest sense and is meant to include production
15 and post-production as commonly used in the industry. In ,
this regard~ the system may also preferably include a color
vldeo camera for providing live color video information to
the system which may be treated in the same manner as previously
; described with respect to prerecorded information to enable
` 20 real time interactive control of this information on an active
pixel by pixel basis. In order to fac~litate the aforementioned
video produc~ion, a separate annotation command display and
image prQcessing monitor are provided in the presently preferred
system so that during processing the user can view the actual
; 25 image unobscured by any additional information in the picture.
In this manner, the user can view the processed video image
which is being treated and subsequently recorded as it will
subsequently appear during normal video transmission of this
information. Thus, as stated above, an analog compo9ite
color video signal is decoded into its red, blue and green
~ 3
_olor video components, digitized, with each of the color
components being simultaneously processed in a 4:3 aspect
ratio configuration of active pixels, is interacted with
in real time, such as at the video frame rate, during the
- 5 video production process, i5 then converted back into separate
red, blue and green analog components which have now been
digitally processed, is ~hen encoded, and is then recombined
into a composite analog color video signal in which the video
characteristics of the active pixels forming each video frame
have been interactively controlled during video production.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a block diagram of the presently preferred
` system confi~uration of the present invention which may be
employed in practicing the presently preferred method of
the present invention; and
FIG. 2 is a block diagram, partially diagrammatic,
oF the presently preferred digital video image processing
portion of the system of FIG. 1~
DETAILED DESCRIPTION OF _ EFERRED EMBODIMRNTS
Referring no~ to the drawings in detail and initially
to FIGo 1 thereof, the presently pref~erred interactive video
production system of the present invention, generally referred
to by the reference numeral 10, is shown. As shown and preferred,
the system 10 includes a video image processor 12, to be
described in greater detail hereinafter with reference to
FIG. 2. The system 10 also preferably includes a conventional
digital video recorder/reproducer 14, such as commercially
available fxom Ampex under the designation ~mpex ESS-2 Digital
Video Production System, for enabling single frame video
production, as will be described in greater detail hereinafter.
~ ~ ~5~3 `
n addition, the system 10 also includes a conventional analog
video reco.rder/reproducer 16, such as an Ampex VPR-2 Video
Production Recorder, for enabling real time multi-frame ~ideo
_ production at the video frame rate, as will also be described
in greater aetail hereinafter. The system 10 also preferably
includes an audio subsystem, generally referred to by the
reference numeral 18, which maintains the synchronization
between the audio and video portions of the video information
recorded on the analog video recorder/reproducer 16. This
audio subsystem 13, which is preferably a conventional synchro-
nization system, will be described in greater detail hereinafter.
The system 10 also preferably includes an NTSC conven-
-tional decoder, such as a LENCO NTSC Chroma Decoder Model
No. PCD363, generally referred to by reference numeral 20,
for decoding a composite color video signal provided to the
input thereof v.ia path 22 into its analog red, blue and green
color video and sync components which are provided as analog
video output signals via paths 24, 26 and 28, and as sync via
path 96, to the inputs of the digital video image processor
12, as will be described in greater detail hereinafter with
reference to FIG. 2. As is further sh~wn and preferred in
FIG. 1, and as will also be described in greater detail herein-
after with reference to FIG. 2, after the video information
is processed in processor 12, the processed red, blue and
green color video component signals are converted into their
. analog equivalents and provided as processed analog output
signals, and sync, via paths 30, 32, 34 and 33, respectively,
to the input of a conventional NTSC encoder 36l such as a
LENCO NTSC Encoder Model No. CEC810. The ouput of the encoder
36 is a composite analog processed color video signal provided.
. , . .,
. ~ 9 ~ 3
la path 38 to one input of a conventional video switcher
40, such as Duca-Richardson Corporation DRC Series 4000 Video
Switching System. This video switcher 40 also is operatively
connected to the digital video recorder/xeproducer 14, the
_ 5 analog video recorder/reproducer 16 and the NTSC decoder
20 in order to enable the controllable routing of video signals
throughout the interactive video production system 10 of
the present invelltion.
AS i5 also shown and preferred in FIG. 1, the video
switcher 40 is operatively connected to a conventional stan-
dard resolution color video camera 42, such as an Ampex scC-14,
for providing a live composite color video signal input to
the system 10. This live composite color video signal input,
as will be described in greater detail hereinafter, is routed
through the NTSC decoder 20 in the same manner as the previously
described analog video signals retrievably stored in the
analog video recorder/reproducer 16. However, if desired, a
conventional color video camera could be employed which directly
provides red, hlue and green color video component signals
as the outputs thereof in which instance the NTSC decoder
20 may be bypassed and these signals provided directly to
the inputs of the image processor 12 via paths 24, 26 and
28. Such an arrangement is illustrated by the dotted lines
in FI&. 1 with such a conventional color video camera being
given reference numeral 42a.
As further shown and preferred,in FIÇ~ 1, the system
10 also preferably includes a separate high resolution conven-
tional color monitor 44 and a conventional annotation monitor
46, which is preferably a black and white type of conventional
monitor. Lastly, as shown and preferred in FIG~ 1, the system
.,' , ' . :''
_9_
. :
~ 9 ~ 3
lO also preferably includes another conventional color monitor
48, such as one commercially availab:le from Conrac, which
is operatively connected to the video switcher ~0 for monitoring
the video information being routed through the video switcher
40. As will be described in greater de~ail hereinafter with
xeference to FIG. 2, the system lO also preferably includes
real time interactive controls for the video image processor
12, such as a conventional keyboard 50, a conventional track
ball 5~, and a data tablet 54 which is preferably arranged .
to have a 4:3 aspect ratio rather than the more standard
l:l aspect ratio. If desired, other conventional real time
interactive devices may be employed with the video image
~; processor 12 to provide real time interactive control thereof.
In addition, although a keyboard 50, a track ball 52 and
a data tablet 54 are illustrated in FIGS. l and 2, any combina-
tion thereof may be employed in the system lO of the present
invention.
With respeet to the previously referred to audio
subsystem 18, this subsystem 18 preferably includes a conven-
tional configuration of an audio console 56, such as a Trident
Fleximix consolet a cnnventional audio~deck 58, such as an
:: Ampex ATR-lO0 whieh is a two traek audio deck having audio
on one track and SMPTE time code recorded on the other track,
a eonventional sy~ehronizer 60 such as the EECO MQ5-lO0 synchro-
~5 nizer, a conventional edit code generator 62, such as the
. . EECO MTG-550, a conventional video character generator 64, .
; such as an EECO BE-400, and an associated conventional black
and white monitor 66 for displaying the time code information
; if desired~ This eonfiguration, as shown and preferred in
FIG. l, and as previously mentioned, is conventional and
, ' '' ~
. . . ;
'., .
~ .
~ 9 ~ 3
is operatively connected to analog video recorder/reproducer
16.
Referring now to FIG. 2, the video image processor
_ 12 of the present invention is shown in greater detail.
This video;image processor 12 is preferably similar to a
conventional image processor, such as the Comtal Vision ONE/20
image processor, but has been modified in a manner to b~
/ described in greater detail hereinafter to enable its use
( for processing of color video information provided at a viaeo
10\ frame rate which can be interacted with in real time, such
as via the keyboard 50, track ball 52 or data tablet 54,
in a 4:3 aspect ratio. Thus, except as otherwise described,
the various portions of the image processor 12 are preferably
identical with corresponding portions commercially available
from Comtal in the Comtal Vision ONE/20 and will not be described
in greater detail. The image processor 12 is preferably
a RAM refresh memory system which affords user access to
a data base which ultimately preferably provides a 4:3 o~
640 by 488 by 8 bit active pixel confiyuration. As shown
and pre~erred in FIG. 2, this may be achieved by providing
a 1,024 by 512 by 8 bit data base and qppropriately conventionally
modifying this data base by either software or appropriate
- hardware to provide the 640 by 488 by 8 bit pixel confi~uration.
If such a configuration is readily available in memory, then
such configuration may be substituted for the modified 1,024 by
512 by 8 bit data base. As is shown and preferred in FIG.
2, 5uch a 640 by 488 active pixel configuration is provided
( for each of the three color component images, namely the
\ xed, blue and green, which is diagrammatically illustrated
in FIG~ 2 by reference numerals 70, 72 and 74 for the refresh
, . ...
. ,
. ''
. ~ 5 3
nory 76. As also shown and preferred in FIG. 2, the image
processor 12 also enables graphics overlay by providing four
one bit graphics, with these graphics being in the same preferred
4:3 aspect ratio, with each graphics preferably being 640 by
488 by 1 bit. These graphics are diagrammatically represented
in FIG. 2 by reference numeral 78. As used throughout the
specification and claims, it should be noted that the term
~ real $ime refers to operation at the video frame rate. The
5//3/~ ~ L-~ the refresh memory 76 is preferably conventionally
7 handled by three separate input/output video cards 30, 82
and 84, with one such video card being provided for each
of the three red, blue and green color video components.
Each of these video cards 80, 82 and 84 preferably comprises
a digitizer which converts the analog color video component
signals into their digital equivalents for processing by
the image processor 12. The actual processing in the image
processor 12 is preferably handled by a conventional pipe
line processor 86, such as the type provided in the Comtal
Vision One/20 with each of the three colox video component
20~ images, that is the red, blue and green, preferably being
< simultaneously separakely processed in the processor 86 as
three separate 8 bit streams of data. IThis conventional
pipe line processor 86 preferably contains look-up tables, and
function memories which enable treatment of the video images
on a single pixel or group of pixels level in accordance
with the desires of the user~ Since this information may
/ be provided at the video frame rate it~ accordingly, may
also be treated at the video frame rate in real time interaction
dependent on the desires of the user e~pressed through means
30 of the keyboard 50, track ball 52 and/or data tablet 54. Further
,.,
,''' . ' , ,.
. ~
- -12-
5~
~nformation with respect to the functioning and operation
of such a conventional pipe line processor 86 may be obtained
by reference to technical i,nformation provided with the Comtal
Vision ONE/20 image processor. The image processor 12 also
~ 5 preferably includes an overlay control 88 which preferably
: is similar to that commercially available in the Corntal Vision
ONE/20 with the exception that the overlay control 88 has
been conventionally modified to include three frame/write
data paths to enable the aforementioned simultaneous processing
of the red, blue and green color video components with the
overlay control 88, of course, also being modified to effect
the aforementi.oned preferred 4:3 as,pect ratio timing. This
overlay control 88 preferably has n~nerous functions such
as cursor/target definition, pseudo color implementation
and final display priority commands similar to these functional
' controls which are effected in the Comtal Vision ONE/20 by .
the conventional overlay control card provided therein.
The major function of the overlay control 88 is preferably
to develop the proper priority selection for ultimate display
~0 on a per pixel basis. This priority selection must preferably
take place within a single pixel display time and must prefer-
ably be switchable interactively at the video frame rate,
which is preferably 30 times per second, within the retrace
and flyback times of the synchroni~ation system in order
, 25 to avoid any flicker or other noticeable image distortion.
It should be noted that the refresh memory 76 is pxeferably -
controlled by the random access memory control card 90, the ,'
refresh memory control card 92 and the frame/write multiplex
contro~ card 94, with random access memory control card 90
pre~erably being ldentical to that provided in the Comtal
. . . '. , ` ' .
. ,, - .: . ' . ':
-13- ' :`
,
`~ ~15~
~.Lsion ONE/20 as is also ~rue wi.th respect to the refresh
" memory control card 92 except for a modification in the timing
\ circuitry to effect a 4:3 aspect ratio, so as to provide
the correct blanking times associated therewith, and with .,
~ 5 the frame/write mu7tiplex card 94 also preferably being identi
cal to that conventionally available in the Comtal Vision ,
ONE/20 except for corresponding conventional software changes
to accon~nodate the preferred 4:3 aspect ratio timing. In
addition, as shown and preferred in FIG. 2, an external master
sync 96 for controlling thP operation of the processor 12
is provided to the system via an input to the frame/write
multiplex card 94. This master sync signal 96 is prPferably
the sync signal provided from decoding of the composite analog
color video signal provided via path 22 with this sync signal
via path 96 being the sync output of decoder 20. The processed
.red, blue and green 8 bit data streams which are prbvided
as outpwts from the overlay control 88, are preferably provided
to a conventional digital-to-analog converter 100 which is
driven by the system computer bus 102 to provide separate
analog color video component output signals, the red, blue
and green color video component signals, as well as a sync
signal, with these signals being provided via paths 30, 32,
33 and 34. These output signals represent the processed
analog video component signals which are provided as the
: 25 inputs to the conventional encoder 36 for encoding into the
composite processed analog color video,signal which is provided
. via path 38 to the video switcher 40.
: As is also shown and preferred in FIG. 2, a conventional
' / I ~ lar~ ~e~i¢e
~ /7~ lntcr~aeY~3l~r61 card 104, such as conventionally provided
5/l3l3o in the commercially available Comtal Vision ONE/20, is
. , .
. ' , . ~
. , -14-
~ 3
~rovided for interfacing the real time interactive devices
50, 52 and 54 with the image processor 12 through the overlay
control 88. As is also shown and preferred, the overlay
88 through a conventional three 8 bit data stream feedback
- 5 bus 106 to the frame/write multlplex card 94 enables the
actual viewed imayery to be read back into the refresh memory
76 in one frame tirne, such as 1~30 second based on a video
frame rate of 30 frames per second. This feedback permits
iterative processing with the pipe line processor tables
and image combination circuits all updateable at the video
frame rate of 30 times per second. As is true with respect
to the commercially available Comtal Vision ONE/20, the image
:~ processor 12 is preferably controlled by a small system computer
110 t such as a commercially available I-SI-ll, through which
firmware commands are initiated thereby allowing user interac-
tion via the aforementioned keyboard 50, track ball 52 or
data tablet 54. The aforementioned firmware comprises the
. conventional LSI-ll program memory 112 which is provided
: with.the system computer 110. The programming of the system
computer 110 to accomplish its control functions is conven-
: tional and is o~ the type employed in the Comtal Vision ONE/20
system with the exception that the sof~ware is conventionally
modified to accommodate for the presently preferred 640 by ~ ,
488 active pixel configuration and the parallel processiny
of the three separate red, blue and green color video imagecomponents. As is also shown and preferred in FIG. 2, the
annotation ou~put is directly provide~ to the separate black :
and white monitor 46 for enabling separate monitoriny of ~ ;~
the annotation commands on monitor 46 and the processed video
information on monitor 44~ The rocessed analog video output
. ,:
. ' . ,~
-15 ,
~5-59~3
signal may be recorded on the analog video recorder/reproducer
16 for future use, as will be explained in greater detail
hereinafter, and/or may be temporarily stored on the digital
_ video record.er/reproducer 14 if single frame processing is
being accomplished.
Now describing the presently preferred interactive
video production method of the present invention. The presently
preferred video production method of the present invention
shall be described in terms of the presently preferred method .
of providing real time video production at the video frame
rate, such as for image correction which includes color correc-
tion, intensity correction and editing within the frame,
. single frame video production in situations where real time
video production becomes complex, such as where small area
correction of a video frame is desired, and single frame
video production per se, such as for use in animation. These
various exemplary approaches illustrate the full range of
flexibility~of the presently prefexred video production method
and system of the present invention. In addition to the
: 20 above, the presently preferred video production method and
:~ system of the present invention may al~o be used in many
othex areas, such as special effects, rotoscoping of video
~ images which could be accomplished electronically as well
: as many other applications of the presently preferred system
; 25 and method o~ the present invention which will readily occur
. to one of ordinary skill in the art based on the 1exibility
of the presently preferred system and method~ .
Initially, the presently preferred system and method
o the present invention shall be described in terms of real
time video production, that is video production at the video
~ , . ;
-16-
~59~3
,rame ra~e. For example, if it is desired to change a particu-
3 /7~ cc~lo~ ~ ~hC cl~f -hi~3 ,
I lar color component of a scene such as the rlc~h--t~nc-e~r
~3(~q of a person appearing in a scene, this may be accomplished
by first locating the desired portion of the scene in which
~ 5 the color is to be changed by means of the data tablet 54.
; Thereafter the user would request from the image processor
12 the particular assignment value of the color of that portion
which has been located by means of data tablet 54, with this
request being handled via the keyboard 50 input/ and thereafter
the user would input through the keyboard 50 the desired
; color values for that portion of the scene. Under such circum-
stances, when the video tape which ~as contained on the analog
video recorder/reproducer 16 was input through the image
processor 12 at the video frame rate, with such information
being input to processor 12 through the video switcher 40 and
thereafter through thè decoder 20 which would decode the
recorded analog composite video into its red) blue and green
color video co~lponents as well as sync:, wherever that corrected
color or substituted color appeared in a scene the new color
would appear in any portion of the scenes being input at
the video frame rate where the previously identifed color
appeared. Under such circumstances, only the corrected or
substituted color which had been identified would be altered
without any othe~ change on a color basis being made to any
other color in the scene. The same procedure could be employed
for multiple colors since each of the scenes is treated on
a pixel by pixel basis. The user in manipulating or treating
, these colors would preferably employ the color monitor 44
in order to visualize what results are being achieved by
the presently pFeferred video production method during processingO
.' .
.~ . ` ' . ;,'
--17--
li '
~ addition, the aforementioned values with respect to the
color changes would appear on the annotation monitor 46.
The treated video tape information which would be output
from the image processor 12 in terms of the processed red,
_ 5 blue and green color video components via paths 30, 32 and
34 as well as sync via path 30 are then provided to the enc,oder
36 whose output is, in turn, the composite processed analog
, color video signal provided via path 38, which is preferably
," an NTSC compatible video output signal such as one having
~0 525 total scan lines at a video frame rate of 30 frames per
second. This composite processed color video signal provided
via path 38 is then routed by switcher 40 back to an analog
video tape recording device for permanent storage of this
processed video information.
Another example of the type of real time interactive
video production which can be accomplishecl with the presently
preferred system and method of the present invention relates
to the real time combination of di-fferent images from different
sources, again taking advantage of the pixel by pixel treatment
of the system and method of the present: invention. This
technique, as will be described hereinafter, is different
from chroma keying which depends on the use of specific colors
to separate images from the backgroundO No such type of color
separation is re~uired in the presently preferred method
of the present invention~ Thus, for example, the input infor-
mation to the processor 12 can be provided by a plurality
of analog tape decks 16 with each tape deck containing the
image information to be combined. Thus, if it is desired
to combine an airplane with a background scene where the
scene with respect to the airplane has been shot with ~ dlfferent
. , ;
~ -18- '
- , ~ 5 3
background, the scene containing the airplane with background
to be changed is first provided to the image processor 12
: and the background is eliminated through the use of the keyboard
50. The residual image remaining is the airplane alone.
This residual image, which has been output to encoder 36
is then preferably rerecorded on an analog video storage
device 16. The rerecorded residual image is then input to
the image processor 12 through decoder 20 along with the
separately recorded scene of the desired background provided .
from a separate analog tape deck 16. The user then, via
the keyboard 50, indicates to the image processor 12 that
the residual airplane image has priority as the foreground.
This is within the software capability of the commercially
available Comtal Vision ONE/20. The image processor 12 then
combines the two scenes blanking out the portions of the
background which are visually overlaid by the residual image
of the airplaneO This combining function may be visually ~:
observed by the user on the color monitor 44 with the combi.na-
tion of images occurring at the video frame rate as the two
separately recorded video tapes are being simultaneously
input to the image processor 12 at the~video frame rate.
This combined image is, thereafter, output at the video frame
rate to encoder 36 and ther~after through video switcher '
40 to an analog storage device for permanent recording of .
the processed video information. Again, it should be noted
that the aforementioned real time interaction which enables
the electronic combination of previously separately recorded . .
images is accomplished on a pixel by pixel basis. In the
instance of prior art chroma keying, the decision must be
made in advance of the recording of the information, that
~' ~ ':
. ;
-lg- , ,
.~ 3
. ~he decision to combine images; whereas in the presently
preferred method of the present invention no such restriction
is required and any prerecorded scene, irrespective of a
prior decision with respect to chroma content, may be combined.
_ 5 If it is desired to isolate a portion of the frame
so as to eliminate an image from the scene, this may be accom-
plished on a frame by frame basis where only a small area
of the scene is to be treatedO An example of circumstances
under which such treatment is desired is where an unwanted
image inadvertently appears in a scene. Under normal circumstance ,
the entire scene must be retaken at considerable cost~ However,
with the presently preferred method of the present invention
that portion of the scene can be eliminated and the correct
background or other video information substituted therefor.
lS This treatment is accomplished in the Eollowing manner.
The first step in the presently preferred method would be
to transfer that portion of the video tape which contains
the unwanted information to the digital video recorder 14.
The digital video recorder 14, which as was previously mentioned,
is conventional, includes its own associated keyboard. The
user utilizes this kéyboard to retrieve a single frame of
video information from video ~ecorder h4. This single frame
of information is routed through switcher 40, through decoder
20 and is input to the image processor 12 where it is displayed
on the color monitor 44. The data tablet 54 is then employed
to outline the specific area of the frame ~eing displayed
which is to be altered. The keyboard 50 is then employed
to inform the image processor 12 to eliminate the video infor-
mation contained in that designated area. If the unwanted
l ¦image located in the background, the keyboard S0 may also
. ~
20-
~-~ 115~9~
b~ employed to inform the image processor 12 to electronically
extend the background to cover the eliminated area. If it
is desired to substitute another image for the unwanted image
rather than to merely extend the background, then the informa-
tion to be substituted can be separately input to the imageprocessor 12 and electronically conformed to the shape of
the area being eliminated. All of the above three features
are within the software capabilities of the commercially
available Comtal Vlsion ONE/20. The combination of the substi-
tuted image may thereafter be accomplished in the same mannerpreviously described above with respect to the combination
of images. In addition to providing substitution or combination
images to the processor 12 from a separate recording device,
~ this information can also be provided from a color video
; 15 camera 42 or 42a if it is desired to combine or substitute
live information with previously recorded information. This
can be accomplished by temporarily storing the live image
provided from camera 42 or 42a in the video cards 80, 82
and ~4 with the single frame of information being provided
;~ from video recorder 14 being stored in the refresh memory
;~ 76. This is possible due to the presence of the feedback
loop 106 which enables the processor 12 to also function
as a frame grabber.
Another example of the advantages of single frame
treatment in the presently preferred video production method
of the present invention is when it is desired to relocate
an image previously recoraed in a scene to a different portion
of the scene. In order to accomplish this, the image to
be moved is outlined with the data tablet 54 and a new target
location iA the scene is selected with the track ball 52.
.
. ;
-21- ,
9 ~ 3
ne keyboard 50 is then employed to inform the image processor
12 to move the outlined image indicated by data tablet 54
to the new target location indicated by track ball 52. This
is within the software capabilities of the commercially
- 5 available Comtal Vision ONE/20. The single frame ~f information
which has been provided from video recorder 14 and which ,
has now been processed is then output through encoder 36.
and routed through switcher 40 for subsequent rerecording
back onto the video recorder 14. This processed single frame
information which has been treated on a frame by frame basis
from recorder 14 i5 then subsequently rerecorded on an analog
recorder 16 for permanent storage of the processed information.
~ The same type of image relocation technique can be employed
: in creating special effects, such as by separating an image
into its component portions which can be relocated to different
portions of the scene on a frame by frame basis, such as
for example if it is desired to create the visual illusion
: of an explogion. Again, this is faci.litated by the treatment
. of the image on an active pixel by pi.xel basis which enables
; 20 the relocation of the portions of the image on such a pixel
by pixel basis.
As was pre~iously mentioned, another example of the
flexibility of the presently preferred method of the present ~
invention is in connection with animation. In employing :~
such a technique, previously drawn black and white cels may
be input to the image processor 12, such as by means of the
color vi.deo camera 42a, with each of the cels being temporarily
stored in the refresh memory 76. The animator may then outline
the designated axeas to be colored by use of the data tablet
54 or may locate target areas fo~ coloring by means of the
~' . . ~ :
: -~2~ .
'11,._,,... ,~15~g~3
ack ball 52 with the animator observing the cel to be treated
on the color monitor 44. The keyboard 50 is then employed
to select the desired color for the designated area and to
inform the image processor 12 to assign that color to the
_ 5 designated area, as was previously mentioned with respect to
color correction. This technique is accomplished for the
entire scene on a frame by frame basis with each frame then
being output through encoder 36 and video switcher 40 for
recording on the digital video recorder 14 until the entire
process has been completed. Thereafter the processed informa-
tion recorded on digital video recorder 14 is pro~ided to
the analog video recorder 16 at the video frame rate for
permanent storage of the completed color animation. This
processed animated analog video information which has been
recorded on recorder 16 can now be combined with other recorded
analog information either live or prerecorded in the same
manner previously described above with respect to the combina-
tion of images.
Summarizing the presently preferred method of the
present invention, the information to be interactively treated
on a real time basis is retrievably stored as a composite
analog color video signal which is pr ~ ided at the real time
video ~rame rate through decoder 20 to the image processor -
12. The image processor 12 then digitizes the separately
; 25 provided red, blue and green anarog video output signals
and digitally processes these signals in a 4:3 aspect ratio
coniguration of active pixels, such as the aforementioned
640 by 488 by 8 bit configuration. The keyboard 50, track
ball 52 and data tablet 54 are employed to enable the real
time interactive control of this digital processing on an
. ` '
. ;
; -23
.. ~ . . . _ .. .. . . . . . . . . . . .
.
:~5~9~
active pixel by pixel basis. As a result of this interactive
processing, the processed video information is reconverted
to corresponding analog video signals, thereafter encoded
and rerecorded ~or permanent storage. When it is desired
to treat the information on a single frame basis, the informa-
tion is recorded on a digital video recorder and is then
provided to the image processor 12 on a frame by frame basis.
fter the image treatment is completed r th~ information is
then recorded on the analog recorder at the video frame rate.
By utilizing the presently preferred method and system
of the present invention the video characteristics of the
active pixels forming each video frame may be interactively
controlled during video production both at a real time or
video frame rate as well as on a single frame basis, depending
~ lS on the desired application, with all of these applications
: coming within the flexibility of the presently preferred
system and method of the presen~ invention. Numerous changes
and modifications to the previously described method and
system of the present invention may be made without departing
from thP spirit and scope thereof since the aforementioned
system~and method are creative tools w~ich by their very
~; nature will suggest alterations and combinations in the vario~s
techni~ues described above.
~ . _.
What is Clai~ed is: ~
11 1
. ~ . '
. i,
, -24-
