Note: Descriptions are shown in the official language in which they were submitted.
CA 02386250 2002-04-02
WO 01/06766 PCT/EP00/06827
METHOD FOR IMAGE DATA PROCESSING AND CAMERA RECORDING SYSTEM
The present invention relates to a method for image data
processing in which object data of an image data set representing
an image foreground is separated from a data set representing an
image background, as well as a camera recording system, which is
particularly suitable for performing a method of this type.
The blue screen method is known from television technology, in
which a surface forming the background of the setting to be
recorded is implemented as a monochrome, particularly blue,
background. The recording of an object in front of this
background is first recorded by a television camera at an image
frequency of, normally, 50 to 60 half-frames per second, while in
parallel to this a background replacing the blue background of
the first image, for example a weather map, is recorded by a
second television camera. The two image sequences recorded in
parallel are superimposed in synchronization on one another in a
mixer and the blue background area is replaced by the image
recorded with the second television camera.
In the blue screen method, two principle methods are
differentiated, the first of which is based on a definition of
the chromaticity (the color representing the background is
defined as transparent), while in the second method, which was
commercially exploited particularly by the firm Ultimatte, the
background is defined via the consistency of the brightness
and/or saturation of the color space of the background.
The advantage of the known methods is that the processing of the
analog image data is relatively simple and occurs in real time.
It is a disadvantage that the typically blue, but also frequently
green, background has to be very well illuminated, since the
color or brightness and/or saturation of the color, respectively,
must be very uniform and of high quality in the image sequence
CA 02386250 2002-04-02
WO 01!06766 PCT/EP00/06827
recorded, which may only be achieved by excellent illumination of
the colored surface representing the background. This
illumination is, however, connected with a very high outlay,
since the typical lighting time for a recording studio is
approximately one week, or often even more. Not considering the
man time connected with this, a large amount of power is consumed
and unpleasantly large amounts of heat are produced for those
working in the vicinity of the recording. Furthermore, it is
disadvantageous that shadows and/or reflections must be avoided,
which makes the lighting even more difficult. Reflections may not
be corrected with the known methods, and shadows only with
difficulty.
However, according to the present invention a method for image
data processing having the features of Claim 1 and a camera
recording system having the features of Claim 8 is suggested.
The present invention is based on the idea that the background is
altered in a defined way at a frequency tuned to the recording
frequency of the recording camera. Tuning is understood here to
mean that the frequency of the background alteration has a known
ratio to the recording frequency. This may be an integer multiple
of the recording frequency, but could also be any desired
multiple. The recording frequency is preferably a multiple of the
playback frequency, for example double the typical television
image frequency, i.e., approximately 100 to 120 half-frames per
second. In principle, the method according to the present
invention functions, as will be described in the following, at
any desired recording frequency up to several thousand images per
second, with the alteration frequency of the background tuned to
this. A requirement is that the content (foreground) of two
images recorded in sequence does not alter or at least alters
only insignificantly, since every second image of the image
sequence is used as a reference image.
2
CA 02386250 2002-04-02
' WO 01/06766 ~ PCT/EP00/06827
The method according to the present invention opens up the
possibility of using digital cameras and of the utilization of
their technical advantages. The recording of an image sequence
at, for example, double frequency with a background changing in a
defined way corresponding to this frequency leads to a sequence
of image pairs, with each two images forming an image pair (at
the typical movement speeds of recorded objects) not or barely
able to be differentiated from one another except for the
background, which alters in a defined way from one recording to
the next. One of the two images of an image pair may thus be used
as a reference image for the respective other image.
Using the advantages of digital recording technology-each
recorded pixel of an image may be directly compared with the
corresponding pixel of the following image-the two images of an
image pair are compared with one another, in that, for example,
each pixel of the reference image is subtracted from the
corresponding pixel of the other image. While, as a result of the
comparison, a value of zero or almost zero results for the
foreground of the object forming the recording if there is no
alteration (non-moving or slowly moving object) and an arbitrary
value not equal to zero results for a slight alteration (rapidly
moving object), the comparison of the background surfaces results
in a predetermined defined value, since the background was
purposely altered in a defined way from the recording of the one
image to the next image. The alteration of the background may be
performed by switching between two states or by gradual
continuous oscillation, for example sinusoid, between the two
states.
In order to now separate the foreground of the object forming the
recording from the background, to "clip it out," all pixels to
which a defined content was assigned during the comparison of the
two images forming an image pair are suppressed and possibly
replaced by the content of an image recorded in parallel (for
example the weather map). This second image may have any desired
3
CA 02386250 2002-04-02
' WO 01/06766 ~ PCT/EP00/06827
source. Instead of the image recorded in parallel as described, a
single image (fixed image) may also be used as a background or,
for example, even a computer-generated image and/or image
sequence.
In a preferred embodiment of the invention, to determine the
assignment of an element of a camera image data set to the object
data set or the background data set, a comparison of the data
sets of camera images recorded at a different instant within a
playback frequency is performed. If the recording frequency is,
for example, three times the later playback frequency, then
during one playback frequency, i.e., within a period
corresponding to the playback frequency, three camera images are
recorded. These three images are compared with one another to
identify any desired element in the camera image as an object or
background. The comparison preferably occurs in pairs. Thus, for
example, the first camera image of this sequence of three images
is first compared with the second image and then with the third
image, and finally the second image is compared with the third
image. In order to design the method more economically, the
comparison may be performed in steps, with sequential comparison
steps only performed if the prior method steps have not yet
provided an unequivocal result, i.e., have not yet allowed
unequivocal assignment of the camera image element as object or
background.
In a particularly preferred embodiment, the comparison is a
subtraction of elements of identical coordinates or regions of
the camera image data sets, with the elements of the camera image
data sets to be compared with one another able to be single
pixels or even pixel regions. A comparison of pixel regions has
been shown to be particularly advantageous, since the method may
then be performed more rapidly. It suggests itself in this case
to first select larger pixel regions, in order possibly to
subdivide these into subregions step-by-step. The method
according to the present invention also opens up the possibility
4
CA 02386250 2002-04-02
WO 01/06766 ~ PCT/EP00/06827
of comparing not only two-dimensional pixel regions with one
another, but also n-dimensional pixel regions.
As described above, the method according to the present invention
may be performed at any desired recording frequency up to several
thousand images per second. The higher the recording frequency
is, the larger the number of images having different background
states (particularly background brightnesses) recorded within one
playback period, which causes the comparison possibilities, i.e.,
the permutations of comparison pairs, to increase significantly.
In this way, a better foundation for the assignment of a camera
image element to the background data set or the object data set
is given.
In addition, the present invention opens up the possibility of
evaluating the reflection behavior of objects in the camera image
at a high recording frequency and gradual background alteration
and drawing conclusions about the refractive index and the
material of the object from this. This, in turn, allows the
adjustment of the relevant data set elements of the object data
set to the new replacement background, which replaces the
original recording background. In contrast to the methods known
from the related art, not only is correction of reflections (and
shadows) possible according to the present invention, but also
their evaluation, in order to allow the recognition of the
affiliation as object or background of an image element or the
adjustment of the image and/or the object to a background used.
In this way, shadows and/or reflections on the object and/or the
background may be recognized according to the present invention
and, depending on the application or the desire of the user,
suppressed or adjusted and processed in another way.
The method according to the present invention is preferably
performed using a neuronal net or other methods of artificial
intelligence (AI). In this way, particularly good recognition and
CA 02386250 2002-04-02
wo 01/06766 pcT/EP00/06827
assignment of the image elements as object, background,
reflection, shadow, etc. is achieved.
Finally, the present invention opens up the possibility of
investigating the camera image data sets not only for an
assignment in the two-dimensional range (object and background
affiliation), but also to obtain further information about the
object and/or the background, such as the speed, among other
things. This in turn allows conclusions about three-dimensional
data of the camera image data set (distance from the camera
etc.), so that the present invention is suitable not only for
two-dimensional, but also for multidimensional image data
processing.
Furthermore, it has been shown to be advantageous to alter the
recording background only partially during the recording.
To perform the method according to the present invention, film-
like or sheet-like elements are preferably used, which may be
optically influenced very rapidly in a targeted way, i.e., their
color and/or brightness and/or saturation may be switched from a
first preset value to a second preset value in a very short time.
For this purpose, electroluminescent or electrochromatic flat
elements are particularly suitable, as they are already known
from the related art.
Further advantages and embodiments of the present invention
result from the description and the attached drawing.
It is obvious that the features previously mentioned and
described in the following are usable not only in the respective
combination indicated, but also in other combinations or alone,
without leaving the framework of the present invention.
6
CA 02386250 2002-04-02
WO 01/06766 ~ PCT/EP00/06827
The present invention is schematically illustrated in the drawing
with reference to an exemplary embodiment and is described in
detail in the following with reference to the drawing.
Fig. 1 shows a block diagram to illustrate the present
invention.
Fig. 2 shows a block diagram to illustrate an evaluation unit
to be used in the present invention.
The central component of the invention is the recording of one or
more objects (objects 1 to m) by one or more cameras (cameras 1
to n), the objects being located in front of a background. The
background is, according to the present invention, influenceable
and/or alterable at a high frequency. The alteration frequency is
tuned to a recording frequency of the recording cameras) in this
case. The alteration of the background particularly occurs in
regard to its color and/or brightness and/or saturation.
The influenceable parameters of the background are thus, besides
the alteration frequency, the color, the brightness, and/or the
saturation. In the camera(s), the influenceable parameter is,
besides the typical camera parameters, the recording frequency,
possibly restricted to a definable image region x*y. These
parameters are controlled by the evaluation unit, with direct
control, particularly for adjustment of the cameras) to the
background, also being possible.
Objects 1 to m included by the background, i.e., surrounded by
the background, are recorded with cameras 1 to n, the recording
frequency of the cameras being a multiple of the later playback
frequency. As a result, Az camera images from n cameras are
provided per time unit z, which are supplied to the evaluation
unit illustrated in more detail in Fig. 2.
7
CA 02386250 2002-04-02
' WO 01/06766 ~ PCT/EP00/06827
The evaluation unit illustrated in Fig. 2 is, for each of the 1
to n cameras, its own camera-specific evaluation unit is
provided, which are each supplied the data of the Az camera
images. In consideration of the additional parameters, as they
were described previously, also supplied to each camera-specific
evaluation unit, in each camera-specific evaluation unit the
objects) is/are separated from a background, in that an
identification and an assignment of individual camera image
elements (pixels or pixel regions) to "object" or "background"
occurs. After the assignment is performed, the image elements
assigned to the background are removed and replaced by image
elements of a replacement background. In simple applications,
this replacement background is a static, unchanging image, for
example the weather map. However, more complex applications are
also possible and are made easier by the present invention, such
as the overlay of separately recorded images or moving images
generated by a computer as a background during studio recordings.
The cycle of the identification was already described above.
The image data produced by the individual camera-specific
evaluation units is then supplied to a central recording-instant-
specific evaluation unit, which includes a buffer for evaluating
multiple time units and is used for synchronization and tuning of
the data supplied by the camera-specific evaluation units. The
usage of multiple cameras allows the generation of image data
which is suitable for playback observable by an observer from
different perspectives as "three-dimensional." A recording and
playback method of this type is, for example, described in German
Patent Application 199 13 853.2.
The data produced by the central recording-instant-specific
evaluation unit is output from the evaluation unit, so that a
single image (frame) is present as a result (cf. Fig. 1), in
which the background and the object are separated from one
another. A typical playback frequency of the frame produced is
approximately 25 to 30 sec-1 (television image frequency).
8
