Note: Descriptions are shown in the official language in which they were submitted.
CA 02240343 1998-06-11
CS- q8~ ~ -
IMAGE SIGNAL PROCESSOR HAVING AN ENLARGED CORRECTION RANGE
FOR PICTURE BLURRING CAUSED BY MOVEMENT OF THE HANDS
BACKGROUND OF THE INVENTION
l . Field of the Invention
The invention relates to a video camera and more particularly to an image
signal processor, for use in a video camera, for preventing blurring of resultant television
pictures caused by unintentional movements of the hands or the camera during optical
zooming in. The image signal processor is capable of correcting picture blurring of an
enhanced magnitude caused by a hand movement during an optical zoom-in operation.
10 2. Description of the Prior Art
Unintentional movements of the hands or the camera (hereinafter referred to as
"hand movements") in shooting result in blurring of resultant television pictures
(hereinafter referred to as "picture blurring"). It is especially true when the user is
zooming in because the higher the zoom power becomes, the more the picture blurring
15 increases in magnitude. For this, the present invention contemplates the picture blurring
caused by hand movements in zoom-in shooting.
Various systems which deal with problems of such picture blurring have been
proposed so far.
A basic scheme to correct the picture blurring is to store a field of shot and
20 A/D-converted image in a field memory, to read out, from the field memory, pixel data in
a reduced-size frame at a position moved from the last frame position in the direction of
and by the distance of the picture blurring, and to interpolate the read-out pixel data to
restore the reduced-size frame to an original size. Doing this enables the correction of the
picture blurring as long as both of horizontal and vertical components of the magnitude of
25 the picture blurring are smaller than halves of horizontal and vertical differences,
respectively, between the original size and the reduced-size frame (the latter quantities are
referred to as "allowable dimensions of picture blurring" or "allowable picture blurring
CA 02240343 1998-06-11
dimensions"). Thus, the basic correction scheme is only available to picture blurring
within the allowable picture blurring dimensions, which we call an allowable dimension
problem.
It is noted that the allowable picture blurring dimensions comprise horizontal
5 and vertical components of the allowable magnitude of picture blurring, which is a
function of not only a hand movement but also the zoom m~gnification or power as
described above.
One of the solutions of the allowable dimension problem is to enlarge the
allowable picture blurring dimensions by simply increasing the field memory in size or
10 capacity. In this solution, the added memory will not be used in ordinary shooting
conditions or very low zoom power, becoming less efficient.
Another solution of the allowable dimension problem is disclosed in Japanese
unexamined patent publication No. HeiS- l 30,48 l (l 993) titled "IMAGE PICKUP
DEVICE." In the image pickup device, if the zoom power is not less than the maximum
15 zoom power which is calculated from a rocking angle of the image pickup device caused
by a hand movement and the capacity of the field memory, the zoom-in operation is
disabled so that the resultant picture blurring remains within the allowable picture
blurring dimensions. Thus, since the zoom power is limited by the magnitude of picture
blurring, the zoom power may not be raised to a satisfactory extent due to a hand
20 movement.
Also, U.S. Patent 5,502,484 issued March 26, l 996 discloses a video camera
and a video signal reproducing apparatus which has a CCD (charge coupled device) and
an image memory for PAL (Phase Alternation Line) to provide a hand movement-
corrected output image signal for NTSC (National Television System Committee). A
25 hand movement correction in the vertical direction is mainly achieved by shifting a
reading out position (the readout starting line) of the CCD, and a hand movement
correction in the horizontal direction is mainly achieved by horizontally shifting a fixed-
CA 02240343 1998-06-11
size cut-out range by a correction amount associated by coefficients of a low pass filter
with the way angle of the video camera. Though "by ch~nging the coefficient of the low
pass filter according to the m~gnification, a wide correction range can be obtained", it is
difficult and makes the system complicated to associate the electronic zoom
5 m~gnification with the coefficient such that the image memory is fully utilized. This is
not only because the association has to be achieved experimentally and needs additional
table but also because, as for angular velocity sensors used for the detection of horizontal
and vertical angles, the characteristic may very from sensor to sensor.
It is therefore an object of the invention to provide an image signal processor
10 and an im~ging system which is capable of correcting picture blurring of an enhanced
magnitude caused by a hand movement during an optical zoom-in operation by switching
into an electronic or digital zoom-in operation when the magnitude of picture blurring
exceeds the allowable picture blurring dimensions (or the allowable magnitude of picture
blurring).
SUMMARY OF THE INVENTION
The foregoing object is achieved by an im~ging system such as a video camera
which comprises an optical zoom arrangement, an im~ging device such as an M X N-pixel
CCD for providing an image signal, an analog-to-digital converter for converting the
image signal into a digital image signal, and a picture blurring (PV) compensation &
20 digital zoom unit or an image signal processor.
In the image signal processor, a field of pixels in the digital image signal is
stored in a field memory for M X N pixels. For each field, a test is made to see if the
picture blurring is within a predetermined range, i.e., if a readout frame is included in a
correctable range (or in the M XN pixels). If so, a conventional PV compensation is
25 performed by an address generator reading out pixel data in a readout frame which has a
predetermined size of Mr XNr pixels (Mr<M, Nr<N) and which is at a position shifted
CA 02240343 1998-06-11
from the center of the field by a PV vector and an interpolator interpolating the read out
pixel data into a predetermined output size of Mo X No pixels.
If the picture blurring is not within a predetermined range, a zoom controller
disables the optical zoom-in operation and determines the size (Mr' XNr') of a reduced-
6 size readout frame such that the picture blurring is just fit in the predetermined range. In
response to receiving the size of the reduced-size readout frame, the address generator
reads out pixels in the reduced-size readout frame, in the field memory, which is located
at a position shifted from the center of the field by the PV vector and which has the
reduced size. In response to receiving the size of the reduced-size readout frame, the
10 interpolator interpolates the read out pixels such that the read out pixels of the reduced-
size are m~gnified up to the predetermined output size.
BRIEF DESCRIPTION OF THE DRAWING
Further objects and advantages of the present invention will be apparent from
the following description of the preferred embodiment of the invention as illustrated in
15 the accompanying drawing, in which:
FIG. l is a schematic block diagram showing a part of an image pickup device
such as a video camera which incorporates an image signal processor according to an
illustrative embodiment of the invention;
FIG. 2 is a diagram of the field memory 62;
FIG. 3 is a flowchart showing an exemplary operation of the zoom controller
66;
FIG. 4 is a diagram illustrating a situation where the picture blurring is within
the correctable range;
FIG. 5 is a diagram illustrating how blurring of a picture within the correctable
25 region is corrected;
FIG. 6 is a diagram illustrating a situation where the picture blurring has
CA 02240343 1998-06-11
exceeded the correctable region in the direction BD running vertically between the two
diagonals Dl and D2 of the field memory 62; and
FIG. 7 is a flowchart showing the operation of the readout frame size calculator68 when the calculator 68 has received either DH or Dv.
Throughout the drawing, the same elements when shown in more than one
figure are designated by the same reference numerals.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
FIG. l is a schematic block diagram showing a part of an image pickup device
l such as a video camera which incorporates an illustrative embodiment of an image
10 signal processor 60 for effectuating picture blurring compensation and/or digital zoom in
accordance with the invention. The image pickup device l comprises an optical system
l0 including a lens group (not shown) for use in a zoom operation, an im~ging device 20
comprising a CCD (charge coupled device) or a set of Cads, an analog to digital (A/D)
converter 30 the input of which is coupled to the im~ging device 20. The image pickup
15 device l further comprises a zoom control switch 40 which permits the user to select one
of three positions, i.e., a zoom-in position (denoted as IN) to raise the zoom power, a
zoom-out position (denoted as OUT) to lower the zoom power and an OFF position to
maintain the current zoom power. The image pickup device 1 still further comprises a
optical zoom section which comprises the above-mentioned lens group (not shown)
20 included in the optical system l 0, an actuator 52 such as a motor for mechanically adjust
a lens position to increase and decrease the zoom power, and an actuator driver 54 for
electrically driving the actuator 52. Also, the image pickup device 1 comprises the image
signal processor 60.
The image signal processor 60 comprises a field memory 62 for storing a field
25 of picked-up and A/D converted image signal from the A/D converter 30; a picture
blurring (PV) vector generator 64 having an input thereof coupled to the A/D converter
CA 02240343 1998-06-11
30 for providing a picture blurring vector (or PV vector) for each field; a zoom controller
66 for receiving the PV vector and a zoom control switch 40 output and for providing an
optical zoom control signal to the actuator driver 54 and providing a correction amount
calculated from the PV vector and horizontal and vertical correction margins Mh and Mv
as detailed later; a readout frame size calculator 68 for receiving the correction amount
from the zoom controller 66 and for providing a readout frame size as described in detail
later; an address generator 70 for receiving the PV vector and the readout frame size and
for providing addresses to the field memory 62; and an interpolator 72 which is coupled
to the field memory 62, which receives the readout frame size and which provides an
10 picture blurring compensated image signal as an output of the image signal processor 60.
Referring now to FIG. l, the operation of the image pickup device l will be
described in the following. If the user is shooting an object, then an image of the object is
formed by the optical system l 0 on the surface of the im~ging device 20, which provides
a picked-up image signal. Assuming that the im~ging device comprises an array of M XN
15 im~ging elements, each field of the picked-up image signal comprises M XN pixels, i.e.,
N horizontal lines each comprising M pixels (M and N are any natural numbers). The
picked-up image signal is converted by the A/D converter 30 into a digital image signal,
which is supplied to the image signal processor or a PV compensation and digital zoom
unit 60.
In the image signal processor 60, each of the fields in the digital image signal is
stored in the field memory 62. The size of the field memory 62 is equal to that of the
im~ging element array of the imaging device 20, i.e., M X N. Increasing the im~ging
element array size and the memory 62 size results in an increase in the allowable picture
blurring dimensions (or magnitude) and accordingly results in an increase in an allowable
magnitude of hand movements and/or in a allowable zoom m~gnification.
In order to enable the picture blurring to be corrected when the picture blurring
is within the allowable magnitude (or dimensions), i.e., when hand movements and/or the
CA 02240343 1998-06-11
zoom power is relatively small, the designer of the image signal processor 60 first
determines the size, in pixels, of a readout (or cut-out) frame or window (hereinafter
referred to as "the readout frame size"), Mr X Nr (Mr<M, Nr<N). FIG. 2 is a diagram of
the field memory 62. In FIG. 2, it is assumed that the readout frame 621 is located at the
5 center of the M-by-N pixel block or the field memory 62. The rest 623 of the field
memory (referred to as a correction margin) is used for the picture blurring correction.
Letting the horizontal and the vertical correction margins be 2 X Mh and 2 X Mv, then we
obtain
2XMh = M - Mr, and
2XMv=N-Nr.
The number No of horizontal SC~nning lines and the number Mo of pixels of
each sc~nning line in the output image signal from the image signal processor 60, i.e., the
interpolator 72 are preferably determined according to one of standard television systems.
Thus, assuming that the horizontally-inserted pixel number (i.e., the number of pixels
15 inserted in each of Nr read-out lines for interpolation) and the vertically-inserted pixel
number (i.e., the number of lines inserted among the Nr read-out lines for interpolation)
are Mi and Ni, respectively, then the horizontally- and vertically-inserted pixel numbers
are expressed as Mi = Mo - Mr, and Ni = No - Nr.
It should be noted that the size of the im~ging element array of the im~ging
20 device 20 and the field memory 62, i.e., M and N may be set either equal to or larger than
that of each field of the output image signal (or an output picture size), i.e., Mo and No.
Thus, if the size of the im~ging element array, M X N, is equal to the output picture size
Mo XNo, the hand movement correction involves a digital zoom enlargement, which
cause a certain reduction of the resolution. If the size M XN of the im~gin~; element array
25 is set for such a value as to provide sufficient horizontal and vertical margins Mh and Mr,
the readout frame size Mr X Nr can be set equal to the output picture size. In this case, the
hand movement correction can be achieved without a digital zoom enlargement.
CA 02240343 1998-06-11
Returning now to FIG. l, the digital image signal from the A/D converter 30 is
also supplied to the PV vector generator 64. The PV vector generator 64 calculates
quantities indicative of the direction and the magnitude (i.e., a PV vector B) of the picture
blurring between the current field and the last field. In this illustrative embodiment, the
vector generator 64 calculates as such quantities, for example, horizontal and vertical
elements Bh and Bv of the PV vector B in a well-known manner in the art. Thus, B = (Bh,
Bv) in this example. The vector (Bh, Bv) is passed to the zoom controller 66 and the
address generator 70.
FIG. 3 is a flowchart showing an exemplary operation of the zoom controller 66.
10 In response to the reception of the vector(Bh, Bv), the zoom controller 66 enters the
operation of FIG. 3. The zoom controller 66 first makes a test in steps 300 and 302 to see
if the picture blurring is within the correctable range, i.e., if the current readout frame 62 l
completely included in the field memory 62 as shown in FIG. 4. If so, i.e., if lBhl < Mh in
step 300 and lBvl < Mv in step 302, then the zoom controller 66 sends (as a correction
15 amount) a signal indicative of correctable blurring to the readout frame size calculator in
step 304. Then, the zoom controller 66 passes the output signal from the zoom control
switch 40 as it is to the actuator dirver 54 in step 306, and ends the operation. Thus, in
this case, if the user has kept and is keeping the zoom control switch 40 at the IN position,
the optical zoom operation continues.
Before the description of cases where the picture blurring is not within the
correctable range, we discuss the operation of correcting the picture blurring within the
correctable range. The operation is performed by the address generator 70 and the
interpolator 72.
In response to the reception of the signal indicative of correctable blurring, the
25 readout frame size calculator 68 in turn sends a signal to this effect to the address
generator 70. On receiving the signal, the address generator 70 generates addresses for
use in reading the readout frame which is at the position shifted by the vector B (= (Bh,
CA 02240343 1998-06-11
Bv)) from the center of the field memory 62 and has the predetermined size of Mr X Nr.
Though the position of the readout frame 621 is indicated by the center of the readout
frame 621 in FIG. 4, the position may be indicated by any convenient point such as any
corner of the readout frame 621. The readout of pixel data from the memory 62 isperformed line by line in synchronism with the operation of the interpolator 72. The read
out pixel data from the memory 62 are temporarily stored in two Mo-pixel line memory
in the interpolator 72 and interpolated in a well-known manner such that each field of the
output image signal from the interpolator 72 has a size of Mo X No. Each of Mi pixels
inserted in each of the Nr lines of the readout frame is given as an average of the two
10 adjacent pixels, and each pixel of each of the Ni lines inserted among the Nr lines is given
as an average of the two vertically adjacent pixels. Thus, blurring of a picture within the
correctable region is corrected as shown in FIG. 5.
Returning now to FIG. 3, cases where the picture blurring is not within the
correctable range will be described. In these cases, the correctable region is virtually
15 expanded by reducing the readout frame size and raising the digital zoom power
accordingly. If the readout frame is vertically out of the correctable range, i.e., if the test
result is NO in step 302, then the zoom controller 66 outputs a vertical correction amount
Dv calculated as:
Dv= lBvl - Mv
20 in step 308 and send a zoom OFF signal as a zoom control signal to the actuator driver 54
to stop the optical zoom operation in step 310. If the readout frame is horizontally out of
the correctable range, i.e., if lBhl > Mh in step 300 and lBvl < Mv in step 312, then the
zoom controller 66 outputs a horizontal correction amount DH calculated as:
DH= lBhl - Mh
2~ in step 316 and again send a zoom OFF signal as a zoom control signal to the actuator
driver 54 to stop the optical zoom operation in step 310.
If the readout frame is both horizontally and vertically out of the correctable
CA 02240343 1998-06-11
range, i.e., if the test result is NO in step 312, then the zoom controller 66 determines
dominant one of the horizontal and vertical directions which is the decisive factor to
define the size of the readout frame by seeing if lDv/Dhl S N/M in step 314. If so,
meaning the direction BD of the picture blurring, (denoted with ~ = arctan(Bv/Bh)),
runs horizontally between the two diagonals D 1 and D2 of the field memory 62 (+ c~ =
+arctan(N/M)), then the zoom controller 66 outputs the above-mentioned vertical
correction amount Dv in step 308. Otherwise (shown in FIG. 6), the zoom controller 66
outputs the horizontal correction amount DH in step 316. After step 308 or 316, the
controller 66 performs step 310 and ends the operation of FIG.3.
In response to the reception of a horizontal or a vertical correction amount DH
or Dv, respectively, the readout frame size calculator 68 determines the size of the
reduced readout frame 626 which is defined by a horizontal pixel number Mr' and a
vertical pixel number Nr' by means of a proportional arithmetic as shown in FIG. 6. FIG.
7 is a flowchart showing the operation of the readout frame size calculator 68 when the
15 calculator 68 has received either DH or Dv. In step 702, the calculator 68 makes a test to
see if it received, say, a vertical correction amount Dv. If so, then, in step704, the
calculator 68 calculates Nr' and Mr' as follows:
Nr' = Nr - 2Dv, and
(1)
Mr' = Mr - 2Dv Mr/Nr.
Otherwise, in step 706, the calculator 68 calculates Mr' and Nr' as follows:
Mr' = Mr - 2Dh, and
(2)
Nr' = Nr - 2Dh-Nr/Mr.
25 After step 704 or 706, the calculator 68 outputs the horizontal and vertical pixel numbers
Mr' and Nr', respectively, to the field memory 62 and the interpolator 72.
In response to receiving the above mentioned PB vector B (= (Bh, Bv)) and the
reduced readout frame size Mr' and Nr', the address calculator 70 generates addresses for
CA 02240343 1998-06-11
use in reading a reduced readout frame which is at the position shifted by the vector (Bh,
Bv) from the center of the field memory 62 and has a reduced size of Mr' X Nr'. On the
other hand, in receiving the reduced readout frame size Mr' and Nr', the interpolator 72
calculates the horizontally- and vertically-inserted pixel numbers Mi' and Ni', respectively,
5 by the following equations:
Mi' = Mo - Mr', and
(3)
Ni' = No - Nr'.
Then, the interpolator 72 performs interpolation in cooperation with the address10 generator 70 in the above-mentioned manner by inserting Mi' pixels in each of the Nr'
lines of the readout frame and inserting Ni' lines among the Nr' lines such that each field
of the output image signal from the interpolator 72 has a size of Mo X No pixels. In this
way, a video camera or an image signal processor can correct picture blurring of an
extended magnitude without increasing a field memory in capacity.
The foregoing merely illustrates the principles of the invention. Thus, though
the above-described illustrative embodiment has used a picture blurring vector generator
64 which calculates the PB vector from the current field and the last field, it is possible to
implement the present invention by using a pitch and a yaw angular velocity sensors to
obtain the pitch and the yaw angles. However, in this case, the embodiment requires
20 means for associating the angles with the horizontal and the vertical elements Bh and Bv
of the PV vector, e.g., a reference table. The reference table will have to be created
through experiments. Considering these, it is much preferable to obtain PV vectors
through the calculation of the image signal.
The above illustrative embodiment has used separate hardware for the zoom
26 controller and the readout frame size calculator. However, these functions may be
performed by a single controller including a microprocessor.
Though the above-described image signal processor or PV compensation &
CA 02240343 1998-06-11
digital zoom unit is shown as comprising discrete components, the whole or a part of it is
realized either as a single dedicated integrated circuit or as a part of an integrated circuit.
Also, the zoom controller and/or the readout frame size calculator may be
included in a controller which controls the entire of the video camera.
Many widely different embodiments of the present invention may be
constructed without departing from the spirit and scope of the present invention. It should
be understood that the present invention is not limited to the specific embodiments
described in the specification, except as defined in the appended claims.
12
