Note: Descriptions are shown in the official language in which they were submitted.
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
METHOD AND SYSTEM FOR FUSING IMAGES
TECHNICAL FIELD OF THE INVENTION
This invention relates generally to the field of electro-optical systems and
more specifically to a method and system for fusing images.
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
2
BACKGROUND OF THE INVENTION
Image fusion involves combining two or more images produced by two or
more image sensors into one single image. Producing one image that mitigates
the
weak aspects of the individual images while retaining the strong ones is a
complicated
task, often requiring a mainframe computer. Known approaches to image fusion
have
not been able to produce a small, lightweight system that consumes minimal
power.
Known approaches to fusing images require a great deal of computing power.
To illustrate, suppose that two image sensors each have the same pixel
arrangement.
Let Nh be the number of horizontal pixels, and let N" be the number of
vertical pixels,
such that the total number of pixels per sensor is Nh ~ N". Let the frame rate
of the
display be Szd, expressed in Hz. The time td allowed for processing each frame
is
given by:
'Ld = 1 ~ S~d
All processing for each displayed image must be done within this time to keep
the
system operating in real time. To calculate the processing time per pixel,
first
compute the total number of pixels of both sensors. Given that each image
sensor has
the same pixel arrangement, the total number of pixels for both sensors is:
2 ~ Ny,NV
The processing time zp per pixel is given by:
zp = z~ _ 1
2NhN,, S2~ ~ 2NHNv
The processing time tp per pixel is the maximum amount of time allotted per
pixel to
calculate a display pixel from the two corresponding sensor pixels, while
allowing for
real time processing. For example, given an average system where ~d = 30 Hz,
N,, _
640, and NV = 480, the processing time zp per pixel is:
zp = 108.5 ns
For handheld or portable applications, processor speed S2p is limited to about
150 MHz. A maximum of approximately two instructions per cycle is allowed in
current microprocessors and digital signal processors. The time required per
instruction z is given by:
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
3
z~ - -
1 = 3.33 ns
2S2 p ,
The number of instruction cycles allowed for each pixel in real time
processixig is
given by:
z
N; _ -'' = 32 instruction cycles
z;
Thirty-two instruction cycles per pixel is often not a sufficient number of
cycles,
especially considering the fact that a simple "divide, floating point" could
easily
require 10 to 100 instruction cycles to complete. Practical image fusion
systems
generally require over 100 instruction cycles per pixel, and sophisticated
image fusion
algorithms often require over 1,000 instruction cycles per pixel.
Consequently,
current image fusion systems are confined to mainframe computers.
While known approaches have not been applied to handheld or portable
applications, the challenges in the field of image fusion have continued to
increase
with demands for small, lightweight systems that consume minimal power.
Therefore, a need has arisen for a new method and system for fusing images.
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
4
SUMMARY OF THE INVENTION
In accordance with the present invention, a method and system for fusing
images are provided that substantially eliminate or reduce disadvantages . and
problems associated with previously developed systems and methods.
A system for fusing images is disclosed. The system comprises two or more
sensors for generating two or more sets of image data. An information
processor
receives and samples the sets of image data to generate sample data and
computes a
fused image array from the sample data. A display receives the fused image
array and
displays a fused image generated from the fused image array.
A four-step method for fusing images is disclosed. Step one calls for
receiving sets of image data generated by sensors. Step two provides for
sampling the
sets of image data to produce sample data. In step three, the method provides
for
computing a fused image array from the sample data. The last step calls for
displaying a fused image generated from the fused image array.
A four-step method for computing a fused image array is disclosed. Step one
calls for sampling sets of image data generated from sensors to produce sample
data.
Step two provides for determining image fusion metrics from the sample data.
Step
three calls for calculating weighting factors from the image fusion metrics.
The last
step provides for computing a fused image array from the weighting factors,
wherein
the fused image array is used to generate the fused image.
A technical advantage of the present invention is that it computes the fused
image from sampled sensor data. By sampling the sensor data, the invention
reduces
the number of instruction cycles required to compute a fused image. Reducing
the
number of instruction cycles allows for smaller, lightweight image fusion
systems that
consume minimal power.
CA 02404654 2002-09-27
WO 01/84828 PCT/US01/12260
BRIEF DESCRIPTION OF THE DRAWINGS
For a more complete understanding of the present invention and for further
features and advantages, reference is now made to the following description,
taken in
conjunction with the accompanying drawings, in which:
5 FIGURE 1 is a system block diagram of one embodiment of the present
invention;
FIGURE 2 is a flowchart demonstrating one method of fusing images in
accordance with the present invention;
FIGURE 3A illustrates sampling with a fixed array pattern in accordance with
the present invention;
FIGURE 3B illustrates sampling with a varied array pattern in accordance
with the present invention;
FIGURE 3C illustrates sampling randomly in accordance with the present
invention; and
FIGURE 4 illustrates a method of computing weighting factors in accordance
with the present invention.
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
6
DETAILED DESCRIPTION OF THE DRAWINGS
FIGURE 1 is a system block diagram of one embodiment of -the present
invention. In this embodiment, a sensor A 102 and a sensor B 104 detect one or
more
physical objects 106 in order to generate image data to send to an information
processor 108, which fuses the sets of image data to produce a single fused
image to
be displayed by a display 110. The sensor A 102 detects the physical objects
106 and
generates sensor data, which is sent to an amplifier A 112. Amplifier A 112
amplifies
the sensor data and then sends it to an analog-to-digital converter A 114. The
analog-
to-digital converter A 114 converts the analog sensor data to digital data,
and sends
the data to an image buffer A 116 to store the data. The sensor B operates in
a similar
fashion. The sensor B 104 detects the physical objects 106 and sends the data
to
amplifier B 118. The amplifier B 118 sends amplified data to an analog-to-
digital
converter B 120, which sends converted data to an image buffer B 122. A field
programmable gate array 124 receives the data generated by the sensor A 102
and the
sensor B 104. The information processor 108 receives the data from the field
programmable gate array 124. The information processor 108 generates a fused
image from the sets of sensor data, and uses an information processor buffer
126 to
store data while generating the fused image. The information processor 108
sends the
fused image data to a display buffer 128, which stores the data until it is to
be
displayed on the display 110.
FIGURE 2 is a flowchart demonstrating one method of image fusion in
accordance with the present invention. The following steps may be performed
automatically using an information processor 108. The method begins with step
202,
where two or more image sensors generate two or more sets of image data. As
above,
suppose that there are two image sensors, each with the same pixel
arrangement. Let
Ny, be the number of horizontal pixels, and let N,, be the number of vertical
pixels,
such that the total number of pixels per sensor is Nh ~ N,,. The sensors may
comprise, for example, visible light or infrared light image detectors. Assume
that
detectable variations in the proportion of the fused image computed from one
set of
image data and from the other set of image data occur in time ts, where:
zs > 1/S2d.
CA 02404654 2002-09-27
WO 01/84828 PCTNSO1/12260
7
Hence, the computation of the proportion does not need to be calculated at
each
frame. Also, assume that the information required to form a metric that
adjusts the
system to a given wavelength ~, proportion can be derived from fewer than N,,
~Nv
pixels.
The method then proceeds to step 204 where the sets of image data are
sampled to produce sample data. FIGURES 3A, 3B, and 3C illustrate three
methods
of sampling image data in accordance with the present invention. FIGURE 3A
illustrates sampling with a fixed array pattern. The sampled pixels (i, j)
302, 304,
306, 308, 310, 312, 314, 316, and 318 may be described by:
i = pOh, where p = 1, 2 ..., Int
h
and
j = q0,, , where q = 1, 2, ..., Int
v
One possible arrangement is to have 0y, = 2 for the horizontal difference
between one
sampled pixel to the next sampled pixel, and 0" = 2 for the vertical
difference
between one sampled pixel to the next sampled pixel. The groups of pixels 320
and
322, each with 2 sampled pixels 302 and 304, and 308 and 310, respectively,
are
sampling blocks. FIGURE 3B illustrates sampling with a varied array pattern.
FIGURE 3C illustrates random sampling. A sequence of sampling patterns may
also
be used, repeating at any given number of sampling cycles, or never repeating,
as in a
random pattern for each continued sampling cycle.
Referring again to FIGURE 2, in steps 206 to 210, a fused image array is
computed from the sample data. In step 206, image fusion metrics are
calculated
from the sample data. The image fusion metrics are values assigned to the
pixels of
the sample data. These values, for example, may give the relative weight of
the data
from each sensor, such that the data from the sensor that produces the better
image is
given more weight. Or, these values may be used to provide a control for the
production of, for example, a false color image. All the pixels may be
assigned the
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
8
same metric, ~3, or each sample pixel may assigned its own metric, (3;;, where
the
subscript ij designates the pixel in the ith row and jth column.
In step 208, weighting factors a;~, where the subscript ij designates the
pixel in
the ith row and jth column, are calculated from the image fusion metrics. The
weighting factors are values assigned to the pixels of the fused image. The
weighting
factors may be computed by, for example, linear interpolation of the image
fusion
metrics.
FIGURE 4 illustrates a method of computing weighting factors in accordance
with the present invention. For example, suppose that the sample data was
sampled
using a fixed array pattern, where every fifth point 402, 404, 406, and 408 is
sampled
in both the horizontal and vertical direction, that is, 0,, = 0" = O = 5. A
sampling
block 410 comprises to two sampled points 402 and 404. The weighting factors
a,~ of
the first row may be computed in the following manner. First, an incremental
value
for the first row in the horizontal direction 8~,~ is calculated using the
following
formula:
Shl = (~16 - ~~~)~0.
Then, the weighting factors between (3~, and ~3i6 in the horizontal direction
are
calculated using the following formula:
ay=fin +bh~ (1-1)
The weighting factors in the vertical direction between ~3» and ~36~ are
calculated in a
similar manner, using the following equations:
Svl _ (h'61 - ~i~)~~
art=~n+sn (i-1)
Referring again to FIGURE 2, the method then proceeds to step 210, where a
fused image array, which is used to generate a fused image, is computed from
the
weighting factors. An array of weighting factors a;~ generates the following
fused
image array:
v~ (~ = v~ (A~.a~ + v~ (B).(1 _ a~)
where i E { l, ..., Nh}, j E { 1, ..., N"}, the superscripts (c~ denotes
display, (A) denotes
sensor A, and (B) denotes sensor B, and V;~ corresponds to the voltage at
pixel (i,j).
CA 02404654 2002-09-27
WO 01/84828 PCT/USOi/12260
9
The fused image array describes the relative weights of the data from each
sensor.
Weighting factor a;; gives the relative weight of the voltage from sensor A at
pixel
(i,j); weighting factor (1-a~) gives the relative weight of the voltage from
sensor B at
pixel (i,j). This example shows a linear weight; other schemes, however, can
be used.
The method then proceeds to step 212, where the fused image generated from the
fused image array is displayed on a display 110.
By sampling the image data, this embodiment allows for more instruction
cycles to calculate [3~ for each sampled pixel. To calculate the number of
instruction
cycles available for each sampled pixel, first calculate the total number of
instruction
cycles per sampled pixel, and then subtract number of cycles per pixel needed
to
sample the pixels and to compute the fused image metrics and the fused image
array.
For example, assume that data is sampled using fixed array sampling. The total
number of instructions for each sampled pixel is given by:
z,.
z;
where is is the processing time per sampled pixel, which is given by:
1
is =
52~, ~ 2n,,nv
where nh and n" are the number of sampled pixels in the horizontal direction
and in
the vertical direction, respectively. Sampling each sampling block, without
double
counting borders, requires about (0 + 1 )[2(0 - 1 ) + 6] instruction cycles.
Each
sampling block contains two sampled pixels, so each sampled pixel loses 1/2(0
+
1 )[2(0 - 1 ) + 6] instruction cycles per pixel. Computing the fused image
array from
the weighting factors requires approximately four instruction cycles for each
calculation, that is:
4N,, N
n,, nv
Therefore, the time left per pixel for calculating the image fusion metrics
~3~ is:
Nt - z.5 - 1/2 (0 + 1) [2(D-1) + (] - 4NHNV
z. nnn~
CA 02404654 2002-09-27
WO 01/84828 PCT/USO1/12260
Using the values given above: S2d = 30 Hz, N,, = 640, Nv = 480, 0h = w = 5,
n,, = 128,
and n,, = 96, the number of instruction cycles is computed to be:
N; = 269 instruction cycles.
This is a dramatic improvement compared with the 32 cycles allotted in
conventional
5 methods. The extra cycles may be used for more complex calculations of ~3~
or other
features. Moreover, if ~3~ is assumed to be the same for all pixels, even more
additional cycles may be available to determine ~i;~, allowing for a more
sophisticated
manipulation.
Although an embodiment of the invention and its advantages are described in
10 detail, a person skilled in the art could make various alternations,
additions, and
omissions without departing from the spirit and scope of the present invention
as
defined by the appended claims.
