Note: Descriptions are shown in the official language in which they were submitted.
CA 0222722~ 1998-01-16
W097/04348 1 PCT/CA96/00476
AUTO~ATIC FOC~S SYSTEM
Field of the In~ention
The present invention relates to automatic focus
systems, and more particularly to an automatic focus system for
the microscopic ~m; n~tion of tissue or tissue components in
medicine and biology.
Bach~lo~d o~ the Invention
The microscopic ~m; n~tion of tissue or tissue
components is a common and valuable practice in both medicine and
biology. In the art, microscopic ~m; n~tion is used to view
blood smears under magnification to establish the density of
certain types of blood components or the presence of disease.
Such procedures typically rely on the visual appearance of the
tissue which is often enhanced by the use of specialized stains
that bind to certain tissue components, foreign bodies or the
products of cellular processes.
With the advent of computer technology, it has now
become possible to automate many of the m~nllAl ~X~m;n~tion
procedures by digitizing the images and placing them into memory
of a computer for analysis, display and storage. However, the
success of known automated imaging systems critically depends on
the ability of the system to focus its optics on the tissue
components of interest without operator intervention. The
utilization of image processing equipment for the diagnostic
analysis of cellular tissue often requires a detailed ~m;n~tion
of the nuclei within the cells. Specifically, it is the
evaluation of the granulation or the "texture" of the nuclei
together with the accurate representations of the borders of the
nuclei that plays an important role in the final diagnostic
decision.
One method known in the art for enhancing the "edgel'
information comprises applying the Laplacian operator. The
Laplacian operator comprises a mathematical operation represented
by the following expression:
-
CA 0222722~ 1998-01-16
W O 97/04348 PCT/C A96/00476
--2--
~j2 ~j2
( - + - ) I (X, y) (1)
I~X2 ~jy2
where I(x,y) is the continuous, planar distribution of light
intensity that makes up an image.
S~mmarY of the Invention
It has been found that the Laplacian operator can be
adapted to enhance the texture content of digitized images if the
texture is considered to be composed of many edges. In the case
of cells ~mi ned in a diagnostic procedure, the texture of the
nuclei is typically ascribed to the presence of many small, thin
strands of genetic material. Such material can be interpreted
as a system of many edges and thereby lends itself to the
enhancement provided by the Laplacian operation.
In addition to the enhancement of edge/texture
information, it has been found that the Laplacian operator can
be modified to yield a quantitative measure of the texture
content of a digitized image.
Accordingly, the present invention provides an
automated focus system for adjusting the positioning of
magnifying optics utilized for microscopic ~m;n~tion of tissue
or tissue components in medical and biological applications. It
is a feature of the present invention that texture
characteristics of the captured images are enhanced and the
texture characteristics are also utilized in the determination
of optimal focusing.
According to the invention, the Laplacian operator is
modified to provide enhanced edge/texture information and thereby
a quantitative measure of the texture content of a digitized
image. According to the invention, the Laplacian operator is
modified in four respects. Firstly, the absolute value of the
Laplacian operation is used rather than its calculated value.
Secondly, the local value of the Laplacian operation is
calculated and added to a running total. Thirdly, the two-
~imPn~ional Laplacian operator (expression (1) above) is modified
to a one-~im~n~ional Laplacian operator as shown below:
CA 0222722~ 1998-01-16
WO 97/04348 PCT/CA96/00476
--3--
I (X, Yk)
,,
where I(x,yk) represents a single line of video data generated by
the digitizing means of the system. In another aspect, the one-
~;m~n~ional Laplacian operation is generalized for discreet
digitized images to yield a "second difference" equation as shown
below:
zj2
I (X; ,Yk) -- (Ij+l,k -- Ij,k) -- (Ii,k Ij l,k) (3)
~x2
In another aspect of the present invention, expression (3) allows
the calculation of the texture content to be done at high
computing speeds with reduced memory requirements. Fourthly, the
Laplacian operation is modified according to a weighting curve.
The weighting curve can be selected, for example, to give darker
regions of the image greater weight in the running total than
lighter regions. The weighting according to the weighting curve
allows darker regions, i.e. those typically associated with
nuclei, to dominate the texture content calculation so that the
method preferentially favours nuclear texture content.
There~ore, according to one aspect of the preferred
embodiment of the present invention, there is provided an
automated focus system comprising an intelligent controlled
electro-mechanical actuation system for manoeuvring a microscope
objective. The system brings a stained biological material into
optimal focus for image acquisition. The system is operable
without human intervention and utilizes a merit function based
on the "texture" of a dark stained biological material in the
field of view of the microscope objective. The system utilizes
a m~;m; zation procedure using a feedback technique related to
the merit function in order to control the position of the
objective lens. The merit function, in turn, is based on a
series of calculations performed on a set of digitized images
captured at different focal positions. The use of an intelligent
control routine to issue instructions to the motion control
CA 0222722~ 1998-01-16
W097104348 PCT/CA96/00476
.
system allows the device to avoid the usual focus and pitfalls
associated with microscopic image capture.
In a first aspect, the present invention provides an
automatic focus system for focusing a microscope objective for
viewing a specimen located on a carrier, said automatic focus
system comprising: (a) actuator means coupled to said microscope
objective for moving said microscope objective in response to
control signalsi (b) image capture means for capturing images of
the specimen located on said carrieri (c) means for calculating
focus numbers for said images; and (d) a controller for
controlling said actuator means, said controller including means
for generating an optimum focus position from said focus numbers
and having means for issuing control signals to said actuator
means for moving said microscope objective to said optimum focus
position.
In a second aspect, the present invention provides an
automatic focusing method for focusing a microscopic objective
for viewing a specimen located on a carrier, said method
comprising the steps of: (a) capturing a plurality of images of
said specimen; (b) calculating a focus number for each of said
images; (c) determining an optimum focus position for said
microscope objective from said focus numbers; (d) moving said
microscope objective to said optimum focus position.
A preferred embodiment of the present invention will
now be described by way of example, with reference to the
following specification, claims and drawings.
Brief Description of the Drawinqs
Fig. 1 is a block diagram of an automatic focus system
according to this invention;
Fig. 2 is a block diagram of a ~'second difference
calculation according to the invention;
Fig. 3 is a graphical representation of a second
difference weighting function according to the present invention;
Fig. 4 is a flow chart illustrating a method for
calculating the Focus Number according to this invention;
CA 0222722~ 1998-0l-l6
W097/04348 PCT/CA96/00476
Fig. 5 is a ~low chart illustrating a method for
initially focusing a microscope objective according to the
present invention; and
Fig. 6 is a flow chart illustrating a method for
tracking the focus of the microscope objective lens according to
this invention.
Detailed Description of the Preferred Embodiment
Reference is made to Fig. 1, which shows an automatic
focus system according to the present invention and denoted by
10. The automatic focus system 10 according to the present
invention is suitable for integration with an image acquisition
module of known microscopic ~m~ n~tion instruments comprising
a digital camera and an objective lens. The details of such
microscopic P~m;n~tion instruments are within the underst~n~;ng
of those skilled in the art.
As shown in Fig. 1, the automatic ~ocus system 10
comprises a digital camera 12, a focus number calculation module
14, and a controller 16. The digital camera 12 is coupled to a
mechanical actuator 18 which in turn is coupled to an objective
lens 20 of a microscope (not shown).
The mechanical actuator 18 is of known design and
carries the objective lens 20 and translates it vertically under
instructions from the controller 16. In one embodiment, the
mechanical actuator comprises a voice-coil actuator with a long
range (1 mm) movement. The voice-coil actuator 18 may also
include a LVDT (Linear Variable Differential Transformer)
position sensor (not shown in Fig. 1). The LVDT sensor as will
be familiar to those skilled in the art provides precise
positioning information to the controller 16, although such a
position sensor is not required for operation o~ the system 10.
The automatic focus system 10 according to the present
invention includes a texture calculation routine. Preferably,
the texture calculation routine is embedded in electronic
hardware. For the system 10 shown in Fig. 1, the texture
calculation routine resides in the focus number calculation
module 14. As shown in Fig. 1, digitized images 22 generated by
CA 0222722~ l998-0l-l6
W097/04348 PCT/CA96/00476
the digital camera 12 are transferred to the focus number
calculation module 14 for processing. As will be described in
more detail below, the texture calculation routine analyzes the
digitized images produced by the digital camera 12 and generates
a figure, i.e. Focus Number, that measures the texture content
of the dark-stained regions of the image.
The controller 16 preferably also comprises an
electronic hardware module. The primary function of the
controller 16 is to receive and store the texture figures, i.e.
Focus Numbers, generated by the focus number calculation module
14 for different focal positions of the objective lens 20. Using
the Focus Numbers, the controller 16 determines how far and in
what direction the objective lens 20 must be moved in order to
maximize the texture, i.e. focus, of the dark-stained regions in
the field of view. After a decision is generated, the controller
16 issues appropriate motion comm~n~ 26 to the mechanical
actuator 18 for re-positioning of the objective lens 20.
Preferably, the controller 16 includes logic for receiving sensor
and position data 28 from the mechanical actuator 18.
The process steps for analyzing the texture content and
controlling the positioning of the objective lens 20 will now be
described with reference to Figs. 2 to 6. According to the
invention, the automated focusing system 10 is operable in two
modes and the mode of operation depends upon the anticipated
distance to focus for the current position of the objective lens
20. I~ the distance from the current position of the objective
lens 20 is determined to be large, for example, as might be the
case on initial start-up or when a large lateral excursion has
been executed, then the automatic focusing system 10 applies an
"initial focusing protocol". On the other hand, if the distance
from the current position to the optimum focal point is likely
to be small, then the automatic focusing system 10 uses a
"tracking focusing protocol".
The method according to the present invention operates
on the digitized images 22 produced by the digital camera 12.
In known manner, the digital camera 12 digitizes the image of the
stained biological material (located on a specimen plate or
CA 0222722~ l998-0l-l6
W097/04348 PCT/CA96/00476
--7--
slide) captured through the objective lens 20. Fig. 2 depicts
in diayLal,u"atic form a digitized image 50. The digitized image
50 is generated by the digital camera 12 and transferred to the
focus number calculation module 14 for processing. A typical
digitized image will comprise 512 scan lines produced by known
CCD-type imaging cameras. Fig. 2 depicts a partial (i.e. not all
512 lines are shown) digitized image 50 comprising a series of
digital scan lines 52 shown individually as 52a, 52b, 52c and so
on. Each digital scan line 52, in turn, comprises a se~uence of
digital scan values 54 denoted individually as 54a, 54b....54n.
Each scan value 54 corresponds to the binary value of the
digitized pixel as will be understood by one skilled in the art.
The method for providing automatic focusing according
to the present invertion utilizes a one-dimensional Laplacian
operator given by the expression:
~2
I(x,Yk)
~x2
where I(x,yk) represents a single line of video data 52 (Fig. 2)
generated by the digitizing means, i.e. digital camera 12, of the
system 10. According to this aspect o~ the present invention,
the one-~im~n~ional Laplacian operation given in expression (1)
is generalized for discreet digitized images to yield a "second
difference" equation as shown below:
~2
I(xj,Yk) - (Ij+l k - Ij,k) ( Ij,k Ij-l,k) (2)
~x
The "second difference" equation given by expression (2) allows
the calculation of the texture content to be done at high
computing speeds and also reduces the memory requirements for the
focus number calculation module.
In another aspect of the present invention, the
absolute value o~ each "second difference" calculation is
weighted according to a weighting function or curve. A weighting
curve denoted generally by reference 40 is shown in Fig. 3. The
vertical axis of the weighting curve 40 represents a weighting
CA 0222722~ l998-0l-l6
W097/04348 PCT/CA96/00476
--8--
factor "w" and the horizontal axis represents pixel intensity "x"
for each pixel in the scan line 52. The weighting curve 40
depicted in Fig. 3 has been selected to give darker regions of
the image 50 greater weight in the running total than lighter
regions. In other words, the weighting according to the
weighting curve 40 allows darker regions, i.e. those typically
associated with nuclei, to dominate the texture content
calculation so that the method preferentially favours nuclear
texture content.
Reference is next made to Fig. 4 which shows in flow-
chart form a method for calculating the Focus Number according
to the present invention. The method denoted generally by 100
operates on a digitized grey-level image 50 (as depicted in Fig.
2) which is generated by the digital camera 12. The method 100
is preferably implemented as a routine embedded in electronic
hardware in the focus number calculation module 14. In the
preferred embodiment, a series of three digitized images are
captured by the digital camera 12 with each image at a different
focal position. In addition, a fourth image designated a "test
image" or 'Iconfirmation image" is also utilized.
The method 100 calculates a Focus Number for each of
the images where each Focus Number provides a measure of the
quantity of "texture" in the image and utilizing the weighting
function 40 of Fig. 3 particular attention lS given to the dark-
stained regions. The darker regions are of interest because they
are generally associated with the objects of diagnostic interest
such as nuclei in the biological material However, it will be
understood that the weighting of the Focus Number or texture
calculation can be altered to accommodate alternative staining
schemes.
Referring to Fig. 4, the first step in the focus number
calculation routine 100 comprises a decision block 102 which
checks if the last line 52, e.g. line 512 in 512 line image, in
the image 50 has been reached. If yes, the routine 100 stops or
returns (block 104) If the last line 52 (i.e. line 512 of the
image 50) has not been reached, the routine 100 determines
(decision block 106) if a new line, e.g. line 52k (Fig. 2), in
,
CA 0222722~ 1998-01-16
W O 97/04348 PCT/CA96/00476
the image 50 is being processed. If a new line is being
processed, the routine 100 inputs the first and second pixels
54a, 54b (Fig. 2) as shown in block 108. If the processing of
the line 52k is already in progress, the routine 100 inputs the
next pixel, e.g. pixel 54h in the image line 52 shown in Fig. 2.
Re~erring to Fig. 4, the routine 100 in block 112
executes the "second difference" calculation according to the
"second difference" equation given above in expression (2). In
block 114, the routine 100 takes the absolute value of the second
difference calculation. Next, the routine 100 performs the
weighting assignment in block 116. In this step, the weighting
function 40 (Fig. 3) is applied to the absolute value so that the
value is weighted inversely with respect to the centre pixel's
value. In block 118, the routine 100 adds the weighted value
determined in step 116 to a running total. The running total
represents the Focus ~umber for the image under consideration.
The routine 100 then moves to step 120 where the pixel
values are updated. As will be understood by one skilled in the
art, the routine 100 utilizes four memory locations for
processing. Three memory locations store the active picture
element or pixel values, i.e pixel I,=~ k, and Ij+1 k in the
"second difference" equation. The fourth memory location stores
the running total value (updated in step 118). In step 120,
pixel 2 (i.e. Ij~) becomes new pixel l (i.e. Ijlk) and pixel 3
(i.e. pixel I,+1 6) becomes new pixel 2 (i.e. Ijk). New pixel 3
is updated in step 110 as the next pixel in the line 52k
(described above). The steps of the routine 100 are repeated
until all of the lines, e.g. 512, in the image 50 have been
processed. At completion of the processing (step 104), the
running total value calculated for the image represents the Focus
Number or texture quantity for the image.
Both the initial focusing routine and the tracking
focusing routine utilize the above described focus number
calculation routine.
Reference is made to Fig. 5 which shows in flow-chart
form the initial focusing routine 200. In operation, the initial
focusing routine 200 utilizes a series of three digitized images
CA 0222722~ l998-0l-l6
W097/04348 PCT/CA96/00476
--10--
taken at different focal positions (i.e. vertical translation
steps for the objective lens 20). The Focus Numbers are
calculated for the images and the controller 16 analyzes the
Focus Numbers as follows. First, the direction of motion for the
next series of three images is determined from the direction of
increasing Focus Number. Second, if the Focus Numbers show a
middle figure bracketed by two lower values, then a test is
performed to ensure that a local maximum has been found. Third,
the step size of the image series is gradually decreased to
efficiently determine the m~imllm and also avoid first-surface
reflections from the cover slip for the microscope slide. Once
the m~imllm Focus Number has been determined, the routine 200
takes the four Focus Numbers corresponding to the three captured
images and the test or confirmation image, and fits the data to
a quadratic polynomial function. The precise position of the
m~;mllm is established from the maximum in the polynomial and the
controller 16 moves the objective lens 20 to this position which
represents the optimal focus for the image.
Referring back to Fig. 5, at step 202 the lens 20 is
advanced one step and a digitized image is captured by the
digital camera 12. A suitable step size at this stage in the
routine is 50 microns. The image is stored in memory and the
focus num-ber calculation routine 100 (Fig. 4) is called in step
204 to calculate the Focus Number for the image. In step 206,
the Focus Number is compared to a m~imllm value. If the Focus
Num.ber is yreater than the maximum value, the Focus Number is
stored as the maximum Focus Number in block 208. The routine 200
next determines in step 210 if the range limit has been reached.
The range limit 210 keeps count of the three images and
additional confirmation image. If the range limit has not been
reached, the lens 20 is advanced one step (block 202) and the
sequence is repeated i.e. another image is captured at the next
step and the focus number is calculated. Once the three images
have been captured, a test or confirmation image is taken to
verify that previously determined maximum Focus Number is not due
to noise.
CA 0222722~ l998-0l-l6
w097/04348 PCT/CA96/00476
--11--
A~ter the m~; mnm Focus Number is determined, the
initial focusing routine 200 proceeds to step 212. At step 212,
the lens 20 is moved to the m~imllm Focus Number position. Next,
the focusing routine 200 goes back one step (block 214) and then
reduces the step size (e.g to 25 microns) for the image series
(block 216). With the step size reduced, the routine 200
advances the lens 20 by one step in block 218. The Focus Number
is calculated for the image at block 220 by calling the Focus
Number calculation routine 100 (Fig. 4). The Focus Number
calculated at step 220 is compared to the last Focus Number in
decision block 222. If the Focus Number is not greater than the
last Focus Number, the routine 200 proceeds to decision block 224
to check i~ the range limit has been reached. I~ the range limit
is reached (i.e. three images and a test image have been captured
and processed), the routine 200 aborts processing (block 226).
If the range limit is not reached, the routine 200 returns to
step 218 and advances the lens 20 one more step and captures the
next image and calculates the Focus Number.
Referring to Fig. 5, if the Focus Number is greater
than the last Focus Number at step 222, the routine 200 moves to
block 228 and executes a vertical translation step for the lens
20 and an image is captured at this new position. Next at step
230, the Focus Numbers for the image is calculated. The routine
200 then determines at decision block 232 if the calculated Focus
Number is the m~imllm If yes, then the routine 200 checks if
the range limit has been reached at block 224 (i.e. three images
and a test image have been processed), and if the range limit has
not been reached, steps 218 to 222 are repeated as described
above. If the Focus Number (determined at step 230) is not
greater than the last Focus Number, i.e. the m~ximl~m has not been
reached, the routine 200 moves to decision block 234 to determine
if the step size can still be reduced. I~ the mi ni mllm step size
has not been reached, the routine 200 goes back one vertical step
at block 214, and steps 216 through 222 are repeated as described
above.
On the other hand, if the minimllm step size (e.g. 10
microns) has been reached as determined in step 234, the search
CA 0222722~ l998-0l-l6 -
W O 97/04348 PCT/C A96/00476
, -12 -
for m~;mllm Focus Number, i.e. texture, is complete and the
routine 200 turns to determining the precise position of the
texture m~iml~m. At step 236, the initial focusing routine 200
takes the Focus Numbers for the three images and the additional
test or confirmation image and fits a quadratic polynomial to the
data. The routine 200 then establishes the position of the Focus
Number m~;mllm from the m~imllm in the polynomial at step 23 8.
One skilled in the art will understand the implementation of the
quadratic polynomial. In response to the position determined by
the focusing routine 200, the controller 16 issues comm~n~ to
the actuator 18 to move the objective lens 20 to the position
which is the optimal focus for the lmage ~block 240). The
routine 200 uses sensor information 28 from the actuator 18 to
verify the position of the objective lens 20 (block 242). If the
positioning of the ob~ective lens 20 is correct, the initial
focusing routine 200 is complete (block 244), otherwise the
position of the lens 20 is adjusted through the actuator 18
(block 240).
Reference is next made to Fig 6 which shows the
tracking focusing routine 300. The tracking focusing routine 3 00
is appropriately used after the system 10 has executed a small
lateral move in which the anticipated focal position is not very
far from the current position of the lens 20. In this context,
"not far" means that it is likely that a series of three of the
smallest vertical steps will be sufficient to bracket the focal
position and allow the use of a quadratic polynomial fit in order
to find the exact focal position.
In operation, the tracking focusing routine 300
executes three small vertical translation steps and calculates
a Focus Number for each position. The routine 300 then uses
these points to calculate the position of the m~l ml~m in a
theoretical parabola. The parabola describes the variation of
the Focus Numbers with respect to vertical position. The
controller 16 then instructs the mechanical actuator 18 to move
the lens 20 to this position in order to complete the focus
routine.
CA 0222722~ l998-0l-l6
W O 97/04348 PCT/CA96/00476
-13 -
Referring to Fig. 6, the tracking focusing routine 300
first checks if the vertical step size (e.g. 10 microns) is at
the m; nimllm in decision block 302. If the step size is not at
the m;n~mllm, it is reduced (block 304). Once the m;n~mllm step
size is set, the tracking routine 3 00 advances the lens 20 one
step at block 306 and an image is captured, and then calculates
the Focus Numbers for the captured image at the step in block
3 08. The results of the Focus Number calculation are stored at
block 310 and the lens 20 is advanced one more step at block 312
and another image is captured The Focus Number calculation is
repeated for the new vertical step (block 314) and stored in
memory (block 316). The lens 20 is then advanced another
vertical step (block 318) and a third image and confirmation
image are captured. The Focus Number calculations are performed
for the captured images taken at the step (block 320) and stored
in memory (block 322).
As shown in Fig. 6, the tracking focusing routine 300
then uses a quadratic polynomial to fit the Focus Numbers to a
parabola (block 324). The parabola provides a relation between
the Focus Numbers and the vertical position of the objective lens
20. In step 326, the routine 300 attempts to ascertain the
m~imllm focus position from the parabola. If the maximum
position for the optimal focus cannot be ascertained, the
tracking focusing routine 300 reverts to the initial focusing
routine 200 (block 328). If the maximum is ascertainable, the
m~imllm is determined in step 330 and the controller 16 issues
comm~n~ to the actuator 18 to move the lens 20 to the position
for optimal focus. The position of the lens 20 is checked (block
334) and adjusted lf necessary (block 332), otherwise the
tracking ~ocusing routine 300 returns control (block 336) to the
calling routine running for example on a central control computer
(not shown).
If the operation of the tracking focusing routine 300
cannot generate a maximum with the three measurements taken at
each vertical step as described above, then the automatic focus
system 10 automatically reverts to the initial focusing routine
200 described above with reference to Fig. 5.
CA 0222722~ l998-0l-l6
W O 97/04348 PCT/CA96/00476
-14-
It is to be understood that the foregoing description
of the preferred embodiment of this invention is not intended to
be limiting or restricting, and that various rearrangements and
modifications which may become apparent to those skilled in the
art may be resorted to without departing from the scope of the
invention as defined in the appended claims.
