CA 02642913 2008-08-19
ALJTOMTIC DETECTION OF A SURGICAL TOOL ON AN INAGE PROVIDED BY A
MEDICAL IMGING SYSTEM
Field of the invention
The present invention relates to computer-assisted
surgical interventions in which the surgeon performs the inter-
vention with the assistance of images provided by a medical
imaging system. The present invention more specifically relates
to the automatic detection of surgical tools in the images
provided by the medical imaging system.
Discussion of prior art
In a laparoscopic surgical intervention, a patient's
abdominal cavity is inflated by means of carbon dioxide. Small
incisions are made on the abdominal wall and a trocart is intro-
duced into each incision. The number of incisions depends on the
type of envisaged intervention and generally varies from 2 to S.
Each trocart corresponds to a tight hollow tube in which a
surgical tool is likely to slide. An endoscope and surgical
tools are introduced into the abdominal cavity through the
trocarts. The image provided by the endoscope is received by a
camera and displayed on a display screen. During the surgical
intervention, an assistant moves the endoscope according to the
instructions given by the surgeon. By means of the video image
provided by the camera linked to the endoscope, the surgeon
CA 02642913 2008-08-19
2
manipulates the surgical tools in the abdominal cavity according
to the intervention to be performed.
An advantage of a laparoscopic surgical intervention
is that it is only very slightly traumatic for the patient since
the performed incisions are small. However, the surgeon does not
directly see the ends of the surgical tools located in the abdo-
minal cavity and only has the video image provided by the camera
linked to the endoscope to perform the intervention. A laparos-
copic surgical intervention thus requires for the surgeon to be
very experienced.
Positioning systems may be used instead of an assis-
tant to move the endoscope. An example of such a system is the
positioning tool sold by Computer Motion Company under trade
name Aesop or the positioning system sold by Armstrong Health-
care Company under trade name EndoAssist. Such positioning
systems can be controlled by the surgeon by means of a
man/machine interface, for example, a pedal board, a voice
control system, or a control system based on the detection of
the motions of the surgeon's head. As an example, following
voice commands such as "left", "right", "up", "down", the
positioning system may displace the end of the endoscope in the
abdominal wall respectively to the left, to the right, upwards
or downwards.
A disadvantage of such positioning systems is that
only elementary controls corresponding to simple displacements
of the endoscope are generally available. The surgeon must then
provide frequent displacement commands to the endoscope posi-
tioning system, for example to follow the displacements of a
specific surgical tool, while simultaneously handling the
surgical tools, which is a big constraint. It would be desirable
to be able to control the displacements of the endoscope posi-
tioning system with more complex instructions. It would for
example be advantageous for the surgeon to be able to designate
a specific surgical tool, which would cause the automatic
displacement of the endoscope toward the designated tool. The
CA 02642913 2008-08-19
3
surgeon would then no longer have to regularly provide new
displacement commands to the endoscope positioning system and
would be free to fully dedicate himself to the intervention. To
implement such displacement commands, a possibility is to auto-
matically determine the positions of the surgical tools on the
video image provided by the camera linked to the endoscope. The
endoscope positioning system can then be automatically con-
trolled based on the determined positions of the surgical tools
and, for example, on an initial command given by the surgeon.
Wei et al.'s publication, entitled "Real-Time Visual
Servoing for Laparoscopic Surgery. Controlling Robot Motion with
Color Image Segmentation", IEEE Engineering in Medicine and
Biology, pages 40-45, 1997, describes a method for controlling
the displacements of an endoscope positioning system based on
the determination of the position of a surgical tool on a video
image in which the tool bears a tracer dye. The detection of the
tracer dye allows determining the position of the tool on the
video image. A disadvantage of such a method is that it is
necessary to modify the surgical tool to apply a tracer dye the-
reto, which adds a significant cost. Further, the tool detection
is impossible as soon as the associated tracer dye is no longer
visible on the video image.
Summary of the invention
The present invention aims at overcoming all or part
of the previously-mentioned disadvantages.
More specifically, the present invention aims at a
method and a device for detecting surgical tools on images pro-
vided by a medical imaging system which requires no modification
of the surgical tools conventionally used in a surgical inter-
vention.
According to another object of the present invention,
the detection of the surgical tools is performed in real time.
To achieve these objects, the present invention
provides a device for detecting a surgical tool in an image,
said surgical tool going through an incision made in a patient's
CA 02642913 2008-08-19
4
skin or one of his organs, the device comprising means for stor-
ing shape parameters of the surgical tool; means for determining
the position of the projection of the incision in the image
plane; and means for determining points and/or straight lines
characteristic of the projection of the surgical tool on the
image based on the shape parameters and on the position of the
projection of the incision.
According to an embodiment of the present invention,
the device comprises means for acquiring first and second images
of the incision; means for determining a first position of a
point of the projection of the incision on the first image and a
second position of said point on the second image; and means for
determining the position of said point with respect to the
patient from the first and second positions.
According to an embodiment of the present invention,
the surgical tool has an elongated shape, the lateral edges of
the projection of the tool in the image plane corresponding to
straight lines, the means for determining the characteristic
points and/or straight lines being capable of determining a set
of pixels of the image such that, for each pixel in the set of
pixels, the gradient of a function which depends on the pixel
color is greater than a threshold; and selecting each pixel for
which the straight line crossing the pixel and perpendicular to
the gradient runs at the level of the projection of the incision
in the image plane.
According to an embodiment of the present invention,
the means for determining the characteristic points and/or
straight lines is capable of determining the axis of symmetry of
the projection of the surgical tool on the image from the
selected pixels; determining the pixels of the axis of symmetry
belonging to the projection of the tool on the image; and deter-
mining the pixel of the axis of symmetry corresponding to the
end of the tool.
According to an embodiment of the present invention,
the means for determining characteristic points and/or straight
CA 02642913 2008-08-19
lines is capable of determining, from among the selected pixels,
the set of pixels such that, for each pair of pixels in the set
of pixels, the perpendicular bisector of the pair runs at the
level of the projection of the incision in the image plane;
5 classifying the perpendicular bisectors in groups of adjacent
perpendicular bisectors; and determining the axis of symmetry
from the group of perpendicular bisectors containing the largest
number of perpendicular bisectors.
According to an embodiment of the present invention,
the means for determining characteristic points and/or straight
lines is capable of assigning to each pixel of the axis of
symmetry a color class from a first or a second color class;
selecting a first group of adjacent pixels containing the larg-
est number of adjacent pixels of the first color class and a
second group of adjacent pixels containing the largest number of
adjacent pixels of the second color class; and selecting, from
among the first or the second group of pixels, the pixel group
closest to the projection of the incision in the image plane.
The present invention also provides a device for
controlling a positioning system of means for acquiring images
of a patient in whom at least one incision has been made to let
through a tool, the control device comprising means for provid-
ing instructions transmitted by a user; a detection device such
as defined previously; and means for providing control signals
to the positioning system based on the points and/or straight
lines characteristic of the surgical tool provided by the detec-
tion device and on the instructions provided by the means for
providing instructions.
According to an embodiment of the present invention,
the image acquisition means comprises an endoscope and a camera.
The present invention also provides a method for
detecting a surgical tool in an image, said tool going through
an incision made in a patient's skin or in one of his organs,
the method comprising the steps of storing shape parameters of
the surgical tool; determining the position of the projection of
CA 02642913 2008-08-19
6
the incision in the image plane; and determining points and/or
straight lines characteristic of the projection of the surgical
tool on the image based on shape parameters and on the position
of the projection of the incision.
According to an embodiment of the present invention,
the surgical tool has an elongated shape, the lateral edges of
the projection of the tool in the image plane corresponding to
straight lines, the method further comprising the steps of
determining a set of pixels of the image so that, for each pixel
in the set of pixels, the gradient of a function which depends
on the pixel color is greater than a threshold; and selecting
each pixel for which the straight line crossing the pixel and
perpendicular to the gradient runs at the level of the projec-
tion of the incision in the image plane.
Brief description of the drawings
The foregoing and other objects, features, and advan-
tages of the present invention will be discussed in detail in
the following non-limiting description of a specific example in
connection with the accompanying drawings, among which:
Fig. 1 schematically shows an example of a system for
controlling an endoscope positioning system implementing the
surgical tool detection method according to the present inven-
tion;
Fig. 2 schematically shows an embodiment of the posi-
tioning system of Fig. 1;
Fig. 3 illustrates the relation between the position
of a point in space and the projection of this point on the
image plane of a camera;
Fig. 4 shows an example of steps of the method for
detecting surgical tools according to the present invention; and
Fig. 5 schematically shows the concept of the surgical
tool detection method according to the present invention.
Detailed description
For clarity, the same elements have been designated
with the same reference numerals in the different drawings.
CA 02642913 2008-08-19
7
The present invention is based on the fact that for
certain surgical interventions, for example, a laparoscopy, the
introduction of surgical material at the level of a vertebra,
etc., a surgical tool is introduced into the patient's body
through a small incision. The present invention thus provides
detecting such a surgical tool on the images provided by a medi-
cal imaging system, considering that the tool necessarily goes
through the incision, which has a position that can be deter-
mined previously, and that the shape of the tool is known. This
eases the tool detection on the images so that a detection can
be performed automatically and in real time.
Fig. 1 shows an embodiment of a device 5 for control-
ling a system 10 for positioning an endoscope 12, implementing
the surgical tool detection method according to the present
invention. For a laparoscopy, positioning system 10 is placed on
abdomen 14 of a patient whose abdominal cavity has been filled
with gas. Positioning system 10 holds a trocart 15 containing
endoscope 12 and penetrating into the abdominal cavity through
an incision 16. A camera 17 is attached to the end of endoscope
12 within the abdominal cavity. Positioning system 10 is of the
type enabling a displacement of trocart 15 and of endoscope 12
according to a number of degrees of freedom, for example, one
degree of freedom for translation and two degrees of freedom for
rotation.
The images obtained by camera 17 are transmitted to an
image acquisition system 20 capable of displaying video images
on a display screen 22. Acquisition system 20 is connected to an
acquisition board 24 via a connection 26, for example, an S-
VIDEO cable. Acquisition board 24 is connected to a processing
unit 28, for example, a computer, via a connection 29, for exam-
ple, a Fire-Wire cable (IEEE 1394 cable). Acquisition board 24
preprocesses the video images which are transmitted to
processing unit 28. Processing unit 28 is capable, as will be
explained in further detail hereafter, of analyzing the video
images to detect the presence of surgical tools on the images.
CA 02642913 2008-08-19
8
Processing unit 28 is further capable of transmitting
displacement commands to a control box 30 via a connection 32.
Control box 30 is capable of translating the displacement
commands transmitted over connection 32 into control signals for
positioning system 10 and of transmitting the control signals to
positioning system 10 via a connection 34. The surgeon can acti-
vate or deactivate control box 30 via a pedal board 36. Further,
the surgeon can provide instructions to processing unit 28 via a
man/machine interface 38 which may comprise a voice control
system and/or a system for detecting the motions of the
surgeon's head.
Fig. 2 shows a more detailed embodiment of positioning
system 10. It for example is the positioning system described in
publication WO 03/094759 issued to PRAXIM. Endoscope 12 has the
aspect of a cylindrical tube of axis Q of some forty centimeters
long, with a diameter of a few centimeters. Axis S2 for example
corresponds to the optical axis of endoscope 12 and of camera
17. Camera 17 is not shown in Fig. 2. .
Positioning system 10 comprises a substantially planar
base 40 comprising a ring-shaped planar central portion 41,
surrounding incision 16, from which four arms 42 extend. Posi-
tioning system 10 may be attached by straps connected to arms
42. Base 40 may also be glued on the patient's abdomen 14. A
fixed ring 43 is arranged on ring-shaped base 41. Fixed ring 43
is solid with base 41. A mobile ring 44 of axis (Oz) , substan-
tially perpendicular to the plane tangent to' abdomen 14 at the
level of incision 16 is rotatably assembled on fixed ring 43
around axis (Oz). Fixed ring 43 comprises peripheral teeth, not
shown, on its external lateral surface.
A stirrup 46 is pivotally assembled on mobile ring 44
along an axis substantially perpendicular to axis (Oz) and
comprised in the plane tangent to abdomen 14 at the level of
incision 16. Trocart 15 is held on stirrup 46 by a mounting clip
48, for example screwed on stirrup 46, which enables simple and
fast connection and separation of trocart 15 and of stirrup 46.
CA 02642913 2008-08-19
9
Trocart 15 comprises a protrusion 50 located on the side of
mounting clip 48 opposite to central portion 41. Protrusion 50
may correspond to handles, sealing valves, connectors, etc. The
inner diameters of mobile ring 44 and of central ring-shaped
portion 41 are selected to enable retrieval of system 10 during
an intervention without displacing trocart 15, or to enable
retrieval of trocart 15 from the patient's abdomen without
displacing system 10.
A first electric motor 56, intended to rotate mobile
ring 44 with respect to fixed ring 43, is immovably assembled to
mobile ring 44 via a plate 58. First motor 56 is connected to
control box 30, not shown in Fig. 2. A toothed wheel, not visi-
ble in Fig. 2, driven by the shaft of first motor 56, meshes
with the teeth of fixed ring 43. The rotating of the toothed
wheel by first motor 56 causes the rotating of mobile ring 44
with respect to fixed ring 43 around axis (Oz).
A second electric motor 60 is attached to stirrup 46,
on the side of stirrup 46 opposite to mounting clip 48, via a
plate 62. The shaft of second motor 60, not visible in Fig. 2,
is oriented along the swivel axis of stirrup 46. Plate 62
comprises an opening letting through the shaft of second motor
60, not visible in Fig. 2. Second motor 60 is connected to
control box 30. A driving element 64 comprises a circular arc 66
having its ends connected by a rectilinear portion 68 attached
to mobile ring 44. Driving element 64 substantially extends in a
plane perpendicular to the plane containing mobile ring 44. The
axis of circular arc 66 corresponds to the swivel axis of stir-
rup 46. The lateral wall of circular arc 66 opposite to rectili-
near portion 68 comprises teeth (not shown). The shaft of second
motor 60 supports a toothed wheel (not shown) which cooperates
with the teeth of circular arc 66 so that, when the toothed
wheel is rotated by second motor 60, stirrup 46 is pivotally
driven with respect to mobile ring 44.
The free end of endoscope 12 comprises a cylindrical
stop 70 from which a pin 72 projects. A compression ring 74
CA 02642913 2008-08-19
bears at one end on cylindrical stop 70 and at the other end on
protrusion 50 of trocart 15. A third electric motor 76 is
attached to stirrup 46, next to second motor 60, via a plate 78.
Plate 78 comprises an opening 80 letting through the shaft, not
5 visible in Fig. 2, of third motor 76. The shaft of third motor
76 is oriented along the swivel axis of stirrup 46. Third motor
76 is connected to control box 30. A winding shaft 82 is
arranged at the free end of the shaft of third motor 76. A
helical threading (not shown) is formed on the external surface
10 of winding cylinder 82. A cable 84 is connected at its ends to
pin 72 and to cylinder 82 and is wound around cylinder 82. When
third motor 76 rotates cylinder 82, cable 84 winds around
cylinder 82 and brings cylindrical stop 70 closer to trocart 15.
Endoscope 12 then slides in trocart 15 along axis 0 and
compresses spring 74. When third motor 76 is no longer actuated,
spring 74 expands and brings endoscope 12 back to a position of
rest. A supporting arm, oriented by the surgeon before the
beginning of the operation, may be provided to support position-
ing system 10 and avoid for all the weight of positioning system
10 to be applied on the patient.
The previously-described positioning system 10 enables
displacing endoscope 12 according to two degrees of freedom for
rotation and one degree of freedom for translation based on
control signals provided by control box 30 to electric motors
56, 60, 76.
Additional trocarts 90, 91 are arranged at the level
of incisions 92, 93 of small dimensions made in the patient's
abdominal wall 14. Additional trocarts 90, 91 enable introducing
surgical tools 94, 95, partially shown in Fig. 2, into the abdo-
minal cavity.
In the following description, call RO (O,x,y,z) a
frame of reference, for example, orthonormal, having its origin
O for example corresponding to the "fixed point" of positioning
system 10, that is, the intersection point between the rotation
axis of mobile ring 44 and the rotation axis of stirrup 46. Axes
CA 02642913 2008-08-19
11
(Ox) and (Oy) correspond to two perpendicular axes, also perpen-
dicular to axis (Oz) which, as described previously, is
perpendicular to abdominal wall 14 at the level of incision 16.
Reference frame Ro is considered as fixed with respect to the
patient during the surgical intervention. Camera 17 and endos-
cope 12 are, in operation, fixed with respect to each other.
Call C a fixed point with respect to camera 17, located on axis
0, at a distance r from point 0. Angle 0 corresponds to the
angle between axis S2 and axis (Oz) and angle cp corresponds to
the angle.between the projection of axis 0 in plane (Oxy) and
axis (Ox). The coordinates of point C in reference frame RO are
(rsin0coscp, rsin0sin(p, rcos6).
Call RC (C, er , eg , e~O) the spherical reference frame
associated with point C. Reference frame RC is a fixed reference
frame with respect to camera 17. Note (X, Y, Z) the coordinates
of any point P expressed in reference frame RO and (1, m, n) the
coordinates of point P expressed in reference frame RC. Before
the surgical intervention, a step of calibration of positioning
system 10 which comprise determining the transfer array provid-
ing coordinates (1, m, n) of point P from coordinates (X, Y, Z)
and the relation between the displacement commands provided by
processing unit 28 to control box 30 and the variation of para-
meters r, A, and cp. During the surgical intervention, processing
unit 28 is then capable of determining at any time the position
of reference frame RC with respect to reference frame RO based
on the displacement commands provided to control box 30. It
should be clear that reference frames RO and RC could be defined
differently from what has been described previously. The only
condition is to be able to consider reference frame Rp as fixed
with respect to the patient during the intervention, to be able
CA 02642913 2008-08-19
12
to consider reference frame RC as fixed with respect to the
camera during the intervention, and to be able to determine the
position of reference frame RC with respect to reference frame
RO at any time.
In Fig. 3, reference frames Rp and RC have been sche-
matically shown with their origins 0 and C located on optical
axis Q of endoscope 12. Camera 17 is considered as operating
according to a pinhole camera model. Such a camera model is
described, for example, in the work entitled "Three Dimensional
Computer Vision - A Geometric Viewpoint" by Olivier Faugeras,
series: Artificial Intelligence, the MIT Press, Cambridge Massa-
chusetts, ISBN 0-262-06158-9 (chapter 3). Camera 17 can then be
represented by a projection center F and an image plane PI.
Image plane PI corresponds to the plane in which the images
obtained by camera 17 and transmitted to acquisition system 20
are formed. Image plane PI and projection center F are fixed
with respect to reference frame RC. Any point P in space
projects in image plane PI to form a point I. A two-dimensional
reference frame RO' (0', eu, ev) is associated with image plane
PI, where 0' corresponds, for example, to the point of image
plane PI on optical axis 0, and the coordinates of point I in
reference frame Rp' are called (u, v).
Before the surgical intervention, a step of cali-
bration of camera 17 is provided. This comprises defining the
transfer array which provides coordinates (u, v) of point I
expressed in reference frame Rp' based on coordinates (1, m, n)
of point P expressed in reference frame RC. According to the
pinhole model, point I corresponds to the point of intersection
of image plane P and of the straight line crossing point P and
projection center F. The transfer array is obtained from the
coordinates of point F and the equation of image plane PI
expressed in reference frame RC, which depend on the specifica-
tions of the used camera 17. Other camera models may be used.
CA 02642913 2008-08-19
13
After the steps of calibration of positioning system
and of camera 17, processing unit 28 can determine, for a
point P having coordinates (X, Y, Z) in fixed reference frame Ro
which have been determined previously and stored in processing
5 unit 28, coordinates (u, v) of point I, which is the projection
of point P in image plane PI, whatever the position of camera 17
in reference frame Ro.
Fig. 4 illustrates the steps of an example of a method
according to the present invention for detecting surgical -tools
10 on an image provided by camera 17, implemented by processing
unit 28.
The surgeon is assumed to have made incisions 16, 92,
93 in the patient's abdominal wall 14, to have placed trocarts
15, 90, 91 in the corresponding incisions, to have introduced
endoscope 12 into trocart 15 and to have arranged positioning
system 10 on the patient's abdominal wall 14. Further, the steps
of calibration of positioning system 10 and of camera 17 have
been completed. Moreover, an identifier, for example, a number,
has been stored in processing unit 28 for each incision-tool
pair. Finally, the radius of each tool 94, 95, assumed to be
cylindrical, is stored in processing unit 28.
At step 100, the positions of incisions 92, 93 in
reference frame Ro are determined. For each incision 92, 93, the
position of a point located substantially at the "center" of the
incision, or incision point, is determined in reference frame
Ro. Determining the incision points may be done by any method.
As an example, for each incision 92, 93, the surgeon may order
the displacement of camera 17 to obtain two images of incision
92, 93 at different positions of camera 17. For each obtained
image, the surgeon may indicate to processing unit 28 the posi-
tion of the projection of the insertion point on the image
displayed on screen 22 via interface 38, that is, indicate the
position of the projection of the insertion point in reference
frame Ro'. As an example, the surgeon may displace a tag on the
image displayed on screen 22 to the position of the projection
CA 02642913 2008-08-19
14
of the insertion point. Based on the positions of the projec-
tions of the insertion point on two different images, processing
unit 28 may determine the position of the insertion point in
reference frame Ro. This may be done according to the method
described in the publication entitled "Reconstruction d'une
arborescence spatiale a partir d'un nombre minimal de projec-
tions : application a l'angiographie numerisee des structures
vasculaires", thesis by Eric Coste for his doctorate in produc-
tics, automatic control engineering, and industrial data
processing of the Lille University of Science and Technologies
(1996), especially chapter 1, section III, and appendix 4. Step
100 is repeated for each incision 92, 93. The previously-
described method for determining the insertion points is partic-
ularly advantageous since it requires no additional instruments
with respect to those conventionally used during the surgical
intervention. According to another insertion point determination
method, the position of each insertion point in reference frame
RO may be obtained via a feeler having its position in reference
frame Rp provided by a tracking system. The position of each
insertion point in reference frame Rp is stored in processing
unit 28. The actual surgical intervention then starts and the
method carries on at step 102.
At step 102, processing unit 28 receives, via acquisi-
tion system 24, an image provided by camera 17.
Fig. 5 schematically shows an image 106 such as
displayed on screen 22. Continuous line 106 corresponds to the
physical limit of the image provided by camera 17. Image 106
thus corresponds to a portion of image plane PI of camera 17.
Each image is formed of a pixel array. As an example, a pixel
Pix has been shown in the form of a square portion of image 106.
The method carries on at step 104.
At step 104, processing unit 28 determines the posi-
tion, in image plane PI, of the projection of the insertion
point of the searched tool or the projections of the insertion
points of the searched tools. In Fig. 5, points 01 and 02
CA 02642913 2008-08-19
correspond to the projections, in plane PI, of the insertion
points associated with incisions 92, 93 and will be called
projected insertion points hereafter. In the present example,
projected insertion points 01 and 02 are outside of the image
5 provided by camera 17, and incisions 92, 93 are not visible on
image 106. For each projected insertion point 01, 02, processing
unit 28 determines a circle C1, C2 having its center correspond-
ing to point 01, 02, and having its radius R1, R2 depending on
the considered point 01, 02. As an example, radius R1, R2
10 substantially corresponds to the apparent radius of tool 94, 95
associated with the considered projected insertion point if the
tool were parallel to image plane PI. The method carries on at
step 108.
At step 108, processing unit 28 determines for each
15 pixel of image 106 whether the pixel is likely to belong to the
contour of the projection of a surgical tool 94, 95. For this
purpose, the present invention is based on the fact that each
surgical tool 94, 95 necessarily involves an incision 92, 93 and
that the shape of each surgical tool 94, 95 is known. In the
present embodiment, each cylindrical tool 94, 95 is considered
as having an elongated cylindrical shape.
Processing unit 28 determines the contours present in
the image by segmentation, according to a gradient method. The
gradient method for images in grey levels is described in the
work entitled: "Analyse d'images : filtrage et segmentation",
which is a collective work coordinated by J.-P. Cocquerez and S.
Phillip, with authors Ph. Bolon, J.-M. Chassery, J.-P. Cocque-
rez, D. Demigny, C. Graffigne, A. Montanvert, S. Phillip, R.
Zeboudj, J. Zerubia, published by Masson, 1995, ISBN: 2-225-
84923-4. For an image in grey levels, calling F the function
which puts in correspondence with each pixel of position (u, v)
the grey level of the pixel, a gradient vector Grad(F(u,v)) which
is normal to the contour at position (u,v) (pixel position) and
which has a norm which is all the higher as the contour is
"sharp", is obtained for each image pixel. For a color image, it
CA 02642913 2008-08-19
16
is possible to break down the color image into three images in
grey levels (an image R for red, an image G for green, and an
image B for blue). For each pixel of coordinates (u,v), three
gradient vectors Grad (R(u, v)) , Grad (G(u, v)) , Grad (B(u, v)) , respec-
tively associated with image R, with image G, and with image B,
can then be defined. The pixel gradient Grad(F(u,v)) (or Grad(F) )
is then defined as being the maximum of the three gradient
vectors Grad(R(u,v)), Grad(G(u,v)), Grad(B(u,v)). Processing unit 28
only considers the pixels of image 106 for which the amplitude
of gradient Grad(F) is greater than a determined threshold. Such
pixels are likely to correspond to a transition between two
objects of image 106, for example, organs, tools, etc. As an
example, the gradient vector Grad(F) associated with pixel Pix
has been shown in Fig. 5.
For such pixels, processing unit 28 determines whether
the straight line crossing the pixels and perpendicular to
gradient Grad(F) cuts one of circles C1, C2. If such a condition
is fulfilled, the pixel is likely to belong to the edge of the
projection of one of surgical tools 94, 95 and is called poten-
tial pixel hereafter. The previous condition translates as the
fact that the searched surgical tool 94, 95 has a cylindrical
shape, that is, the edges of the projection of the tool on image
106 must correspond to straight lines, and that surgical tool
94, 95 goes through an incision 92, 93, that is, the edges of
the projection of the tool must run through a projected inser-
tion point 01, 02. In the example of Fig. 5, straight line D
perpendicular to gradient Grad(F) of pixel Pix cuts circle C2.
Processing unit 28 deduces therefrom, at this step of the
method, that pixel Pix perhaps belongs to the edge of the
projection of the surgical tool associated with projected inser-
tion point 02. A set of potential pixels is finally obtained.
Unit 28 then implements various processings to decrease the
number of potential pixels. As an example, the potential pixels
which are isolated from other potential pixels and which
obviously do not belong to the edges of the projection of a
CA 02642913 2008-08-19
17
surgical tool can be eliminated. At the end of step 108,
relatively dense sets of potential pixels, called candidate
pixels, each roughly having a rectilinear elongated shape, are
obtained. The method carries on at step 110.
At step 110, processing unit 28 determined the axis of
symmetry M1, M2 of the projection of each tool 94, 95 on image
106. Indeed, since each tool is cylindrical, the edges of the
projection of the tool correspond to two straight lines D1, D1',
D2, D2', which thus have an axis of symmetry. It should be noted
that axis of symmetry M1, M2 of the projection of a tool foes
not correspond, except in specific cases, to the projection of
the axis of symmetry of the tool since the central projection
does not keep the proportions between distances. The axis of
symmetry M1, M2 of the projection of a tool thus does not neces-
sarily cross the projected insertion point 01, 02 associated
with the tool. Processing unit 28 determines, for each pair of
candidate pixels, whether the perpendicular bisector associated
with the pair of candidate pixels cuts one of circles Cl, C2. If
so, this means that the perpendicular bisector, called candidate
perpendicular bisector, is likely to correspond to an axis of
symmetry. Processing unit 28 then implements a sorting method
based on the Hough method described, for example, in the work
entitled "Use of the Hough Transformation To Detect Lines and
Curves in Pictures" by Richard 0. Duda and Peter E. Hart,
Communications of the ACM, Vol. 15(1), pp. 11-15, 1972. Each
candidate perpendicular bisector associated with a projected
insertion point, for example, point 01, is defined by a couple
of parameters (p,(x), where p is the distance separating the
candidate perpendicular bisector from projected insertion point
01 and a is the angle between the candidate perpendicular bisec-
tor and a privileged direction, for example, straight line (01,
eu). For a given candidate perpendicular bisector, p can vary
within interval [0, R1] and a can vary within interval [0, 2n].
Interval [0, R1] is divided into adjacent sub-intervals [ai,
ai+l], where i is an integer varying between 1 and N and where
CA 02642913 2008-08-19
18
a1=0, aN+1=Rl and ai<ai+l. Similarly, interval [0, 2n] is
divided into adjacent sub-intervals [bj, bj+l], where j is an
integer varying between 1 and M, and where b1=0, bM+l=27t and
bj<bj+l. An array T with N lines and M columns having its line
of index i associated with sub-interval [ai, ai+l] and its
column of index j' associated with sub-interval [bj, bj+l] is
then defined. Each element Ti,j of the array corresponds to a
counter initially set to 0. For each candidate bisector, counter
Ti,j associated with sub-intervals [ai, ai+l] and [bj, bj+l]
which contain the parameters (p,(x) associated with said candi-
date bisector is incremented. When all candidate perpendicular
bisectors have been considered, processing unit 28 determines
the indexes iMAX and jMAX of the highest counter TiMAX,jMAX. The
axis of symmetry of the surgical tool associated with projected
insertion point 01 can then correspond to the straight line
defined by p=(aiMAX+aiMAX+1)/2 and a-(bjMAX+bjMAX+1)/2.
Processing unit 28 also stores the pixel pairs which
have taken part in the incrementation of each counter Ti,j.
Straight lines Dl, Dl' corresponding to the edges of the projec-
tion of the surgical tool associated with projected insertion
point 01 are defined based on the pixel pairs associated with
counter TiMAX,jMAX. To increase the number of pixels, processing
unit 28 may add, for each pixel associated with counter
TiMAX,jMAX, a new pixel corresponding to the symmetrical of the
pixel with respect to the axis of symmetry. The first edge of
the tool projection can then correspond to straight line Dl
obtained by linear regression from the pixels located on one
side of the axis of symmetry and the second edge of the tool
projection can then correspond to straight line Dl' obtained by
linear regression from the pixels located on the other side of
the axis of symmetry. The method is repeated for all the tools
searched by the surgeon and then.carries on at step 112.
At step 112, for each axis of symmetry Ml, M2,
processing unit 28 determines which pixels of the axis of symme-
try belong to the tool. For this purpose, a pixel selection
CA 02642913 2008-08-19
19
method based on Otsu's method which comprises assigning to each
pixel a color class from among two color classes may be used.
Otsu's method is, for example, described in the publication
entitled "A threshold selection method from gray level histo-
grams" by N. Otsu, IEEE Trans. Systems, Man and Cybernetics,
Vol. 9, pp. 62-66, 1979. As an example, processing unit 28
determines, for all the pixels of the axis of symmetry, a histo-
gram of a function H, representative of the color of the pixel,
according to a number of levels or color classes which depends
on the accuracy used for the coding of function H. A threshold
is then defined according to Otsu's method. A first color class
is then assigned to the pixels for which function H is smaller
than the threshold, for example, the lowest color class in the
histogram, and a second color class is assigned to the pixels
for which function H is greater than the threshold, for example,
the highest color class in the histogram. The pixels of the axis
of symmetry then distribute into groups of adjacent pixels of
the first or of the second color class. For each color class,
the pixel group having the greatest number of pixels is then
determined. To determine the color class which corresponds to
the surgical tool and the color class which corresponds to the
image background, it is considered that the pixel group of image
106 on the axis of symmetry M1 closest to projected insertion
point 01 necessarily belongs to the tool associated with projec-
tion insertion point 01. The color class associated with the
tool is thus obtained. It is possible for step 112 to only be
implemented for the first image provided by camera 17. For the
images subsequently provided by camera 17, it can be considered
that the tool corresponds to the pixel group having the largest
number of pixels and associated with the color class of the
previously-determined tool. The method carries on at step 114.
At step 114, processing unit 28 determines the posi-
tion of the end of the projection of surgical tool 94, 95 on
image 106, or tip S1, S2 of the tool. Tip S1, S2 corresponds to
the end of the pixel group associated with the tool opposite to
CA 02642913 2008-08-19
projected insertion point 01, 02. When tip S1, S2 thus deter-
mined is on an edge of image 106, this actually means that the
tip of the tool is not visible on image 106. It may be neces-
sary, to more finely determine the position of the tool tip, to
5 consider in further detail the pixels of axis of symmetry M1, M2
around the tool tip and to apply color constraints thereto to
determine whether, effectively, they belong or not to the tool.
At the end of the surgical tool detection method, processing
unit 28 thus has determined, for each surgical tool 94, 95 the
10 axis of symmetry M1, M2, edges D1, D1', D2, D2', and tips S1,
S2. The method carries on at step 102 for the processing of
another image provided by camera 17.
The present invention enables implementing complex
displacements commands by processing unit 28 for the displace-
15 ment of positioning system 10 of camera 17. As an example,
processing unit 28 may control positioning system 10 to displace
endoscope 12 until tip S1, S2 of a specific surgical tool 94, 95
is at the center of image 106 provided by camera 17. For this
purpose, the surgeon may provide a command of "center on iden-
20 tifier" type to processing unit 28 via voice control system 38.
Processing unit 28 can then detect the tools present on the
image provided by camera 17. If the tip of the tool correspond-
ing to the identifier provided by the surgeon is present in the
image, processing unit 28 then controls positioning system 10 to
displace endoscope 12 so that the tool tip is at the level of a
predefined central region of the image. If the tool correspond-
ing to the identifier provided by the surgeon is present on the
image but the tool tip is outside of the image, processing unit
28 is capable of controlling positioning system 10 to displace
endoscope 12 towards the tool tip since the position of the
insertion point associated with the tool is known. If the tool
corresponding to the identifier provided by the surgeon is not
present on the image, control unit 28 may control positioning
system 10 to direct endoscope 12 towards the insertion point
associated with the searched tool. The tool must then be on the
CA 02642913 2008-08-19
21
image provided by camera 17 and the centering on the tool tip is
performed as described previously.
Another control example comprises displacing endoscope
12 so that the tip of a surgical tool permanently is at the
level of the central region of the image provided by camera 17.
For this purpose, the surgeon may provide a command of "follow
identifier" type to processing unit 28 via a voice control
system 38. Processing unit 28 then controls positioning system
as described previously to center the image provided by
10 camera 17 on the tip of the tool corresponding to the identifier
provided by the surgeon. After, for each new image received by
processing unit 28, unit 28 controls positioning system 10 to
center back, if necessary, the image on the tool tip.
Another control example comprises controlling a
"magnification" of the image at the level of the tip of a
specific tool. For this purpose, the surgeon may provide a
command of "zoom on identifier" type to processing unit 28 via
voice control system 38. Processing unit 28 then controls
positioning system 10 as described previously to align the image
provided by camera 17 on the tip of the tool corresponding to
the identifier provided by the surgeon. Processing unit 28 can
then determine the distance between straight lines Dl, D1', D2,
D2' of edges associated with the considered tool at the toll tip
and deduce therefrom a value representative of the image magni-
fication. Processing unit 28 can then control positioning system
10 to displace endoscope 12 to increase or decrease such a
magnification to reach a predetermined magnification or a magni-
fication provided by the surgeon. According to a variation,
camera 17 may comprise a zoom function. Processing unit 28 is
then capable of controlling the zoom function of camera 17 based
on the detection of the distance between straight lines Dl, D1',
D2, D2'.
The duration of the processing of an image by
processing unit 28 may vary from a few tens of milliseconds to
some hundred milliseconds. Processing unit 28 can thus receive
CA 02642913 2008-08-19
22
new images to be processed at a high frequency. The surgical
tool detection method according to the present invention can
thus be implemented in real time.
The present invention thus enables, during a surgical
intervention, relieving the surgeon who does not have to syste-
matically control each displacement of endoscope 12. Indeed, the
previously-described controls enable the surgeon to focus his
attention on the surgical intervention, while processing unit 28
simultaneously automatically controls the displacements of
endoscope 12 according to the initial command given by the
surgeon.
The present invention may also be implemented when the
surgical tools are not cylindrical. The above embodiment may be
readily applied to a tool having an elongated shape, its projec-
tion in a plane having rectilinear edges. Such is the case for a
conical, prismatic, pyramidal shape, etc. More generally even
for tools having more complex shapes, geometrical parameters
characteristic of the tools may be stored in processing unit 28.
Steps 108 to 114 of the previously-described tool detection
method may then be adapted according to the geometrical parame-
ters characteristic of each surgical tool.
Of course, the present invention is likely to have
various modifications and variations which will occur to those
skilled in the art. In particular, although the present inven-
tion has been described for surgical interventions, for which
the video images are provided by a camera linked to an endos-
cope, the present invention may apply to any type of image
provided by a medical imaging system. Such is the case, for
example, for ultrasound images, fluoroscopic images, or scanner
images. Further, although the present invention has been
described for a specific positioning system example, it may
apply to any type of positioning system.
