Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
CA 02399713 2002-08-09
WO 01/60076 - 1 - PCT/SE00/00282
Design, function, and utilisation of an equipment for
capturing of three-dimensional images.
DEFINITION
Design, function, and utilisation of an equipment for
capturing of three-dimensional images of different objects.
TECHNICAL AREA
The present invention refers to composition, utilisation
and function of an equipment for capturing of three-
dimensional images for medical purposes, for example for
usage in micro-invasive surgery, for capturing of three-
dimensional images of objects.
Not long time ago humanity wasn't aware of an existing
separate binocular 'depth-sense'. Through ages individuals
such as Euclid and Leonardo have understood that we see
different images of reality with each eye. It was
Wheatstone, who, with his stereoscope and his drawings,
explained to the world 1838 that there is a unique 'depth-
sense', stereopsis, produced by differences between retinal
projections. Euclid, Kepler, and others wondered why we
don't see a phantom image of reality. V~lheatstone explained
that the problem was the actual solution, by demonstrating
that the brain merges the two planar retinal images into
one with stereopsis ("solid seeing") .1
Stereoscopic images
A stereoscopic image presents to the observer's left and
right eyes an image with pixels from different
perspectives, exactly as the observer sees the visual
reality. _ These two images, with slightly different
perspectives, are synthesised by the cortex of the brain
1 Wheatstone, Charles. On some remarkable, and hitherto unobserved, phenomena
of
binocular vision (Part the first). Philosophical Transactions of the Royal
Society of
London, 1838, 371-94.
CA 02399713 2002-08-09
WO 01/60076 - 2 - PCT/SE00/00282
into one with stereoscopic depth. The synthesised image is
bigger than the sum of the images.
The reality can also be captured by a camera. Irrespective
of the image is calculated or captured, the generation or
copying of images from two perspectives, one for the left
and one for the right eye, must take place. If a
perspective can be produced, it is conceptually possible to
create another perspective.
The monocular, or extra-stereoscopic, depth-variables are
the conditions for the perception of depth on visual
displays. They are as important as stereopsis to create
images, which are percepted as three-dimensional. These
variables include light and shadow, relative size,
interposition, text gradient, spatial perspective, motion
parallax, and the most important is the perspective. Images
rich on monocular depth-variables will be even easier to
visualise when binocular stereoscopy is added.
Stereoscopic displays
A stereoscopic display is an optical system, where the
human brain is the end-unit. It functions by presenting
left and right images of reality to the brain.
Those stereoscopic images are more realistic than their
planar equivalence is accepted by most. Though, the
addition of stereopsis to a display means a depth-sense, it
can be discussed if this addition makes the display more
realistic. Stereopsis adds information in a format, which
is both sensoric, comfortable and useful.
A stereoscopic image, which has a one-to-one conformity,
isomorphic, with reality, can be uncomfortable to look at,
and be of debatable value for the scientist, engineer,
technician, doctor and the artist.
There are many different ways a stereoscopic image can
diverge from being isomorphic with reality, psycho-physics,
the psychology of depth-perception, and geometric or
3~ optical changes. They can have the following states:
breaking down of accommodation/convergence, interacting
variables around the monitor, and ortho-stereoscopic
states.
CA 02399713 2002-08-09
WO 01/60076 - 3 - PCT/SE00/00282
There are many different means of producing time-shared
multiplex images for electronic stereo displays. Further,
the formatting of an image is separated from the selection
technique, or the way to supply each eye with its necessary
S image (and to erase unnecessary images). Formatting and
image selection must be simultaneous, and by design of such
system, the selection technique decides the format.
Stereoscopic image capturing equipment
With stereoscopic image capturing equipment means
equipment, which capture a three-dimensional image of
reality.
First, there can be two cameras with parallel lenses. The
object is captured simultaneously with the two lenses and
thereby creates two images with different perspectives
captured at the same moment.
Second, there can be cameras with one lens, which moves
mechanically in telescopic, vertical, horizontal or
rotational direction. Here is also two images captured from
the periphery of the movement. The images are captured with
a specified time difference and are therefore never in real
time. This has only importance if the object and/or the
lens are moving, and must therefore be corrected for.
The patent concerns format and capture equipment which
function together with visual presentation equipment and
uses time multiplex methods. Such visual presentation
equipment can be active by using liquid crystal (LC)
shutter, or it can be passively polarising.
The invention is defined as a way to use equipment for 3D
seeing (three dimensions) to capture objects close to or in
real time.
More specifically the invention can be defined as a way to
use such equipment to capture objects better and create a
depth-sense to see the relation of the object to its
environment.
CA 02399713 2002-08-09
WO 01/60076 - 4 - PCT/SE00/00282
THE PRESENT STATE OF THE ART
The camera model
The geometry of the basic algorithm to produce computer
generated electro-stereoscopic distortion-free images is
illustrated in Figure 1, with left and right camera
positioned in the data area. Their inter-axial separation
is given by t~. The axes of the camera lenses are parallel
in the z-direction. The distance between the cameras to the
object is do.
Imagine that the cameras are two still images or video
cameras whose lens axes are parallel, mounted on a planar
bed. t~ is the inter-axial separation or distance between
the cameras, or more exactly the centre of the lenses. The
cameras are mounted such that the lens axes', the lines
l~ through the centres of each lens with right angle to the
image plane, all the time are parallel.
Both the cameras produce two different perspectives of the
same object, because they are horizontally displaced by the
distance t~ . Both the cameras use lenses with the same
focal distance. Lenses with short focal distance produce a
wide-angle image, and lenses with long focal distance
produce a narrow-angle image. For example wide-angle lenses
for 35-millimeter photography are usually under 50
millimetres, and long focal distance or lenses of tele
2~ optics are usually over 50 millimetres.
Let us assume two video cameras with lenses directed
straight ahead. Each camera captures a perspective of an
object. The change between the two video camera signals
occurs with a certain rate.2 If we look at the images on a
TV-monitor, two non-identical images can be seen, because
the cameras are t~ apart from each other. Because the
cameras are directed straight ahead with parallel lens
axes, they will get different parts of the object. Let us
displace these two images horizontally in relation to each
other, such that part of the images fuse. Whenever the
image is fused the parallax is zero, and that part of the
Z Lipton, Lenny, and Meyer, Lhary. A time-multiplexed two-times vertical
frequency
stereoscopic video system. International Symposiasm Digest, SID 1984, VoI.XV.
CA 02399713 2002-08-09
WO 01/60076 - 5 - PCT/SE00/00282
object (or the scene) is in the monitor plane. It has been
called 'convergence', but the term is easily exchanged with
the human physiology term 'eye rotation' which is necessary
for fusion. Therefore we use the term HIT (horizontal. image
translation), which is used to get ZPS (zero parallax
setting).
If the image is horizontally displaced such that any part
of the image lies perfectly on top of the other, that part
of the image has ZPS. The object doesn't exist only on x-
and y-axes, but also on the .--axis. Because the object is
three-dimensional, we can reach ZPS for only one point (or
set of points oriented in one plane) of the object. If ZPS
is used for the mid part of the object, the parts behind
the object have ZPS positive parallax, and the parts in
front have ZPS have negative parallax.
The algorithm for parallel lens axes for stereoscopic
computer generated images uses two camera perspectives,
with parallel lens axes with a distance t~ between them.
Both perspectives and the camera lenses, have the same
angle. The degree of horizontal translation (HIT) of the
images is more a question about taste. Previously, the
advantages of creating images with low parallax were put
forward. One has to consider the usage of ZPS to produce
the best compromise of parallax for the whole image.
2~ This approach doesn't mean any rotation and results in a
geometric distortion producing vertical parallax, but the
two necessary perspectives for a stereoscopic image have
been achieved.
If HIT is used as described, with other conditions (lens
angle, distance to the object, t~) constant, there won't be
any change of the depth contents. It will take some time
for the eyes to reconverge for the different values of
parallax, but the total depth contents of the image will be
the same. After adjustment of the eyes, the image will be
as deep as before. One can produce, by HIT, images shown
completely in front of or behind the monitor. As previously
has been discussed, is it best to position ZPS at or near
the centre of the object to reduce separation of the
accommodation and the convergence, and to reduce artefacts.
CA 02399713 2002-08-09
WO 01/60076 - 6 - PCT/SE00/00282
Parallax
It has been suggested to use small t~ values and a big
angle, at least 40 degrees horizontally. The equation for
the degree of depth,
y
Pn: - Mfctc do _d~
,which is used by stereographers, describes how the maximal
value for parallax (Pm ) changes when we change the camera
set-up. Imagine usage of HIT, the parallax value for an
object on distance do will be zero. Then, one says that ZPS
is reached on the distance d°. An object on a maximal
distance dm now has a parallax value of Pm.
The aim is to produce the strongest stereoscopic effect
without exceeding a maximum of 1.5°parallax. The form of
the equation is helpful at understanding of this. The value
of the magnification (M) changes the value of Pm. For
example: An image on a big screen has more parallax than
the image on a smaller screen. A 24-inch monitor has twice
as much parallax as a 12-inch monitor, with other
parameters unchanged. To reduce t~ will reduce the value of
monitor parallax. To reduce the focal distance of the lens
f~ (use a wide-angle lens) also reduces Pm.
The most important factor controlling the stereoscopic
effect is the distance t~ between the two camera lenses.
The bigger t~ the bigger parallax values and bigger
stereoscopic depth-sense. The contrary is also true. If we
look at the obj ect very close to the camera, for example a
coin, insects or small objects, t~ becomes small and still
produce a strong stereoscopic effect. On the other hand, if
we look at distant hills maybe t~ must be hundreds of
metres to produce any kind of stereoscopic effect. To
change t~ is a way of controlling the depth of a
stereoscopic image.
Small values on t~ can be reached by using wide-angle
optics (low f~ ). Considering the perspective, we can see
that the relative position of the object's surrounding
CA 02399713 2002-08-09
WO 01/60076 - 7 - PCT/SE00/00282
parts and the distant part of the object, for wide-angle,
will be exaggerated. This exaggeration of the perspective
decides the degree of the stereoscopic depth effect. With
usage of wide-angle optics, t~ can be reduced.'
The approach with parallel lens axes is particular
important with usage of wide-angle for close objects. If
rotation is used in this case, the geometric distortion
will exaggerate the generation of falsified vertical
parallax.
The stereoscopic effect's strength is controlled by the
inter-axial distance t~, which in turn controls parallax
values for the object. G~lhenever the inter-axial values are
established, HIT will be used to control the ZPS of the
images, and will often be received by software, through
1~ moving of two parts of the buffer relative to each other.
Horizontal movement of left and right images in the upper
or lower part of the buffer produces HIT, which is used to
control ZPS. For an image with an object, the mid part of
the object is a good position for ZPS. If this
recommendation is followed the user will get difficulties
because of broken down because of accommodation,
convergence and concomitant 'cross talk'. It is important
to implement ZPS conditions of the image processing
software as default values. In addition, when the size of
the image is changed it is wished to keep ZPS, and this can
be implemented in the software. In the camera model for
parallel lens axes, when t~ changes, ZPS will also change;
and therefore, it is wished that the software implements a
constant ZPS even when t~ varies. It gives the user a
better quality and it is easier to look at the stereoscopic
Image.
There are two important requirements to get stereoscopic
three-dimensional images on a monitor:
1. The software must perform a perspective with a given
offset-for each eye, thereby simulating what each eye
would see if it had been in a three-dimensional world
upon which the software rendering is based, and
3 MacAdam, D.L. Stereoscopic perceptions of size, shape, distance and
direction.
SMPTEJoz~rnal, 1954, 62:271-93.
CA 02399713 2002-08-09
WO 01/60076 - $ - PCT/SE00/00282
2. The rendering of left and right eye must be formatted
for the computer monitor in a way, which allows the
stereoscopic hardware to separate information for the
left and right eye.
Parallel Lens Axes
Calculation of a stereoscopic image includes two monocular
perspectives from two different positions. In the old days,
computer generated stereoscopic pairs were created through
rotation of an object a few degrees, but it is not
recommended (see Figure 2). The use of rotation to generate
stereoscopic pairs results in vertical non-adjustment of
corresponding left and right image points. Such a vertical
non-adjustment causes the eye muscles to work in an
extraordinary way, which most persons experience as
uncomfortable. If the eyes try to fuse vertical parallax
without any depth information, the result can be painful.
The rectangle ABCD, on the left side has the vertical axis
marked with a dotted line. Rotated around the axis as in
the figure, the corner points A', B', C', and D are higher
or lower than corresponding points A, B, C, and D. Despite
facts, the example contains a simple figure, which is
typical for what is happening when rotation is used to
produce stereoscopic pairs. Instead it is recommended to
produce two perspective with a horizontal translation along
the x-axis, with a horizontal displacement for the resulting
images (HIT 'horizontal image translation') to establish
ZPS ('zero parallax setting').
The invention
The invention is based on mechanical displacement of the
lens in telescopic, horizontal, vertical or rotational
direction. The displacement is adjustable to the wished
distance between the two parallel images of the captured
3~ object, two different perspective of the same object not in
full real time. The image rate is dependent on, among
others, the shutter speed of the camera.
CA 02399713 2002-08-09
WO 01/60076 - 9 - PCT/SE00/00282
The advantage is an image capturing equipment with a much
smaller size (diameter) than two parallel cameras, and is
an important factor where the size of the capture equipment
is decisive.
It is referred to the following patent:
US4956705 A A set-up to electronically capture images
of an object to be used for three-dimensional
representation of the object.
It is referred to the following publications:
Wheatstone, Charles. On some remarkable, and hitherto
unobserved, phenomena of binocular vision (Part the first).
Philosophical Transactions of the Royal Society of London,
371-94, 1838;
Lipton, Lenny, and Meyer, Lhary. A time-multiplexed two-
times vertical frequency stereoscopic video system.
International Symposium Digest, Vol.XV, SID 1984;
MacAdam, D.L. Stereoscopic perceptions of size, shape,
distance and direction. SMPTE Journal, 62:271-93, 1954.
DESCRIPTION OF THE INVENTION
The purpose of the invention is to solve such problems,
which have been described above, involving bad overview and
spatial orientation of the captured object in relation to
its surrounding, by adding the third dimension i.e. 3D.
The invention
The invention consists of a tube, called the optics, with
an optical and/or a fibre optic system transmitting
reflected light from the object. Through fibre optics,
light is transmitted from an external light source through
the tube. In the end of the tube is a lens in a sledge or a
cradle. On both sides of the lens' cradle there is a
cushion filled with gas/liquid. This cushion can be filled
and emptied alternately with micrometer precision, where
the lens is displaced when the cradle is displaced
CA 02399713 2002-08-09
WO 01/60076 - 10 - PCT/SE00/00282
sideways. The gas/liquid is regulated by an external pump
connected to a controller card of a computer.
In the other end of the tube is a so called 'beam splitter'
to split the incoming light between different cameras
considered for different tasks. A 'beam splitter'
physically means that part of the light will be lost when
it is split between two external units/cameras.
A camera, for example a CCD-camera, is screwed on tight at
the end, and is concerned to capture the incoming light in
a Bitter. Thereby all reflected light points from the
object get a dot. Every dot has a value registered in an
image. The shutter of the camera decides immediately how
long the time of exposure is needed. It is wished that the
choice of camera, results in as short time of exposure as
possible, because blur contributes strongly to lose the 3D-
information of the image. Additionally, it is optimal that
a camera takes at least 30 images/second, i.e. twice as
many for 3D-images.
Also an IR-camera can be used at the same time. The purpose
of the IR-camera is to detect heat changes and gradients in
and around the object, and it can for example be used to
measure circulation, separate between vital and dead tissue
and see tissue reactions aso.
Peripheral equipment
The computer processes later the two images by using
stereographic mathematical algorithms. The images are
delivered to the computer from a controller card as a video
signal. The video signal from two images with different
perspective of the same object are processed and later
fused into one image, then presented in the display for the
observer.
The computer controls the pump with a controller card,
after the interpreted information has been received, which
can be extracted after the image processing. The accuracy
is on micrometer level.
Two controller cards: one for the video signal, which is a
normal video card with high resolution; and one for the
CA 02399713 2002-08-09
WO 01/60076 - 11 - PCT/SE00/00282
control of the displacement of the lens, after filling and
emptying of the side-cushions.
Head mounted displays (HNm) are used to visualise the 3D-
surrounding to the observer. For example surgeons can use
Hl~s to get a spatial perception of a body cavity or hollow
viscus. A HNB7 is connected to the computer, which transmits
a video signal of one/two perspective of the object. The
image shifts between the eyes take place after the computer
has commanded it.
3D-glasses are cheaper alternatives with considerably less
precision and resolution, but serves it purpose to
visualise the captured object and its surrounding in 3D.
Different variants with polarising and LCD-technique exist.
3D-glasses are connected to the computer, which transmits a
video signal of one/two perspective (-s) of the object.
Images shifts between the eyes take place after the
computer has commanded it.
Instead of two cameras, i.e. double systems, use one lens
system is an advantage,
1. when the lens can be displaced a certain distance in
telescopic, vertical, horizontal or circular direction,
2. because the size or the diameter of the instrument
become smaller,
3. because the equipment can be used in narrow localities,
and where other factors of size determine about the usage
of the instrument,
4. where, the invention is of specific value to
applications when the size (diameter) of the capture
equipment is decisive upon if the production of three-
dimensional images will take place.
Example of products with such requirement are medical rigid
and flexible endoscopes, for example arthroscopes,
laparoscopes, cystoscopes, bronchoscopes aso.
CA 02399713 2002-08-09
WO 01/60076 - 12 - PCT/SE00/00282
Other examples are fibre (opto)scopes, rigid as well as
flexible, and other camera applications where the size of
the instrument is the decisive factor for the usage of the
instrument.
FIGURE TEXT
One presently suggestive variant of the invention will be
described below with reference to drawings included where
Figure 1: Distortion free images. The camera axes (the axes
in the centre of the perspective) must be parallel.
Figure 2: Rotation produces distortion.
Figure 3: The test bench (from above). To the left is the
endoscope mounted in a test bench, and to the right is a
concomitant controller unit with belonging controller card.
The figure shows a camera controller unit (7), a light
source (8), a linear step activator with screw (9), a 9-way
D-type for the Stepper motor controller unit (10), a
programmable motor controller unit with a simple axis and a
chip on the card (11), a back plate (12), to the computer
( 13 ) , a PSU ( 14 ) , a KM6 II 3U rack in a rigid box ( 15 ) , a
linear sledge with the endpoint connected (16), an
endoscope (17), a test plate (18) and a camera (19).
Figure 4: Test bench (from side). To the left is the
endoscope mounted on a test plate, and to the right is
concomitant front panel of the controller unit. The image
shows power on button (20), fuse button (21), on/off button
(22) and the rack(23);
Figure 5: Test bench (close-up).
Figure 6: Endoscope (from the side of the endoscope and
frontal of the edge of the optics).
Figure 7:-The lens of the optics (frontal). The image shows
the so called side cushions (24).
Figure 8: Displacement of the lens of the optics (frontal).
Figure 9: The lens (details).
CA 02399713 2002-08-09
WO 01/60076 - 13 - PCT/SE00/00282
Figure 10: Linear step activator (details).
DETAILED MANUFACTURING DESCRIPTION
Below is a couple of examples of suitable displacement for
the invention.
Example Figure 1 shows a horizontal or vertical
displacement of the lens. Horizontal displacement is
preferred, when vertical parallax is created at vertical
displacement. Horizontal displacement gives a less loss of
the of clearness the periphery, but only a small side way
displacement of the lens must take place. In every side way
an image is captured by the CCD-camera, which later are
fused into one 3D-iamge. The figure shows the outer fitting
of the endoscope (1), the inner fitting (2), cradle/sledge
(4), lens (5), lens holder (6) and displacement distance
(FA) .
Example Figure 2 shows circulatory displacement of the
lens. The lens rotates around its own axis. Rotational
movement can create problems with the accuracy of the end
points. In two decided positions are images captured, which
later are fused into one 3D-image.
Example Figure 3 shows telescopic displacement of the lens.
The distance the lens is displaced decides what ordinate
light reflected by the captured object will achieve. The
closer the object the lens lies, the more peripheral the
2~ dot, and thereby increased number details, i.e. the farther
the lens is from the object the more central is the dot,
and thereby decreased number of details. In the inner and
outer position is one image captured, which later are fused
to a 3D-image. The figure shows the protection glass (3).
JO
In connection with the following figures some examples are
shown of usage of the invention in different contexts.
In surgery are all operations associated with more or less
tissue damage. It is considered important to try to
35 minimise the tissue damage, which could occur and occurs
during surgery. The tissue damage occurring during surgery
CA 02399713 2002-08-09
WO 01/60076 - 14 - PCT/SE00/00282
delays wound healing, rehabilitation and full recovery to
normal function.
Surgery can be carried out either open, closed,. or a
combination of the both techniques so called-micro-invasive
surgery.
Open surgery is an operation where the surgeon releases
anatomical structures and organs to get to the organ (-s)
which the operation is focused on. The tissue damage is
more when the release is more.
Closed surgery is an operation where the surgeon
manipulates the organ without using the scalpel on the
patient, i.e. no external wound is brought about. Example
on closed surgery is fixation of fractures, where the
fixature itself constitutes a wound, and closed correction
of fractures, where inner wounds might occur. The method
exposes the body for essentially less tissue damage in
comparison with open surgery.
The combination of open and closed could be characterised
as micro-invasive or minimal invasive surgery. A small
incision, usually 1-2 cm, is made for an optical
instrument, which later is conveyed into the body. One
strives to create a hollow viscus if doesn't already
naturally exist like a joint. The hollow viscus is created
by pumping a fluid or air/gas mixture into it.
The invention considers among others a way to use an
endoscopic instrument for 3D-seeing (three-dimensional) in
the body at the procedure:
More specifically the invention can be defined as a way to
use such equipment to visualise the field of the operation
to create a better spatial perception.
In Figure- 3 is shown an example of one of the test benches
we have been using to test the invention. The purpose was
to get the least necessary side displacement and to compare
different side displacements and their 3D-effects. The test
results have then been used to build the prototype of the
invention. The test bench consists of a CCD-camera
CA 02399713 2002-08-09
WO 01/60076 - 15 - PCT/SE00/00282
connected to the invention and an object possible to
displace sideways. A computer co-ordinates the capturing
and displacement of the object.
The camera has a connection to the computer, which receives
the incoming video signal. The camera is controlled by a
controller card of the computer. Parameters such as
exposure time, frames per second and light sensitivity can
be controlled.
The invention consists of a stiff optics (described later)
connected to the camera. It is anchored in the bottom
plate.
The object has been positioned in a linear sledge with
right angle to the lens. The sledge can be pulled and
pushed back and forth, such that the object is displaced in
horizontal direction with micrometer precision. The sledge
is pulled by a linear step activator, and it is in turn
controlled by a controller card of the computer.
Figure 4 shows the test bench from the side. The controller
box' s front side has two buttons, one power on/off and one
to fuse captured images. Additionally there is a power
lamp.
Figure 5 shows the test bench in more detail.
Figure 6 shows the invention together with a connected CCD
camera. The end of the optics illustrates that the optics
consists of a tube with different insulating envelopes.
Figure 7Fe1! Hittar ante ref erenskalla. shows an image
in detail how the lens of the invention lies between two
horizontal cushions. These cushions can be filled with
air/gas or any fluid. By filling one and emptying one
cushion, is a side displacement of the lens in horizontal
direction attained. Additionally, other vibration
mechanisms can be used to achieve side displacement, for
example oscillating fibres. The distance the lens shall be
displaced is decided by a number of parameters and is
controlled by the connected computer.
Figure 8 shows only the lens displacement in detail.
CA 02399713 2002-08-09
WO 01/60076 - 16 - PCT/SE00/00282
Figure 9 shows the optics, which consists of an outer
envelope protecting from thrusts and bending. Inside there
is an isolating layer for the optical components. The lens
system rests in a sledge supported on both sides
horizontally by a system of cushions. In front of the lens
there is a cover glass to protect from external damage of
the lens. Along the sides, parallel with the tube, lies the
mechanism for side displacement as an open channel, because
the pump itself must lie outside the optics.
Additionally, one can see frontal figures of the lens and
its side movement/displacement.
Figure 10 shows the linear step activator in all
projections.
The invention is not limited to the above given designs
l~ described above which can be varied within the framework of
the following patent description. Thus, the mechanical
displacement can be in both or several of the following
directions; telescopic, horizontal, vertical and
rotational. The invention is neither limited to medical
applications, endoscopy, but can also be used for other
kinds of three-dimensional captures, especially where the
size of the instrument is a limiting factor.
