Note: Descriptions are shown in the official language in which they were submitted.
2079094
PATLNT APPLICATION
TNREE-DIMENSION~L GRAPHICS SIMUL~TION
AND ~CTUAL IM~GING DAT~ COM~OSITE DISPL~Y
CROSS-REFERENCE TO A RELATED APPLICATION
This application is a continuation-in-part application of U.S.
Patent Application Serial No. 07/290,316, entitled Method and
Apparatus for Video Presentation from a Variety of Scanner Imaging
Sources, to Tyrone L. Hardy, filed on December 23, 1988, further
filed as Canadian Patent Application Serial No. 612,019, filed on
September 19, 1989, and U.S. Patent Application Serial
No. 07,428,242, entitled Three Dimensional Laser Locali7ation
Apparatus and Method for Stereotactic Diagnoses or Surgery, to Tyrone
L. Hardy, et al., filed on October 27, 1989, the teachings of which
are incorporated herein by reference.
BACKGROUND OF THE INVENTION
Field of the Invention (Technical Field):
The present invention relates to an apparatus and method for
combining real or actual three-dimensional scanning or imaging data
with simulated graphics so that the three-dimensional nature of
structures within a given area or region, such as a head, can be
viewed in relationship to the original image data.
Background Art:
There are primarily two methods in the art for three-
dimensional (3-D) medical image simulation using various diagnostic
scanning techniques, e.g., magnetic resonance imaging (MRI) or
nuclear magnetic resonance (NMR) imaging, computer tomography (CT),
various isotope imaging techniques, other multi-planar scanners, and
the like. These two methods are as follows:
1. Line Tracing or Boundary Marking of Data Taken From Image
Sections. In this method, line tracing or boundary marking or
contouring of image data taken from various image sections are used
to create "wire fr~me" graphic simulations of the contour margins of
each section and thereby simulate image data in a three-dimensional
fashion. Such wire frame simulations are usually rotated so that
~-~ 2 2079094
they -can be viewed from several perspectives to create a three-
dimens1Onal i~age. In some cases, the wire frame simulations are
shaded to produce solids which can then be simulated in a three-
dimensional fashion with associated simulation of light source views
froo various directions. The disadvantage of this technique is that
the original image, from techniques such as CT, MRI, etc., is
discarded once the wire frame tracings have been made. Valuable data
with regard to the quality and features in the original image data
are lost or not used. For example, this is somewhat like comparing a
photograph of an ob~ect to a line tracing of an ob~ect. In essence,
the original image is used to create another image which is a rough
simulation of some features of the original image.
Examples of this method are disclosed in the following patents.
U.S. Patent No. 4,791,934, entitled Computer Tomography Assisted
Stereotactic Surgery System and Method, to Brunnett, discloses a two-
dimensional "shadowgraph" comparison of two images to obtain an
indication of the relative spacial orientation and position between
the patient and a map. The actual CT data is discarded after
rendering the images.
2. Use of Original Imaee Data to Create a_Three-Dimensional
i_- In this particular method, image sections are combined to
produce a three-dimenslonal reconstructed solid of the sub~ect area
or ob~ect. The original image data is not lost but there is no
transparency; hence various structures within the sub~ect area cannot
be viewed in a three-dimensional manner in relationship to
surrounding areas. Usually, the reconstructed solid form is
processed in such a fashion that various two-dimensional surface
sections can be viewed. Although this may be superior to standard
two-dimensional imaging, there is still difficulty in viewing three-
dimensional relationships of subject parts within the reconstructeds~lid. For example, if a computer simulation of the head with all
its contents is reconstructed to produce a solid simulation, one
initially has no more information than if they were viewing the head
with the naked eye. Thus this kind of reconstruction does not
present any more useful data than can be obtained by visual
observation. In an attempt to overcome this difficulty, methods for
producing computer sections along different planes have been done to
.. . .. .. .. .... . . . .... ...... . .. . . . . .
: ; , , -: . :: : , ,: - . . .~: : :~ : ; . .:. . : : . .:: :: ;
207909~ .
_ ~ 3
"seen-inside the solid. As can be appreciated by those skilled in
the art, two-dimensional sections from various angles can produce
images which are themselves confusing in terms of anatomical
relations.
An example of this method includes U.S. Patent No. 4,777,598,
entitled Image Processing Systems and Methods, to Kellar, et al.,
which generates a three-dimensional reconstruction from two-
dimensional slices. However, once the solid is rendered, one cannot
see structures or features inside the solid. Also, the `598 patent
does not disclose mixing of images generated from actual data with
simulated graphics.
Other methods are utilized in the art for manipulating actual
image data, however, these do not combine actual data image
representations with simulated images of selected features of
diagnostic, therapeutic, or surgical devices or body ob~ects or
features. Examples of such methods are disclosed in: U.S. Patent
No. 4,259,725, entitled Cursor Generator for Use in Computerized
Tomography and Other Image Display Systems, to Andrews, et al., which
is addressed to overlaying a displayed image with another two-
dimensional simple image, such as a pointer, arrow, star, circle,rectangle, and the like; and U.S. Patent No. 4,608,635, entitled
~ethod and Apparatus for Tomographic Diagnosis, to Osterholm, which
discloses a tomographic diagnosis system utilizing an apparatus which
images the radiodensity of various regions of the body and compares
actual images with predetermined images generated from actual patient
data; and U.S. Patent No. 4,651,732, entitled Three-Dimensional Light
Guidance System for Invasive Procedures, to Frederick, which
discloses a three-dimensional guidance system which develops a line
of light above a patient's body to indicate the entry point and path
of an invasive instrument.
None of the methods or devices described above combine real or
actual three-dimensional imaging data with simulations of features of
actual structures or techniques to be employed pertaining to such
structures. For exa~ple, the present invention can combine images
representing actual data of a patient's tumor, and com~ine this image
with simulations, such as tumor volume; treatment zones to treat the
. ,. .; . : , i. ., :.. ~ , . . -
4 207~
tumor, such as radiation implant zones; path of a probe to reach the
tumor; and the like.
SUMMARY OF TH~ INVENTION
~DISCLOSURE OF THE INVENTION~
The present invention comprises an apparatus for displaying a
combined three-dimensional representation. This apparatus comprises
means for calling from storage and displaying in a two-dimensional
representation, at least one of the selected actual features of an
actual three-dimensional object, such as tumors, lesions, abscesses,
or abnormalities; means for generating and displaying in a two-
dimensional representation a graphic simulation of at least one
associated three-dimensional feature (a feature or structure of the
actual ob~ect, such as a transparent solid volume or wire frame
structure, or a feature or structure of another ob~ect, such as a
15 stereotactic frame, probe, probe trajectory, radiation implant, -
radiation zone resulting from a radiation implant, or radiation
beam); and means for combining the graphic simulation display with
the display of selected actual features as a composite ~mage in a
manner suitable for viewing. Useful imaging techniques include CT,
NMR, PET, DSA, isotope imaging, and the like.
In the preferred embodiment, the composite image display allows
for selection of a two-dimensional planar slice through the actual
object and displaying the selected slice in combination with the
three-dimensional graphic simulation two-dlmensional display. This
two-dimensional planar slice can be displayed at essentially any
viewer determined perspective. Also in the preferred embodiment, the
graphic simulation provides for the display of a volume above and/or
below the planar slice.
The selected actual features and graphic simulations may be
displayed in any form, including wire frame representations,
transparent solid representations, relative transparencies, and
multiple color representations.
The present invention further comprises a method for displaying
a combined three-dimensional representation. This method comprises
the following steps of:
207909~
-a. imaging an actual three-dimensional object and storing
the data;
b. calling from storage and displaying in a two-dimencional
representation at least one selected actual feature of the actual
ob~ect;
c. generating and displaying in a two-dimensional
representation a graphic simulation of at least one associated three-
dimensional feature; and
d. combining the graphic simulation display with the display
of the selected actual feature in a manner suitable for viewing by a
user.
The discussion above is applicable to the method of the invention.
It is a primary ob~ect to provide an apparatus and method for
combining three-dimensional imaging data with simulated computer
graphics for improved diagnostic, therapeutic, and surgical
techniques and other clinical procedures.
It is another object to provide an apparatus and method for
providing enhanced information about structures in the body to the
diagnostician or surgeon.
An advantage of the apparatus and method of the invention is
that it improves the utilization of scanning and imaging techniques
and devices.
Other ob~ects, advantages and novel features, and further scope
of applicability of the present invention will be set forth in part
in the detailed description to follow, taken in con~unction wi.h the
accompanying drawings, and in part will become apparent to those
skilled in the art upon examination of the following, or may be
learned by practice of the invention. The objects and advantages of .
the invention may be realized and attained by means of the ;~
instrumentalities and combinations particularly pointed out in the
appended claims.
2~79~9~
~ BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated into and form
a part of the specification, illustrate several smbodiments of the
present invention and, together with the description, serve to
explain the principles of the invention. The drawings are only for
the purposes of illustrating a preferred embodiment of the invention
and are not to be construed as limiting the invention.
Fig. 1 is a photograph of a computer screen display of an
actual CT image section or scan slice combined with a simulated
stereotactic frame and a stereotactic surgical probe directed towards
a tumor lesion within the confines of the head;
Fig. 2 is a photograph of a computer screen display of an
enlarged image section of Fig. 1, showing a wire frame simulation of
a tumor volume; ~ -
Fig. 3 is a photograph of a computer screen display of an
enlarged image section of Fig. 1, showing a simulated wire frame and
shading to present the tumor as a three-dimensional shell;
Fig. 4 is a photograph of a computer screen display of an
actual scan image slice with simulated wire frame tumor volume above
and below the slice and simulated stereotactic probes and radioactive
seeds;
Fig. 5 is a photograph of a computer screen display of multiple
views of actual scan image slices in relationship to other
corresponding CT images and showing simulated radiation treatment
zones within a simulated shell of the tumor volume;
Fig. 6 is a photograph of a computer screen display of an
enlarged view of an image, such as shown in Fig. 5;
Fig. 7 is a photograph of a computer screen display of multiple
views of an example of simulated tumor volumes within the confines of
a brain and simulated radiation treatment zones;
.~' ' ' ' ' '', ~ .: '', ;,' ,', ' , ':. ' .. : ' " ' " "',' '' ' " ''' .', '.' ' ' . ., ' '''' '' . ' . '. `
207909~
-Fig. 8 is a photograph of a computer screen display of a wire
frame presentation of a stereotactic frane, a tumor, and a probe
tra~ectory;
Fig. 9 is a photograph of three-dimensional simulation of a
5 stereotactic frame and actual MRI images in frontal, saggital, and ~ -
horizontal vLews using a whole brain voxel proportional method for
simulated mapping of each voxel area of the brain viewed; and
Fig. 10 is a block diagram of the apparatus and method of the
invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
(BEST MODES FOR CARRYING OUT THE INVENTION~
The present invention combines or overlays actual image
representations with simulated wire frame or solid graphics such that
the three-dimensional nature or features of structures or objects
within a given area or region, e.g., head, can be displayed and
viewed in relationship to one another. The advantage of the present
invention is that the original image data is not discarded to create
the three-dimensional simulation, yet associated data is used to
construct wire frame or solit contours of three-dimensional forms and
structures within the area of interest.
With reference to Fig. 10, the apparatus and method of the
invention provide for the following: (a) scanning, acquiring images,
or imaging 10 an actual three-dimensional object (e.g., brain),
preferably with a multi-slice imaging technique (e.g., from MRI, CT,
DSA (Digital Subtraction Angiography), PET (Positron Emission
Topography), and other isotope scanners) and storing 12 the data
obtained; (b) calling from storage 12 and displaying 14 in a two-
dimensional representation (e.g., on a computer display screen or
video monitor) a selected feature or structure (e.g., tumor) of the
actual object; (c) generating and displaying 14 in a two-di~ensional
-representation (e.g., on a computer display screen or video monitor)
a graphic simulation of associated three-dimensional features or
structure of the actual object or other objects; and (d) combining 14
the graphic simulation display with the display of the selected
actual feature or structure in a composite image in a manner suitable
8 2079~
for viewing by a user. The scanner 10 may be directly connected to
the storage 12 and display 14 means, such that images are directly
acquired, or images may be transferred to the storage 12 and
display 14 means via tape, ethernet or camera. Digital data of
images may be stored in the storage 12 and display 14 means, such as
a computer memory. The digital data in the images is analyzed and
displayed to define certain selected features or structures, or to
outline the boundaries of certain features or structures. Similar
data from other image sections are also analyzed and stored. These
data are then utilized to create three-dimensional simulations of
selected features or structures, such as wire frame, solid, surface
renditions, or transparent three-dimensional renditions. The images
created thereby are then displayed with various originally obtained
images, such as slices, to create a three-dimensional sLmulation of
selected features or structures rendered in various three-dimensional
manners, noted above, such that they can be seen in relationship to
the actual CT, NMR or other scanning data. Various original scan
data can be selectively displayed in relationship to the images
created or simulated and rotated and manipulated in various ways.
Images created, as noted above, can be further simulated in various
manners to define graphical three-dimensional simulation of other
ob~ects (e.g., stereotactic frame, probe, probe tra~ectories,
radiation implants, and radiation zones, radiation and laser beams,
and the like). Images created and stored can be recalled for further
display.
As can be appreciated by those s~illed in the art, the
simulation can be of a feature of the actual object (e.g., tumor
volume or structure, or other sub-structures, such as blood vessels
or ventricular systems within the brain or other body areas), or a
feature of another object (stereotactic frame, probe, probe
tra~ectories, radiation implants, and radiation zones resulting from
proposed radiation implants), which combined with the image from the
actual data provides an effective tool for the clinician. The
features simulated can be represented as transparent solids, wire
frame structures, in varying colors and transparencies, and
combinations thereof, and other renditions, depending on the desired
visual effect. As an example, a two-dimensional planar slice through
the brain generated from actual data, can be combined with graphic
2~79094
simulation of a volumetric entity, such as a tumor, above and below
the planar slice. The planar slice may be viewed at essentially any
viewer determined perspective.
The apparatus and method of the invention preferably utilize
the imaging methods and apparatus of U.S. Serial No. 07/290,316,
entitled t~ethod and Apparatus for Video Presentation from a Variety
of Scanner Imaging Sources, to Tyrone L. Harty, to generate more
accurate images from actual data or actual data combined with known
data (e.g., brain maps), although other techniques for generating
images from actual data could be utilized in the invention. The
resulting images from the actual data is then combined with simulated
graphics data to yield three-dimensional information useful for
diagnostic, therapeutic and surgical techniques and other clinical
procedures. The graphics simulations can be accomplished by methods
well known in the art using standard three-dimensional transformation
algorithms or methods, or by the methods described in patent
application Serial No. 07/290,316. The position of wire frame or
other renditions of structures or features within any image is
transposed to the actual image by these techniques.
In the preferred embodiment, the computer is connected to a
video printer, camera, magnetic tape, ethernet image acquisition
interface, or the like, so that black and white or preferably color
copies or photos can be printed whenever the viewer desires to make a
record of the procedure or images obtained. Also in the preferred
embodiment, all resulting images are stored (e.g., on magnetic or
video tape) so that an archive record can be maintained by the *
clinician.
The preferred apparatus and method of the invention provide for
one or more of the following features: acquisition of images
directly from any scanner; pseudo-coloring; edge detection; pixel
analysis; precise volume and point-to-point measurements; perspective
viewing of multiple image sections; and Flicker-Frame-Overlay for
transparency viewing (an image display technique in which the viewer
of a computer video monitor screen is presented with alternating
frames of different images or images from different sources in a
rapidly alternating manner such that images from two different
- : .. : . . .. :;: . . : , ...
', '' `, , ' ` ;'' , `.: ', ''-",'," ','` ;, , :-', , , ` ' '' ''. ','
. r,. :: '. , .
207909~
sources appear to be transparent to one another, i.e., two images are
presented in the same frame of reference for comparison). For
medical applications, the invention may provide for one or more of
the following features: preoperative planning and simulation of the
operative procedure; and tailored graphic scaling of brain maps,
probe tracks, electrophysiological maps, and volumetric and
coordinate measurements to patient images. For stereotactic
techniques, the invention may provide for one or more of the
following features: probe coordinate determination; probe tra~ectory
simulation; anatomical or electrophysiological diencephalic mapping;
whole brain mapping (e.g., Talairach/Tournoux); three-dimensional
brachytherapy optimization; three-dimensional laser localization for
open craniotomy, such as disclosed in co-pending application Serial
No. 07,428,242, entitled Three Dimensional Laser Localization
Apparatus and ~ethod for Stereotactic Diagnoses or Surgery, to Tyrone
L. Hardy, et al.; three-dimensional microsurgical simulation; three-
dimensional angiography; three-dimensional radiosurgery; and three-
dimensional imaging and graphic simulation of the stereotactic frame.
The mapping systems can be individually scaled and superimposed over
any patient's image for specific mapping purposes. For probe
simulation, the invention provides for rapid probe coordinate
determination and various simulations of the probe's position;
perspective; a graphic simulation of the probe as it travels through
a series of stacked slices; a three-way graphic view of the probe on
combinations of sagittal, frontal and horizontal images from various
scanners; and three-dimensional graphic simulation of the probe,
pointing indicators, or the like, within a stereotactic frame.
Throughout the specification and claims, the term "probe" includes
other pointing indicators, such as laser beams, X-ray or radiation
beams, other beams, tra~ectories, and the like.
Reference is now made to the drawings which illustrate
photographs of actual computer screen displays of actual imaging data
combined with simulated graphics such as a stereotactic frame, probe
and probe tra~ectories, radiation implants, wire frame, solids,
shaded volumes, and shells. The icons and numbers shown in the
photographs represent co~puter graphics control symbols. Although
the Figures show black and white photographs, the computer graphics
. ~. ~ : : . :. ,: - ` ~ : . : . :: .
11 207909~
: `
displays are preferably in varying colors so that the information can
be more readily discerned.
Fig. 1 shows an actual CT scan planar slice simulated within a
stereotactic frame with a stereotactic surgical probe directed toward
a tumor lesion within the confines of the head. Figs. 2 and 3 are
magnified or enlarged views of the image section shown in Fig. 1.
Fig. 2 is a three-dimensional wire frame simulation of a tumor volume
and structure relative to the image slice, and Fig. 3 shows wire
frame simulation shading to present the tumor as a three-dimensional
shell. In Fig. 3, the wire frame simulation of the tumor within the
confines of the brain has been shaded such that it appears as a
shell.
As can be seen by the drawings, by using the method and
apparatus of the present invention, one can choose a feature of
structure of interest within the confines of the imaged area and
simulate that structure in wire frame or solid form showing its
relationship to the image slices. The relationship of the structure
of interest to the surrounding anatomical areas is enhanced. Images
created in this fashion can be viewed from a number of perspectives
such that simulations of a structure above and below an image slice
can be viewed. An important feature of this method of the present
invention is the creation of image sections having true transparency.
That is, by example in Figs. 2 and 3, the three-dimensional
construction of the tumor volume and structure can be seen above and
below the image slice. By analogy, this is somewhat like having a
painting on a very fine mesh screen by which one can simultaneously
see through the screen to see ob~ects behind the screen, yet also be
able to see the painting on the screen.
Fig. 4 is another example of a three-dimensional wire frame
simulation of a tumor lesion within the confines of the brain viewed
in relation to an actual CT scan image slice. In this particular
display, again the tumor volume and structure above and below the
image slice can be viewed. Vertical stereotactic surgical probes are
simulated to show possible tra~ectories to target areas within the
confines of the tumor volume and structure. The bright white spheres
on the ends of the si~ulated probes simulate the position of
. .
2~7909~
12
radio-isotope seeds placed within the confines of the tumor as a
therapeutic treatment modality. The larger three-dimensional balls
or spheres within the confines of the three-dimensional wire frame
simulation of the tumor simulates the kill zones or therapeutic zones
for some additional isotope seeds. The radioisotope seeds are
present at the center of the spheres. This Figure is an example of
how multiple areas of interest can be graphically simulated relative
to an actual image slice. The surgical or therapeutic techniques can
be optimized by varying the position of the radioisotope seeds and by
varying the type of radioisotopes, which can be simulated prior to
employing the desired procedures.
Fig. 5 shows multiple views of the same three-dimensional tumor
volume and structure relative to another corresponding CT image
showing three-dimensional simulated radiation treatment zones within
shell and solid renditions of the tumor volume and structure. The
simulated radiation treatment zones (dark areas) which extend beyond
the confines of the simulated tumor volume and structure (light
areas) can be readily discerned. For demonstration purposes, an
enlarged computer rendition is shown in Fig. 6. Fig. 7 is another
rendition showing radiation treatment zones within the wire frame
simulation of the tumor volume and structure and where treatment
zones extend beyond the confines of the tumor. Such simulated
radiation zones around each seed are used in the preliminary stages
of radiation treatment optimization before final dose calculations
are performed. The grids (e.g., 1 cm square) can be used as a
measuring aid. This particular method can be used to optimize
treatoent of tumor lesions, that is the goal is to have the radiation
treatment zones touch the margins of the tumor volume and structure,
but not break out substantially beyond the margins.
This combining of actual image data with graphics simulation of
related three-dimensional entities and image transparency can be used
to simulate other therapeutic diagnostic and relational properties.
For example, Fig. 8 illustrates a wire frame presentation of a
stereotactic frame, a tumor within the confines of the brain and the
simulation of a therapeutic probe's tr~a~ectory toward a target area
within the confines of the tumor.
:~, , :
~ 13 2~79~9~
-Fig. 9 is another example showing a three-dimensional
simulation of MRI images in frontal, saggital, and horizontal views
using a whole brain voxel proportional method for mapping each voxel
area of the brain viewed. Various image slices can be presented in a
single fashion or in various combinations to study specific areas of
interest. Although this photograph shows orthogonal images, images
at other angles and in varying combinations may be utilized depending
upon the area of interest. This method can also be used, as stated
before, to show the three-dimensional relationships of various
structures within the confines of the brain in relationship to other
structures. For example, the ventricular system, certain blood
vessels, tumors, and other structural areas, lesions, abscesses,
abnormalities of the brain (or other areas of the body), can be
simulated in their three-dimensional relationships to other brain (or
other body) areas.
In the drawings, actual image data points are utilized in the
construction of the wire frame renditions. The lines connect these
actual data points to complete the frame. ;.
Although the drawings all show actual imaging data of brains in
combination with a stereotactic frame, the invention is not limited
to heads or brains, but could be ut$1ized on any body portion wherein
accurate and deta$1ed information is desired to be combined with
simulation~. For example, the apparatus and method of the invention
could be useful in the placement of objects into the body, such as
blood vessels during an angiogram, radioisotope therapy, biopsies,
radiosurgery, the optimization of laser or stereotactic positioning,
surgical or diagnostic methods for other body parts with a frame
system suitable for such body parts (e.g., cylindrical coordinate
system for a spinal stereotactic frame, or the like). The present
invention provides for a high degree of accuracy and one can tell
p~ecisely what specific anatomical areas are within the simulated
regions. The apparatus of the invention is also useful in
con~unction with attachments to frames, such as microscopes, lasers,
robots, other stereotactic devices or pointers, and radiation
delivery systems.
237~99~
: : 14
"
Although the invention has been described with reference to
these preferred embodiments, other embodiments can achieve the same
results. Variations and modifications of the present invention will
be obvious to those skilled in the art and it is intended to cover in
the appended clai~s all such modifications and equivalents.
