Note: Descriptions are shown in the official language in which they were submitted.
CA 02365045 2001-12-14
Method for the defection of guns and ammunition in X Ray scans
of containers for security assurance
Field of the Invention
The present invention relates to a method and system for detecting guns
s and ammunition in X-ray scans of containers for security assurance.
Summary and Advantages of the Invention
X-ray based 2D and 3D security imaging scanners, such as used at
airports, have requirements to detect guns. The. problem of detecting a gun is
complicated by the fact that it can have many forms and may be disassembled
into several components. Some or all of the constituent components may not be
easily recognizable as components of a gun and the components could be in
different items of luggage.
According to the present invention, the above problems of detecting a
gun can be reduced to the detection of a barrel of a gun and the bullets. The
~5 detection of the barrel is further reduced to that of finding a hollow
cylinder within
a predetermined range of diameters corresponding to currently available light
ammunition calibers. It's possible that the gun or components thereof could be
in
one baggage item and the ammunition in another. Therefore; the present
invention also describes a method of detecting the bullets which is reduced to
2o finding a metallic cylinder with the same diameter range as described for
the
barrel and a predetermined length range.
A further understanding of the other features, aspects, and advantages of
the present invention will be realized by reference to the following
description,
appended claims, and accompanying drawings.
2s Detailed Descrgation of the Preferred Embodiments)
According to one aspect of the present invention, there is provided a
system and method (algorithm) of detecting guns and ammunition in X-ray scans
CA 02365045 2001-12-14
2
of containers for security assurance.
According to one embodiment of the invention, a security scanner
computer workstation for use in conjunction with an X-ray based scanner,
either
2D or 3D CT based having shape detection software which detects the specific
shapes and sizes described above. Well known existing shape detection
algorithms such as Hough transforms and software are used with auto-
separation and labeling of suspect images and automatic noise elimination to
compensate for the effects of operator fatigue. The resultant detected threats
and other baggage contents are displayed on the computer screen in 3D views
using volume rendering graphics techniques.
The present invention includes a software shape detection algorithms
tuned to detect the presence of hollow cylinders (a bullet. is a hollow
metallic
cylinder containing explosive chemicals).
The present invention will, be further understood by the additional
descriptions A, B and C attached hereto.
While the present invention has been described with reference to specific
embodiments, the description is~illustrative of the invention and is not to be
construed as limiting the invention. Various modifications may occur to those
skilled in the art without departing from the true spirit and scope of the
invention
2o as defined by the appended claims.
CA 02365045 2001-12-14
Additional D~scription A
"IAP PrS Segmentation Architecture"
CA 02365045 2001-12-14
3--r
IAP PrS Segmentation Architecture
Introduction
Imaging Application Platform (IAP) is a well-established platform
product specifically targeted at medical imaging. IAP supports a wide set
of functionality including database, hardcopy, DICOM services, image
processing, and reconstruction. The design is based on a client/server
architecture and each class of functionality is implemented in a separate
server. This paper will focus on the image processing server (further
referred to as the processing server or prserver) and m particular on the
segmentation functionality.
Segmentation, or feature extraction, is an important feature for any
medical raging application. When a radiologist or a physician look s ar
an rage he/she will mentally isolate the structure relevant to the
diagnosis. If the structure has to measured and/or . visualized by the
computer the radiologist or physician has to identify such structure on
the original images using the software, and this process is called
segmentation. For the purpose of this document, segmentation is a.
process in which the user (radiologist, technician, or physician) identifies
which part of one image, or a set of images, belongs to a .specific
structure.
The scope of this white paper is to describe the tools available in the IAP
processing server thax can automate and facilitate the segmentation
process. They are presented in terms of how they operate and how they
can be used in combination with the visualization and measurement
functionality. The combination of these functionality allows the
application to build a very effective system from the user's perspective in
which the classification and measurements are carried- out with a simple
click. This is referred as Point and Click Classification (PCC).
Several segmentation tools have been published. The majority of them
are designed to ' segment a particular structure in a specific image
modality. In the IAP processing server we implemented algorithm, which
are proven and have a large applicability in the clinical practice. The tools
have been chosen in order to cover a large set of clinical requirement s.
Since we recognize that we can not provide the best solution for all
segmentation needs, our architecture is designed to be extensible. If the
application requires a specific segmentation algorithm,, it is possible to
emend the functionality supported by the processing server through a dll
or a shared library.
This white paper assumes that the reader is familiar with the IAP
processing server architecture and has minimal experience with the IAP
Cedara Software Corp. Page 1
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
3D visWization functionality. The reader can refer also to the "IAP-PrS
Image Processing" White Paper and "PrS Rendering architecture" White
Paper.
Cedara Software Core. Page 2
CA 02365045 2001-12-14
-3
lAP PrS Segmentation Architecture
Glossary
Vohytne Rendering A technique used to visualize three-dimensional sampled
data that does not require any geometrical intermediate
structures.
Surface Rendering A technique used to visualize three-dimensional surfaces
represented by either polygons or voxels that have been
previously extracted from a sampled dataset.
Interpolation A set of techniques used to generaxe missing data
between known samples.
Voxel A three dimensional discrete sample.
Shape Interpolation An interpolation technique for binary objects that allows
users to smoothly connect arbitrary contours.
Mttltiplanar or Curved Arbitrary cross sections of a three-dimensional sampled
Reformatting daxaset.
Binary Object (Bitvol) . Structure which stores which voxels, in a slice
stack,
satisfy a specific property (for example the voxels
belonging to an anatonucal structure).
ROI Region of interest: Irregular region which includes only
the voxels that have to be processed It is very often
represented as a bitvol
Segmentation Process which lead to the identification of a set of
voxels in an image or' set of images, which satisfy a
specific property.
C,edara Software Corp. Page 3
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Segmentation Tools
The segmentation process can vary considerably from application to
application. This is usually due to the level of automation and the
workflow. This is related to how the application uses the tools rather
then the tools themselves. The IAP processing server doesn't force any
workflow. A general approach could be to automate the process as much
as possible and allow the user to review and. correct the segmentation.
The goal then is to minimi-rx the user intervention rather then make the
segmentation entirely automatic.
Overview
The IAP processing server supports both binary tools, which have been
proven through the years as reliable, as well as advanced tools with very
sophisticated functionality.
The binary tools operate on a binary object; they do not use the original
density from the images. These tools include binary region growing and
extrusion.
The advanced tools operate .on a gray level image. They typically allow an
higher level of automation. These tools are based on gray level region
growing.
Figure 1.0 shows a schematic diagram of how these tools operate all
together.
Cedars Software Corp. p~ 4
CA 02365045 2001-12-14
IAP PzS Segmentation Architecture
Shape interpolation Gray Level R~'o~n growing
~, 1 threshold ~ day Level
Object ~ ~ Object
Extrusion
Rcgion growing
Figure 1.0 Taxonomy of the segmentation tools supported by the
IAP processing server.
The scope of each tool in Figure 1.0 is as follows:
Shape Interpolation: Reconstruct a binary object by interpolating
an anisotropic . stack of 2D ROIs. This functionality is
implemented in the Recon object
~ Extrusion: Generate a binary object by extruding in one direction.
This functionality is implemented in the ExtBv object
~ Thresholding: Generate a binary object by selecting all the vozels
in the slice stack within the range of densities. This functionality
is implemented in the Thr3 object.
~ Binary Region Growing: Connectivity is evaluated on the binary
image. This functionality is implemented in the Seed3 object..
~ ~Grap Level Region Growing: Connectivity is evaluated on the
gray level image, with no thresholding necessary before the
segmentation process. This functionality is .implemented in the
Seg3 object.
The IAP processing server architecture allows to connect these
objects in any possible way. This is a very powerful feature since the
segmentation is usually accomplished in several steps. For example
Cedars Software Core. Page 5
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
the slice stack can be threshold and then the structure isolated using a
region growing. Figure 1.1 was generated using this technique.
A . B
Figure 1.1: Region growing after a normal threshold can be used to
isolate an objects very e~ciently. Image A is the result of the
threshold with the bone wiadow in the CT dataset. The bed is
removed with a single click of the 'mouse. The resulted binary
object is used as a mask for the Volume Renderer.
Figure 1.2 shows .the. part of the pipeline which implements the
segmentation in Figure 1.1. The Ss object is the input slice stack, and
contains the original images. The Thr3 object is the object that performs
the thresholding, and the Seed3 object perfornis the region growing on
a point specified by the user.
__ Ss Thr3 ---~ Seed3
Figure 1.2 : The pipeline used for the generation of the binary
object in Figure 1.1.
The IAP processing server also supports several features for
manipulating binary objects diiecdy:
~ Erosion
~ Dilation
~ Intersection
Ceda.ra Software Coxp. Page G
CA 02365045 2001-12-14
3W
IAP PrS Segmentation Architecture
~ Union
~ Difference
~ Complement
For example as we'll see in the next section it is necessary to dilate a
binary object before using it as a clipping region for Volume Rendering.
For example, in the pipeline in Figure 1.3, the binary object has to be
dilated before the connection to the Volume Rendering pipeline. This
can be accomplished by simply adding a new Bv object at the end of the
pipeline, as shown in Figure 1.3.
Figute 1.3 : Pipeline in Figure 1.2 with the Bv object added which
will perform a dilation on the result of the region growing.
Binary Tools
The tools presented in this sections implement the well known
techniques that have been used for several years in the medical market.
Some of them, like Seed3, extend the standard functionality in order to
minimize xhe user intervention.
Thresholding (Thr3)
Thresholding is on of the most basic tools and is often used as a starting
point in order to perform more complicated operations, as shown in
Figute 1.1. The Thr3 object reconstructs a binary object selecting all the
voxels in a specific density range in the slice stack. If the slice stack is
not
anisotropy.the application can choose to generate the binary object using
cubic or nearest neighbor interpolation.
Thr3 also supports an automatic dilation in the X and the Y direction.
This is useful in situations, like the one in Figure 1.3, where the binary
object be has to be used in the Volume Rendering pipeline.
Extrusion (ExtBv)
Extrusion projects a 2D shape along any direction. This feature is very
powerful when it is used in conjunction with 3D visualization. In fact, it
Cedara Software Corp. p~ 7
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
allows irrelevant structures to be eliminated very quickly and naturally, as
shown in Figure 1.4.
Figure 1.4 : F,xtiusion is a mechanism which works very well in
conjunction with 3D visualization. The user draws ROI which
defines the region of the dataset in which he is interested in. The
data outside the region is removed.
Figures 1.4.A and 1.4.B show the application from the user's perspective.
The user outlines the relevant anatomy and only that area will be
rendered Figure 1.4.C shows the binary object that has been generated .
through extrusion. This object has been used to restrict the area for the
volume renderer, and so eliminate unwanted structures.
Shape Interpolator (Recon)
The shape interpolator reconstructs a 3D binary object from a stack of
2D ROIs. The stack of 2D ROIs can be unequally spaced and present
branching, as shown in Figure 1.5. The Recon object supports nearest
neighbor and cubic interpolation kernels, which are guaranteed to
generate smooth surfaces; (see Figure 1:6). This functionality is used
when the user manually draws some ROIs on the original slices or
retouches the ROI generated through a threshold.
Cedara Software Corp. , ~ Page 8
CA 02365045 2001-12-14
3~~
IAP PrS Segmentation Architecture
A B
Figure 1.6 The shape interpolation can be accomp4shed with cubic
interpolation (A) or nearest neighbor interpolation (B).
Binary Connectivity (Seed3)
Connectivity is a well proven technique used to quickly and efficiently
isolate a structure in a binary object. The user, or the application,
identifies a few points belonging to the structure of interest, called seed
points. All the voxels that are connected to the seed points are extracted,
typically removing the background. This process is also referred as
"region, growing".
Region growing is very often used to 'clean' an . object that has been
obtained by thresholding. Figure 1.1 shows an example of a CT dataset
where the bed has been removed simply by selecting a point on the skull.
Cedars Software Core. page 9
Figure 1.5 The Shape interpolation prpcess reconstruct a 3d binary
object from a set of~2D ROIs, even if they are not equally spaced
and have branching.
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
'This functionality is very effective when combined with the Coordinate
Query functionality of the prserver volume renderer. Coordinate Query
allows the application to identify the 3D location in the stack when the
user clicks on the rendered image. By combining these two tools, the
entire operation of segmentation and clean up can be done entirely in 3D
as shown in Figure 1.7. See the "PrS 3D Rendering Architecture" White
Paper for more details on the Coordinate Query.
A ( B..
Figure 1.7 MR pheriperial angio dataset. The Volume Rendering
visualization of the vasculature also includes other unrelated
structures (A). By just clicking on the vessel, the user can elimixiate
the background irrelevant structures (B).
The Seed3 object implements a six-neighbor connectivity. It also
supports the following advanced functionality in order to facilitate the
identification of the structure:
1. Tolerance: when a seed point is not actWy on the object,.Seed3 will
move it to the nearest voxel belonging to the binary object, if the
distance is less then the tolerance specified by the application: This
functionality allows the applicaxion to compensate for rounding
errors and imprecision from the user.
2. Filling: This functionality removes all the holes from the object
segmented It is sometimes used for volume measurements.
3. Small links: When a bitvol is. generated using a noisy dataset, several
parts of could be connected by narrow structures. The Seed3 object
allows the "smallest bridge" in which the region growing can grow to
be specified. This functionality allows the system to be insensitive to
noise. Figure 1.8 'shows how this feature can extract the brain in an
MR dataset.
Cedara Software Corp. Page 10
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
A
Figure 1.8 : MR dataset of the -brain. The seed point is set in the
brain. In (A) the region growing fails in extracting the brain since
there are small connections from the brain to tire skin. In (B) the
brain is extracted because the small connection ate not followed.
4. Disarticulation: ~ This is the separation of different anatomical
structures that are connected. The application can specify two sets of
seed points, one for the object to be.kept and one for the object to
be removed. Seed3 will perform erosion until these two sets of
points are no longer,. connected and then. perform a conditional
dilation the same amount as the erosion. This operation is
computationally intensive. It works well if the bitvol has a well-
defined structure, i.e. the regions to be separated do not have holes
inside them, and narrow bridges link them. On the other hand, if the
regions are thin and the thickness is comparable to the thickness of
the bridges, then the result may not be optimal. Figure 1.9 shows
how this feature can be applied to select half of the hip in a CT
dataset.
,: .
n..
Cedara Software Corp. Page 11
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Figure 1.9 In the binary volume in (A) the user sets one seed point
to select the part to include (green) and one seed point to select the
part to remove (red). The system identifies the part of the two
structures with minimal connection and separates the structures
there. (B) shows the result.
Gray Level Connectivity
The concept of connectivity introduced by the binary images can be
extended for the gray-level images. The gray level connectivity between
two voxel measure the "level of confidence" for which these voxels
belong to the same structure. This definition leads to algorithm which
enables a more automate method for segmentation and reduces user
intervention. Grey level connectivity tends to work very well when the
structure under investigation has a narrow range of densities with respect
to the entire dynamic range of the 'images.
The Basic Algorithm ' w
The algorithm takes a slice stack as input and a set of seed points. For
each voxel in the stack it calculates the "level of confidence" with which
this voxel belongs'to~the structure identified by the seeds. Voxels far
from the seeds or with a different density than the seeds have a low
confidence value, whereas voxels close to the seed points and with
similar density have high confidence values. Note that no thresholds are
required for this process. From the mathematical point of view the
"confidence' level""is defined as connectivity from a specific voxel to a
seed point. The definition of connectivity from a voxel to a seed point
according to' Rosenfeld is
C(seed,voxel)'MaicP(,~~,~EP(s~~ ~,(Z)~
Where P(seedvoxel) is any possible path from the seed point to the
voxel, ~.(.) is a function that assigns a value between 0 and 1 to each
element in the stack. In our application w(.) is defined as follows:
p.(voxel)-1-~ density(voxel)-density(seed) ~
The connectivity is computed as:
C(seed,voxel)=1-Minn",~,~ ~axzEp(xed,v~ I density(z)-density(seed)
In simple terms the' connectivity of awoxel to a seed point is obtained by:
1. Considering all the paths from the seed point to the voxel.
Cedars Software Corp. Page 12
CA 02365045 2001-12-14
~3
IAP PrS Segmentation Architecture
2. Labeling each path with the maximum difference between the seed
point's density and the density of each voxel in the path.
3. Selecting the path with minimum label's value.
4. Set the connectivity as 1.0 minus the label's value.
For multiple seed the algorithm in step 2 uses the average density of the
seed points.
The algorithm computes the connectivity values for each voxel in the
stack and produces an output slice stack which is called "connectivity
map". Figure 1.10 shows a 2D example :using an MR image arid its
connectivity map. The area in which the seed has been placed has the
connectivity values higher then the rest of the image, and so it appears
brighter.
._ I B
Figure 1.10 : Image A' shows an MR slice and the place where the
seed point has been. Image B shows the connectivity map.
Figure 1.11 shows a 3D example in which the connectivity map is
rendered using MIP. In this example the dataset is an MR acquisition of
the head, a.nd several seeds point have been set in the brain, which
appears to be the brightest' region.
Cedara Software Corp. . Page 13
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Figure 1.11: MIP image of a connectivity map. In this example
several seed points have been set in the brain of this MR dataset.
The MIP image shows that the brain is the brightest area.
The connectivity map is thresholded at different values in order to
extract the area of interest as a bitvol~. Note that in this case the user
need only to control one threshold value. The connectivity map has
always to be threshold from the highest value (which represent the seed
points ) to a lower one defined by the user. The user increasing the
threshold removes irrelevant structures and refines the anatomical
structure where the seed has been planted. From the user perspecti ve
this method is quite natural and effective, the user visually browse the
possible solution interactively. Figure 1.12 shows an example of user
interaction; the connectivity map shown in figure 1.11 is thresholded at
increasing values until the brain is extracted.
Cedara Software Core. ~ Page 14
CA 02365045 2001-12-14
3 /~~-
Figure 1.12. : The connectivity map shown in figure 1.11. is
threshold and the 'binary object is used to extract the anatomical
feature from the original dataset. This process is done interactively.
In figufe 1~: the binary object computed thresholding the connectivity
map is applied as a mask for .the volume renderer . As mention in the
previous section iris necessary a small dilation before the connection to
the volume rendering pipeline. Figure 1.13 shows the complete pipeline.
Seg3 ~--1~ T~3~-.~. ( Bv ~-.-~ Bvf
Ss Vol ~ Cproj _.~
,-
Figure 1.13_;,pipeline used for t>~e generation of the images in 1.12.
The connectivity map , is thresholded interactively changing the
threshold.fpt the :Th3 object. The Bvf object is used .to avoid the
pre-processing in the Vol Object. Please refer to the "PrS 3D
Renderiag~A~chitetture" for a detailed explanation of the possible
usage of the. clipping tools in -the Volume Rendering pipeline.
Contrast Table
In order to optimize the segmentation in terms of accurary and speed the
Seg3 object can use a contrast table which enhance the contrast between
the anatomical structure under investigation and the rest of the anatomy.
The region growing process will operate on the image once it has been
remapped with the contrast table. The connectivity will be calculated as
follow:
Cedara Software Corp. - ~ ' ~ -''r, Page 15
IAP PrS Segmentation Architecture
CA 02365045 2001-12-14
~-~6
IAP PrS Segmentation Architecture
C(seed,voxel)=1-Min ~s~~ [Ma~EP~s~~ ~ contrast table(density(z))-
contrast table(density(seed)) ~
The applicaxion can take advantage of this functionality in several ways:
~ Reducing the noise in the image.
~ Increase. the accurary of the segmentation eliminating densities
that are not part of the anatomy under investigation. For example
in a CTA dazaset the liver doesn't contain intensity as high as the
bone. Hence they can be remapped as zero (background) and
excluded from the segmentation.
~ Limit the user mistakes: if the user sets a seed point in a region a
region which is reriiapped on a low value ( as defined by the
application ) the seed point will be ignored For example if the
user in the intent to segment the liver in a GTA dataset sets a
seed point: on the bone, it will not considered during the region
growing process.
'The application is not force to use the contrast table, when it is not used
the system will operate on the original density values. For example the
brain m figure 1.12 was extracted without contrast table.
The application can expose this functionality directly to the user, or if
appropriate, use the rendering setting used to view the images.
~ In order to segment structure withy high density the window
level that the user set iii the 2D view can be used as contrast
table.
The opacity ctuwe to render a speafic tissue can be used a
remappmg table. The users in order to visualize the tissues
properly has to set the opacity to 100% for all the densities in the
tissues and then lower values for density which partially contains
also part of ~ other tissues. So the opacity curve implicitly
maximizes the contrast between the tissue and the rest of the
anatomy.
The Extended Algorithm
The basic algorithm is a very powerful tool, as shown in figure 1.12. In
order to extend its functionality the Seg3 object implements two
extensions:
Cedara Software Corp. Page 16
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
1. Distance path. In some situation' the structure, which the user is
trying to extract, is connected with something else. For example for
the treatment planning of the AVM shown in figure 1.14 the feeding
artery and the draining vein have to be segmented from the nodule of
the AVM. The density of these three structures is the same ( since it
is the same blood which flaw in all of them ) and they are connected.
A B
Figure 1.14 MR dataset of the head region showing an AVM.
The image A if the MIP~of the dataset the image B is the binary
segmentation of the dataset. Binary region growing is not able
to segmented the three anatomical structures ( veins, artery,
avm ) require for the treatment planning.
In order to facilitate the segmentation of the structures Seg3 can
reduce the connectivity value of any voxel proportionately to the
distance' oa the path from the seed point. Seeding the vein, as shown
in figure 1.15 will cause voxel with the same density, but in the
nodule of the avm, to have lower connectivity value; and hence
exclude them. Note that the distance is measured along the vessel,
since the densities outside the vessel's range will be remapped to zero
by the contrast table arid not considered. The user browsing through
the possible solution will visually see the segmented area following
the vein as shown in figure 1.15.
Cxdara Software Corp. Page 17
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Figure 1.15 Vein segmented at different threshold value. The
user changing the threshold can visually follow the vein. In
order to generate these images the pipeline in figure 1.13 has
been used. Distance path is usually used in conjuction with
contrast table.
Figure 1.16 shows the example analyzed by Dr: Eldrige. Iri this case
the functionality was used not just for segmentation but for
increasing the understanding of the pathology following several
vessels and see how they are interacting.
As we mention in the section "Binary Connectivity" disarticulaxion
can be similar situation. However disarticulation .is mainly designed
for bone structures and doesn't allow any level of control. Distance
path is instead designed for vessels and allows the user to have a fine
control on the region segmented
2. Growth along an axe. In some protocols, like the peripheral
angiography, the vessel will follow mainly one axe of the dataset. The
applicatiori'c:ari use 'thisunformation to facilitate the segmentation
process at;d force the region growing process to follow the vessel
along the main axe, and so select the main lumen instead of the small
branches: Figure 1:17 shows an example of this functionality.
. , .,-., . ... .
Cedara Software Core. Page 18
Figure .1.16 Example ~ 1.14 analyzed by Dr. Eldrige. Dr. Eldrige
used the distance path functionality to follow the vessel
involved. in the aneurysm and analyzed their interaction.
CA 02365045 2001-12-14
~ ~9
IAP PrS Segmentation Architecture
Figure 1.17: Segmentation of a~ vessel in a MR Peripheral
angiography. Seg3 allows to associate weights to each one the axes,
each weight represent an'~ncremental reduction of the connectivity
value for the region growing to follow that axe.
Embedding the knowledge
The benefit of the algorithm; goes behind the fact that the application
doesn't have to set a priori threshold. The application can embed the
knowledge of the structure that the user is segmenting in several ways:
1. As presented in the previous section the contrast table is the simplest
and effective way for the application to guide the region growing.
2. The number of densities expected in the object can be used to guide
the region growing. Note that the process requires only the number
of densities and not to specify the densities included. The threshould
in the connectivity map identify the number of connectivity values
included in the solution and hence the number of densities as defined
by the C(vogel,seed)~ formula: . Note that when the distance map or
the growth along an axe: is used the voxels with same contrast value
can have different connectivity according to their distance to the seed
point.
3. The volume size ( as number of voxels) of the object can be used to
guide the region growing. The volume size of the object can be
simply measured querying the histogram of the connectivity map and
adding all the values from the threshold to the maBimim value.
4. The relative position of the seed point can be used to guide the
application in forcing the region growing process to follow a
Cedara Software Core. Page 19
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
particular axe. For 'example in the dataset in figure 1.17 the user will
set several seed points along the Y axe. Evaluating the displacement
of the points in the:ZX plane the application can estimate how much
the vessel is following the Y axe and so how much the region
growing has to' be bound to that.
The information gathered with the previous 4 points can be used by the
application for two different purposed:
Optimize for performances. The time requested by the Seg3
object is proportional to the number of voxels selected. Avoid
including unwanted. structure will speedup the process. For
example in protocol used for dataset in figure 1.12 the volume of
the brain can not be more then 30% of the entire volume since
the whole head is inane field of view. So the solutions first two
solution could be not even included in the connectivity map since
the region growing should have been stopped before.
~ Identify the best solution, the one that most likely is what the
user is looking for. This solution can be proposed as default.
More specifically the previous information can be used for these two
puiposed in the following way:
Opdcnize for Pexfom~aacesIdentify best solution
Setting to zero
the densities,
which are guaranteed
not to belong to
Contrast ~e ~ to segment,
Table will improve performances
and also the
quality of the segmentation
by reducing the
number of possible
solution.
The Seg3 object The threshold set
accepts as a in the
input the number connectivity map
of densities to is actually the
include in the solution.number of densities
It will to be
Number of select the closer included in the
densities after solution' after
Densities the contrast ' tablethe contrast table
as been as been
aPPli~ ' ~~ these applied
densities
have been included
the process
will stop: ,
The Seg3 object This value can be
accept as a used to limit
input the number the threshold in
of voxd to the
include. connectivity map.
Querying the
Volume Size histogram of the
connectivity
map the application
can
estunate the volume
of the
object segmented
for each
threshold.
Relative . Comg ~e region Not Applicable.
position growing
of the in rocess will mdir
ut seed reduce the
C,edara Software Core. Page 20
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture '
points number of voxel to include.
Using this functionality will
increase the cost associate in the
evaluation of each voseL
The applications is expected to use some conservative values for the
volume size and number of densities to stop the region growing process,
and the realistic value to select the default solution.
The ability to embed the knowledge of the object under investigation
makes gray level region growing well suited for being protocol-driven.
For each protocols the application can define a group of preset which
target are the relevant anatomical structure.
Binary versus Gray ~,evel Region Growing
The basic algorithriz as presented in the previous section can be proven
to be equivalent to a binary . region growing where the threshold are
known m advanced. So this process doesn't have to be validated for
accuracy since the binary region growing is already m use right now.
Avoiding a priori knowledge of the threshold values has a major
advantage for the application: .
1. The number of solution that the user has to review are limited and
pre-calculated. Requiring the user to set only one value, for the
selection of the solution, means that the user has to evaluate ( at
worst ) 256 solution for an 8 bit dataset, while using the binary region
growing the user will have to evaluate 256'256= 65536 soluxion since
all the combiriatiorl of the low and high threshold have to be
potentially analyzed.
2. Finding the best threshold is not natural from the user's perspective.
Figure 1.18.B shows a CTA of the abdominal region in which the
Aorta has been segriiented. To obtain this result the user has seeded
the Aorta with the settings shown in figure 1.18.A.
Cedara Software Corp. Page 21
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Figure 1:18 Image A shows the Aorta extracted from the dataset
shown in figure B. In this case only one seed point was used.
In order to obtaimthe same result with the binary region growing the
user has to manually identify the best threshold for the Aorta, which
is shown in figure 1.19 and then seeded. Looking at figure 1.19.A is
not clear that these are the best setting for the segmentation and so
they can be easily overlooked.
Figure 1.19 Threshold setting necessary to extract the Aorta as
in $gure 1.18. The image A appears with several holes and it is
not clear with the Aorta is still connected with the bone.
3. The threshold set by the user can be dictated. by the volume of the
object rather then the density values.
Our experience shows that the quality of the result achievable with this
functionality is not achievable with the normal thresholding. Even in
situation in which the thresholds are known in advanced is preferable to
use this information as contrast table, and avoid binary segmentation.
Advance .usage
The gray level region growing allows reducing the time to perform the
segmentation from the user perspective. It guides the user in the
selection of the best threshold of the structure under investigation.
In the previous section we have been using the connectivity map for the
extraction of the binary object using the pipeline in figure 1.13. In some
C~edara Software Corp. Page 22
CA 02365045 2001-12-14
~ c~3
IAP PrS Segmentation Architecture
situation it could be valuable to use the connectivity map to enhance the
visualization. The connectivity map tend to have bright and uniform
values in the structure seeded and darker values elsewhere. This
characteristic can be exploded in the MIP visualization to enhance the
vessels in the MR.A and GTA dataset. Figure 1.20 shows an example of
this applicaxion; image 1.20.A is the NIIn of an MRA dataset, ah~e
1.20.B is the MII' of the connectivity map. Figure 1.21 shows the MII' of
the dataset in which the connectivity map and the original dataset have
been averaged, and compared it with the NIIP of original dataset in the
upper left corner. In this case it is visible that the contribution of the
connectivity help to suppress the background values enhancing the
vessels.
tai:
G h;p
.
;
i
1;:e:
. y:.
W : ~:y:
i :::
.. .,. B
. . . ..
..<>. .,
Figure 1~0'' Image A is the MIP of a MRA pheripet~ial dataset.
Image B shows the connectivity .map of the same region when the
main vessel has been seeder~_
Cedara Software Core. ' _ Page 23
:, ...
CA 02365045 2001-12-14
lAP PrS Segmentation Architecture
Figure 1.21 MIP. of the dataset obtained averaging the connectivity
map and the original dataset shown in figure 1.20. In the upper left
corner it is superimposed the MIP of the original dataset. It is
visible the averaging,helps in suppressing the background density
while prese~ing ~the~details of the vessel.
In some situation it is not necessary to have the user setting directly the
seeds for the identification of the anatomical structures. The seeds point
can be identified as a result of a threshold. on the images under
investigation. Note the threshold is necessary to identify only some point
in the structure and riot the entire structure, so the application can use
very conservative values. For example in the CTA a very high threshold
can be used to generate some seed point on the bone. An MRA a very
high threshold can be used to generate seed points on the vessels. The
seed points in Figure 1.21 ;and 1.20 have been generated using this
method. Figure 1.22 shows an example of this technique.
",
A r
~ B
Figure 1.22 Image !A shops the seed point generated in a CTA
dataset of therneckregio~ for the identification of the bone. Image
B shows the; ,b9ny~ ; segmented from the same dataset. In this
example th~.user interveption is minimized only in the selection of
the best tle~sh~ohd:-~ ," p,.::
c . _ .
Seg3 suppers ~thisltech,iiique since it is designed to accept a bitvol as a
input for, the ident~cation of the seed points. The bitvol cars be
generated°by a threshold, and edited automatically of manually.
Cedara Software Core. , Page 24
. ', ~~ r.~.. ,~..y,.
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
VISLI~IZltlon anC~ Segmentation
Visualization is the process that usually follows the segmentation. It is
used for the validation of the segmentation as well as correction through
the tools presented.on the previous section.
The IAP processing server supports two rendering engines that can be
used for this task. A very sophisticated Volume Renderer and a Solid
Renderer. Please refer"to the "PrS 3D Rendering Architecture" D~hite
Paper for .a detailed description of these two engines: This section will
focus mainly on how these engines deal with binary objects.
~r,. ,
Volume Rendering
Volume Rendering allows~'the direct visualization of the densities in a
dataset, using opacity and colour tables, which are usually referred as
classification.
The IAP processing server extends the definition of classification. It
allows define several regions (obtained by segmentation) and associate a
colour and opacity to each one of them. We refer as a local classification,
since It allows to specify colour and opacity based on voxel's density and
location. It is descnbed in details in the "Local Classification" section of
the "PrS 3D Rendering Architecture" White Paper.
Local chass'~cation' 'is~ necessary in the situation in wlich several
anatomical'' striicture sliace the some densities; situation extremely
common: in=the. medical imaging. For example in figure 1.18 the read
density of the Aorta appears also in the bone due to the partial volume .
artifact.
So the goal of an application using Volume Rendering as a primary
rendering tools is to classify the dataset, not necessarily to segment the
data. The goal is to allow the user to point on an anatomical structure on
the 3D image and have the system classify that properly. This is the target
of the "Point and Click Classification (PCC)" developed in Cedara's
applicaxions which is based on the tools and techniques presented in this
wlute paper.
As we'll describe in this section segmentation and classification are tight
together. ' .
Cxdara, Software Corp. Page 25
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Segmentation as a Mask
The IAP volume renderer uses a binary object to define the area for each
classification. ~ When an application classifies an anatomy that shares
densities with other anatomical structure, it has to remove the
ambiguities on the shared densities defining a region ( binary object )
which includes only the densities of the anatomy.
The binary object has to loosely contain all the relevant (visible) densities
of the anatomy, it doesn't have to define the precisely boundary of the
anatomy. It is supposed to mask out the shared destines which do not
belong to the anatomy. The opacity function allows the Volume
Renderer to visualize the fine details in the dataset. The section "Local
Classification" of the "PrS 3D Rendering Architecture" White Paper
describe this concept as well.
Figure 2.0 show an -example of this important concept. 2ØA is the
rendering of the entire dataset, where the densities of the brain are shared
with the skin and other structures. Although the dataset in 2Ø A has a
very detailed brain it is not visible since the skin is on top of it. Using
the
shape interpolation the user can define the mask 2ØB which loosely
contains the 'brain, this mask removes the ambiguities of which are the
densities, of the brain and allow the proper visualixation of. the brain,
2ØC.
Cedara Software Corp. , . . . . Page 26.
Figure 2.0 MR dataset in which the brain as been extracted. The
dataset A bas been masked with the binary object B to obtain the
visualization of the brain C. In this case Shape Interpolation was
used to generate the mask.
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
The benefits of this approach are two:
1. The definition of the mask is typically time-effective, even in the case
of figure 2.0 where the mask is created manually it takes about 1 "2
minutes to an trained user.
2. The mask doesn't have to precisely define the object, small error or
imprecision are tolerated In figure 2.0 the user doesn't have to
outline the brain in fine details, rather to outline where it is present
on a few slices.
Opacity and Segmentation
The segmentation setting which are used for the definition of the binary
object ( mask ) are related. to the opacity used for the visualization. For
example in figure 2.0 is the user lower the opacity enough it will
visualized the mask 2:0.B instead of the brain 2ØC.
'This will happens when the segmentation is based on some criteria and
the visualization on different ones.
Figure 2.1 shows an example in which the user click on the skull in order
to remove from the bed in the background. The application uses a region
growing based on the dataset thresholded by the opacity, and the result is
shown m figure 2.f.B."("the pipeline in figure 1.3 was used ). The mask
generated contaiiis'~the ~ skull of the dataset, only the bone densities
connected to the seed point. If the user lowered the opacity he will
eventually see the mask itself, 2.1.C:
B. ( C
Figure 2.1 In order to identify the object selected by the user the
application can 'threshold the dataset, based on the opacity, and
apply a binary region growing, as shown in image B. This method
Cedara Softarare Corp. , . Page 27
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
will generate an a mask C which is dependents on the opacity used
for the threshold.
In general'if a "mask has been generated using a threshold (tl,t~ can be
used with a' classification in which densities outside (tl,t~] have to set to
zero.
There is a very simple method to get around this limitation if necessary.
The mask can be regenerated each time the opacity is changed. This will
have a behavior more natural from the users' perspective as shown. in
figure 2.2.
i ~ i
Figure 2.2 The opacity mask can be recomputed ad each opacity
change. Image A shows: the original opacity settings, image B
shows the result of the threshould and seeding of the bone
structure. Once the opacity is lowered from B the mask is
recomputed, and the result is shown in figure C.
The pipeline used in 2.2 is shown in figure 2.3, the difference with the
pipeline used in 1.3 is that tlie'connection between the pvx opacity object
and then Thr3 object is kept:after that the mask is generated.
~3- T Seed3 Bv Bvf
.:..>,. . w
Ss Vol -~. Cproj _.i.
Figure 2.3 keeping ; the : connection of the Thr3 object with the
opacity pvx will allow regeneration of the mask on the fly.
Pipeline 2.3 will resolve the limitation imposed by the threshold used for
the region growing, but it will trigger a, possibly expensive, computation
for each opacity changes. It will hence limit the interactivity of this
Cedars Software Corp. , : Page 28
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
operation. The performances of rotation and colour changes will remain
utichat~ged.
Normal Replacement
As explained in the ."PrS 3D 'Rendering Architecture" White Paper the
shading is computed using the normal for each voxel. The normal is
approximated using the central difference or Sobel operators, which
utilize the neighbor densities of the voxel. Once the dataset is clipped
with a binary object the neighbor of the voxel on the surface changes,
since . the voxel outside the binary object are not utilized during the
projection. So the normal of these voxels has to be recomputed.
Figure 2.4 shows why this operation is necessary. When the bitvol clip s in
an homogeneous region; like. iri 2.4.A, the normals have several
directions, and so the ' cut surface will look uneven with several dark
points. Replacing the normal will make the surface look flat, as expected
by the user. . , ..
A B
Figure 2.4 Image A shows the cut surface if the normals
replacement doesn't take place. The surface looks uneven with
some dark point since the normal could .be zero in the
hotriogeneous region. When the normal is replaced with the binary
abject's normal, Image B, the surface looks flat as expected by the
user.
The replacement of the normal is necessary when the application uses the
binary object to cut the ~ anatomy, as in figure 2.4, not when it uses to
extract the anatomy as in figure 2.0 or 2.1.
Since the same binary object can be used for both purposes at the same
time the IAP renderer will replace the normal all the time. In situation s in
which the binary object is used to extract some feature the application
can simply dilate the binary mask to avoid the normal replacement. In
Czdara Software Core. . , page 29
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
this situation the mask is . based on a threshold or gray level region
growing and the dilation grill guarantee that the boundary of the mask
will be on transparent voxels and hence invisible. It is suggested a
dilation of 2 or 3 voxels.
Note that in the situation in which a binary object is used for extracting
the anatomy and cut at the same time, it is generated as intersection of
binary objects. The binary object used for the extraction has to be dilated
before the intersection. In this way the quality of the final render is
guaranteed while allowing the user to edit the object. Figure 2.5 shows an
example of this scenario.
r:
A g
Figure 2.5 Image A shows the skull which has been extracted as
described in 2.1...This object is cut using extrusion, as shown in 1.4.
The bitvol for the extcactioir is dilated, white the bitvol extruded is
not, image $ ..shows the final binary object superimposed on the
image A. The'cutting area is exacly as defined by the user, while
the mask is dilated to guarantee the image quality.
For unshaded volume rendering the dilation is not necessary and, it will
not affect. they final quality.
Mufti Classifications
As presented in the "PrS 3D Rendering Architecture" White Paper the
IAP processing server allows: ale definition of several classifications in
the same dataset, each one associate with its own binary object.
The binary objects can be overlapping, and so several classifications can
be defined in the same location. This represent the fact that in the,dataset
the one voxel taxi .contain several anatomical structures, and it is in the
Cedara Software Corp. ~ - Page 30
,._ . .. . . :, ..
CA 02365045 2001-12-14
~-~l
IAP PrS Segmentation Architecture
nature of the data. This is also the same reason for which these structures
shares densities. The next section will analyze this situation in details,
since it is relevant for the measurements.
The IAP processing. server supports two policies for the overlapping
classifications; and the application can extend with a . customized
extension.
Surface Rendering
The Surface Rendering engine renders directly the binary object. The IAP
processing server supports the visualization of several objects, each one
with its own rotation matrix. Other features include texture mapping of
the original gray level, and depth shading. Please refer to "PrS 3D
Rendering Architecture" White' Paper for a full description of the
functionality.
Cedara Software Corp. Page 31
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
Measurements and Segmentation
One. of the most important functions in any medical imaging application
is to quantify the abnormality of the anatomy. This infotrnation is used to
decide on the treatment to apply and to quantify the progress of the
treatment delivered.
The IAP processing server supports a large number of 2D and 3D
measurements. In this white paper we'll describe how the measurements
can be used in conjunction with segmentation.
The measurement model has to follow the visualization model in order
to be consistent with it and measure what the user is actually seeing,
binary measurements for Surface Rendering or gray level measurements
for Volume Rendering.
In this white paper we'll focus on surface and volume measurement since
they are the most commonly used. For a complete list of measurement
supported please refer to the Meas3 man page. .
Definition of the Measurements
During the sampling process the object is not necessarily aligned with the
main ayes. This cause that some voxels are partially occluded, so in other
terms the volume sampled by this voxel is only partially occupied by the
object being sampled. This is know as "Partial Volume Artifact" , and
cause that the object spans across voxels with several different densities.
Figure 3.0 shows graphically this situation. Object A is next to object B.
dA is the density of voxel 100% filled with object A which we'll consider
also hoinogelieous. dB is the density of voxel completely filled with object
B. We we'll also assume in this example that dA < dB and the value of the
density of the background is zero.
Figure 3.0 In this e~cample there are two objects, A and B, which
have been sampled along the grid shown in the picture.
Cedara Software Corp. Page 32
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
The voxels included by the object A can be divided in three different
regions:
~ Red Region : Vosel completely covered by object A. They have
density dA.
~ Yellow Region : voxel partially covered by object A and
background.
~ Blue Region : Voxel covered by object A and B.
Since in the scanning process the density of a vo$el is the average density
of 'the materials in the volume covered by the vogel, we can conclude
that the yellow densities have value less then dA, the blue range greater
then due. In the picture there is also highlighted the green area, which are
vosel.partially occluded with the material B. The range of densities of the
green vogels overlap with the red, yellow and blue. Graphically the
distribution of.the densities is shown in figure 2.1
den~itier
Figure 2.1 The yellow region has voxels partially occluded with
object A and the background, hence the density will be a weighted
sum of dA and background density, which is zero. Similarly the
blue region has voxel with densities between d~ and d$. The green
region has voxel partially occluded with object B and background;
hence they can have ,the full range of densities.
Depending on the model used by the application (binary or gray level)
the volume and the surface of the object A can be computed according
to the following rules:
Gray Level Binary
For each voxel The number of
is the voxel
Volume ~taset the volumeof the binary
object
covered by the representing
object the object
A has to be added.A are counted.
Surface Volume Rendering The vosel at
the
doesn't define bound of the
an bin
Cedara Software Corp. Page 33
CA 02365045 2001-12-14
IAP PrS Segmentation Architecture
surface for the object, object representing
so this measurement is voxel A are counted.
not applicable.
Gray Level
Let us assume that with the previous segmentation tools we can identify
all the voxels in which obrect A lays, even partially. Figure 2.2 shows the
object and also the histogram of the vogel in this area.
demiitier
Figure 2.2 The oulined area represent the voxels identified by the
segmentation process. The histogram of the voxel in this area is
shown in the left. The digerence between this histogram and the
one in figure 2.1 is that the voxel in the green region are not
present.
The difference of the histogram in figure 2.2 and 2.1 is that the vogel of
the green area are removed. As described in the section "Segnyentation as
a Mask" this correspond in removing the densities which are shared'
across the objects and,do not belong to the object segmented.
Note that inside xhe iiiaslc, the density of each vogel represents the
occlusion (percentage) of the Object A in the volume of. the voxel,
re~a_rdless its location.
To correctly visualize and measure the volume of the Object A we have
to define for each density the percentagc of the volume occupied. Since
the density is the average of the objects presented in the vowels, the
estimate is straightforward: ,
1. For the vosels in the yellow region the occlusion is simply density/d~
since the density of the background is zero.
2. For the vogels in the 'blue region the occlusion for a density is
(density-dB)/(dB-d~.
Setting the opacity according with these criteria guarantees good quality
of the volume rendered image. This opacity curve is shown in figure 2.3.
Cedara Software Core. . Page 34
CA 02365045 2001-12-14
3-~
IAP PrS Segmentation Architecture
voxels
Object B ( Opacity )
densities
Figure 2.3 Opacity curve for the voxels in the segmented region.
The opacity for each density represent the amount of object A In
the region covered by the voxel.
So in order to measure the volume the application can request the
histogram of the densities values in the segmented area and weigh each
density with the opacity:
(*) volume = ~dE~~~a h~togram(d) * opacity( d )
As explained in the section "Opacity and Segmentation" and "Gradient
Replacement" the application should dilate the bitvol is generated
thought a threshold and region growing. This operation will include
vosels semi-transparent and so it will not affect the measurement of the
volume as defined in (*)
It is not really possible to measure the surface on the- object, since the
borders of it are not known. However looking at picture 2.2 it is clear.
that it can be estimated with the yellow and blue areas since these are the
voxels in which the Border of the object lays.
The reader can find more information and more detailed mathematical
description in "Volume v Rendering" A. Drebin et altr. Computer
Graphics August 2988.
Binary
To segment an object : the application typically set two thresholds,
depending on those some part of the yellow and .blue region can be
included or excluded by the. segmentation. Figure 2.4 shows an example
of this situation. The area of the histogram between the two thfesholds is
the volume.of the object created.
densities
Cedara Software Corp. Page 35
CA 02365045 2001-12-14
3 ~
IAP PrS Segmentation tlrchitecture
Figure 2.4 When the application sets two threshold will include
some voxels in the blue and yellow region. The picture on the left
shows the voxels included in the segmentation, while the figure on
the right shows the histogram and the two thresholds, the area
covered by the histogram between the two thresholds (in green)
represent the volume of the segmented object.
The surface of the voxel can be simply estimate as the number of voxels
on the boundary of the binaiy object.
IAP Object Model
The example described in the previous section explain s how the
application can measure volume and surface on the binary and gray level
objects. In the real case scenario there are several objects involved and
overlapping, and they usually don't have constant densities. However the
same principle it , is still applicable to obtain an estimate of the
measurements:
The IAP processing server with the Meas3 Object supports all the
functionality,.required to perform the. measurements described in the
section. Meas3 computes ; histogram, average . value, max, min and
standard deviation of the stack. If a binary object is connected to it the
measurements will be limited to the area included.
Meas3 also computed the volume and surface of binary objects, it
supports two different policies for the surface measurements
1. Edges : measure the perimeters of each plane in the binary object
2. Coin : count the voxels in the bitvol which have at least one
neigbour not belonging to the bitvol (ie the voxel on the surface
of the bitvol ).
Cxdara Software Corp. . Page 36
CA 02365045 2001-12-14
4
Additional Description B
"PrS 3D Rendering Architecture - Part 1"
CA 02365045 2001-12-14
~/
PrS 3D Rendering Architecture White Paper
Introduction
Imaging Application Platform (IAP) is a well-established platform product
specifically targeted for medical imaging. Its goal is to accelerate the
development of applications for medical and biological applications. IAP has
been used since 1991 to build applications ranging from review stations to
advanced post-processing workstations. IAP supports a wide set of
functionality
including database, hardcopy, DICOM services, image processing, and
reconstruction. The design is based on a client/server architecture and each
class of functionality is implemented in a separate server. All the servers
are
based on a data reference pipeline model; an application instantiates objects
and
connects them together to obtain a live entity that responds to direct or
indirect
stimuli. This paper will focus on the image processing server (further
referred to
as the processing server or PrS) and, m parttcular, on the three-dimensional
(3D) rendering architecture.
Two different rendering engines are at the core of 3D in IAP: a well proven
and
established solid renderer, and an advanced volume renderer, called Mufti Mode
Renderer (NflvlR). These two renderers together support a very large set of
clinical applications. The integration between these two technologies is very
tight. Data structures can be exchanged between the two renderers making it
possible to share functionality such as reconstnic-~ion, visualization and
measurement. A set of tools is provided that allows interactive manipulation
of
volumes, variation of opacity and color, cut surfaces, camera, light positions
and
shading parameters. The motivation for this architecture is that our clinical
exposure has lead to the observation that there are many different rendering
techniques available, each of which is optimal for a different visualization
task.
It is rare that a clinical task does not benefit from combining several of
these
rendering techniques in a specific protocol. IAP also extends the benefits of
the
3D architecture wlth its infrastructure by making additional functionality
available to the MMR: extremely flexible adaptation to various memory
scenarios, support for mufti-threading,. and asynchronous behavior.
All this comes together in a state of the art platform product that can handle
not only best-case trade-show demonstrations but also real-Life clinical
scenarios
easily and efficiently. The Cedara 3D platform technology is powerful, robust,
and well-balanced and.it has no rivals on the market in terms of usage in the
field, clinical insight, and breadth of functionality.
Cedara Software Corp. Page 1
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Glossary
Volume Rendering A technique used to visualize 3D sampled data that does
not require any geometrical intermediate structure.
Surface Rendering A technique used to visualize 3D surfaces represented
by either polygons or voxels that have been previously
extracted from a sampled dataset.
Interpolation A set of techniques used to generate missing data
between known samples.
Voxel A 3D discrete sample.
Shape Interpolation A,n interpolation technique for binary objects that allows
users to smoothly connect arbitrary contours.
Multiplanar or Curved Arbitrary cross-sections of a 3D sampled dataset.
Reformatting
MIP Maximum Intensity Projection. A visualization
technique that projects the maximaim value along the
viewing direction. Typically used for visualizing
angiographic daxa.
ROI Region of Interest. An irregular region which includes
only the voxels that have to be processed.
Page 2 ~ Cedara Software Corp.
. CA 02365045 2001-12-14
.~-3
PrS 3D Rendering Architecture White Paper
Application Scenarios
Binary and Gray Level Functionality
For many years a controversy has raged about which technology is better for
visualization of biological data: volume rendering or surface rendering. Each
technique has advantages and drawbacks. (for details, see "Error! Reference
source not found." on page Error! Bookmark not defined..) Which
technique is best for a specific rendering task is a choice that is
deliberately
deferred to application designers and clinicians. In fact, the design of the
processing server makes it possible to combine the best of both techniques.
The processing server provides a unifying framework where data srxuccures
used by these two technologies can be easily shared and exchanged. In fact,
IAP
is designed for visual data processing where the primary sources of data are
sampled image datasets. For this reason, all our data strictures are voxel-
based.
The two most important are binary solids and gray level slice stacks. Several
objects in our visualization pipelines accept both types of data, providing a
high
level of flexibility and interoperability. For example, a binary solid can be
directly visualized or used as a mask for the volume rendering of a gray level
slice stack. Conversely, a graylevel slice stack can be rendered directly and
also
used to texture map the rendering of a solid object or to provide gradient
information for gray level gradient shading.
A gray level slice stack is generally composed of a set of parallel cross-
sectional
images acquired by a scanner, and can be arbitrarily spaced and offset
relative to
each other. Several pixel types are supported from 8-bit unsigned to 16-bit
signed with a floating point scaling factor. Planar, arbitrary, and curved
reformats (with user-defined thickness) are available for slice stacks and
binary
solids. Slice stacks can also be interpolated to generate isotropic volumes or
resampled ax a different resolution. They can be volume rendered with a
variety
of compositing modes, such as Ma~mum Intensity Projection (MIP), Shaded
Volume Rendering, and Unshaded Volume Rendering. ' Multiple stacks can be
registered and rendered together, each with a different compositing mode.
To generate binary solids, a wide range of segmentation operations are
provided: simple thresholding, geometrical ROIs, seeding, morphological
operations, etc. Several of these operations are available in both 2D and 3D.
Binary solids can be reconstructed from a slice stack, and shape interpolation
can be applied during the reconstruction phase. When binary solids have been
reconstructed, logic operations (e.g., intersection) are available together
with
disarticulation and cleanup functionality. Texture mapping and arbitrary
visibility fikers are also available for binary solids.
Cedara Software Corp. Page 3
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
C ' 'cal Scenarios
The following list is not meant to be exhaustive but it does capture the most
significant clinical requirements for a complete rendering engine. We want to
underline the requirements of a real-life 3D rendering system for medical
imaging because the day to-day clinical situations are normally quite
different
from the best-case approach normally shown in demonstrations at trade shows.
The Cedara platform solution has no rivals in the market in terms of usage in
the field, clinical insight and breadth of functionality.
Tumors
Tumors disturb surrounding vascular stru~re ands in some cases, do not have
well defined boundaries. Consequently, a set of rendering options needs to be
available for them which can be mnced in a single image, including surface
rendering (allowing volume calculations and providing semi-transparent
surfaces
at various levels of translucency, representation of segmentation confidence,
etc.), volume rendering, and vasculature rendering techniques such as MII'. An
image in which multiple rendering modes are used should still allow for the
full
range of measurement capabilities. For example, it should. be possible to make
the skin and skull translucent to allow a tumor to be seen, while still
allowing
measurements to be made along the skin surface. ,
Figure 1- Volume rendered tumors.
In (a), the skin is semi-transparent and allows the user to see the tumor and
its relationship with the vasculature. In (b), the location of the tumor is
shown relative to the brain and vasculature.
a.
Display of Correlated Data from Different Modalities
There is a need to display and explore data from multiple acquisitions in a
singe
image (also called mufti-channel data). Some of the possibdiues include: pre-
and post-operative data, metabolic PET data with higher resolution MR.
anatomic info, pre- and post-contrast studies, MR and MRA, etc.
Page 4 Software Corps
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
The renderer is able to fuse them during the ray traversal, and with a real 3D
fusion, not just a 2D overlay of the images.
Figure 2 - Renderings of different modalities.
In (a), the rendering of an ultrasound kidney dataset~ The red shows the
visualization of the power mode; the gray shows the visualization of the B
mode. Data provided by Sonoline Elegra. In (b), the volume rendering of a
liver study. The dataset was acquired with a Hawkeye scanner. The low
resolution CT shows the anatomy while the Spect data highlights the hot
spots. Data provided by Rambam Hospital, Israel. In (c), the MRA
provides the details of the vasculature in red, while the MRI provides the
details of the anatomy. (In this image, only the brain has been rendered.)
b.
c.
Cedars Software Corp. Page 5
CA 02365045 2001-12-14
~.- 6
PrS, 3D Rendering Architecture White Paper
Dental Pac~c'age
A dental package requites orthogonal, oblique, and curved reformatting plus
accurate measurement capabilities. In addiuoii, our experience suggests that
3D
rendering should be part of a dental package.
In 3D, surface rendering with cut surfaces corresponding to the orthogonal,
oblique and curved dental reformats are required. In addition the ability to
label
the surface of a 3D object with lines corresponding to the intersection of the
object surface with the reformat planes would be useful Since a dental package
is typically used to provide information on where and haw to insert dental
implants, it should be possible to display geome~c implant models in dental 3D
images and to obtain suing and drilling trajectory. information through
extremely accurate measurements and surgical siiriulation. The discussion
above
on prosthesis design applies here also.
Large Size Dataset
Modern scanners are'able rto acquire a large amount of data, for example, a
normal sto~;a spiral: CT scan can easily be several hundreds megabytes. The
system mu"xt be able to handle this study without any performance penalty and
without slowing down the work$ow of the radiologist. The processing server
can accomplish this since it directly manages the buffers and optimizes, or
completely avoids, swapping these buffers to disk. For more information on
this functionality, please refer to the Memory Management section in the PrS
White Paper.
Since volume rendering requires large data yes, Memory Management is
extremely relevant in these scenarios. Figure 4 shows the rendering of the CT
dataset of the visible human project. The processing server allows you to
rotate
~s ~~~ ~~°~ an3' ~PP~g on disk after the pre-processing has been
completed.
Page 6 Cedara Software Corp.
Figure 3 - Application of a. curvilinear reformat for a dental package.
CA 02365045 2001-12-14
'~ 7
PrS 3D Rendering Architecture ~e Paper
Figure 4 - Volume rendering of the visible human project,
Since the processing server performs the interpolation before the rendering,
the rendered dataset has a size of Sl2~cS12x1024,12 bits per pixel-. Including
gradient information, the dataset size is-1 Gigabyte. Since the processing
server performs some compression of the data and optimizes the memory's
buffer, there is no swapping during rotation, even on a 1 Gigabyte system.
Cedara Software Corp. Page 7
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Volume Rendering
The Basics
Volume rendering is a flexible technique for virsualizing sampled image data
(e.g., CT, MR, Ultrasound, Nuclear Medicine). The key benefit of this
technique
is the ability to display the sampled data directly without using a
geometrical
representation and without the need for segmentation. This makes all the
sampled data available duringahe visualization and, by using.a variable
opacity
transfer function, allows inner structures to appear semi-transparent.
The process of generating an image can be described intuitively using the ray-
casting idea. Casting a ray from the observer through .the volume generates
each
pixel m the final image. Samples have to be interpolated along the ray during
the
traversal. Each sample is classified and the image generation process is a
numerical approximation of the volume rendering integral.
Figure 5 - Conceptual illustration of volume rendering.
~d~ :,~ viewer
'i
li~
Voliim
Imege
In volume rerideting, a fundamental role is played by the classification. The
classification defines the color and opacity of each voxel in the dataset. The
opacity defines "how much" of the vaxel is visible by associating a value from
0
(fully transparent) to 1 (fully opaque). Using a continuous range of values
avoids
aliasing because it is not a bmaiy threshold. The color allows the user to
distinguish between the densities (that represent different tissues in a CT
study,
for example) in the 3D image. Figure 6 shows three opacity settings for the
same CT dataset.
A classification in which each voxel. depends only on the density is called
global, while a classification in which each voxel depends on its position and
density is called local. A global classification function works pretty well
with CT
data since each tissue is characterized by a range of Hounsfield units. In
MRI,
the global classification function has a very limited applicability as shown
in
Figure 7. To handle MR data properly, a local classification is necessary.
Page 8 . Cxdara Software Corp.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 6 - Different opacity settings for a CT dataset.
The first row shows the volume rendered image, while the second row
shows the histogram plus the opacity curve. Different colors have been
used to differentiate the soft tissue and the bone in the 3D view.
I
' '
Figure 7 - Different opacity settings for an MR dataset.
In MR, the underling'physics is not compatible with the global opacity
transfer function; it is not possible to select different tissues just by
changing' it. As will be explained in the next section, the processing server
overcomes thisproblem by allowing the application to use a local
classification. .
,., _ . , ~ ,
t :~ . , ..,..
s., ,
.. . i
Cedars Software Corp. Page 9
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
The IAP renderer also takes advantage of optimization techniques that leverage
several view-independent steps of the rendering. Specifically, some of the
view-
independent processing optimizarions implemented are:
~ interpolation;
~ gradient generation; and
background suppression.
Naturally, all these optimizations are achieved at the cost of the initial
computation and memory necessary to store this pre-calculated information
and, therefore, can be selectively disabled depending on the available
configuration. The application developer also has several controllable
configurations available that allows the developer to gracefully decrease the
amount of informaxion cached depending on the system resources avat'lable.
Rendering Modes
As we saw, the general model that the IAP volume renderer uses to generate a
projection is to composite all the voxels that lie "behind" a pixel in the
image
plane. The eompositmg process differs for every rendering mode, and the most
unportant are illustrated here.
In the compositing process, the following variables are involved:
~ Crp(d) opacity for density d. The opacity values are from 0.0 (complete
transparent) to 1.0 (fully opaque).
~ Color(d) color for density d.
~ V(xy,z) density in the volume at the location x, y, z.
~ I(u,v) intensity of output~image at pixel location u,v.
~ (xyz) = SampleLocation(Ray, i) this notation represents the computation
of the i-th sample point along the ray. 'This can be accomplished with
nearest neig)abor or trilinear interpolation.
~ Ray = Compu,v); this notation represent the computation' of the ray
"Ray.. passing through the pixel u,v in the image plane. Typically, this
involve the definition of the sampling step and the directlon,of the ray.
~ Normal(xyz) : normal of the voxel at location xyz. Typically, this is
approximated with a central difference operator.
~ L light vector.
~ ° Represents the dot product
Figure 8 shows a graphical representation of these values.
Page 10 Cedara Software Corp.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper ~ ~ "'
Figure 8 . Values involved in the ray casting process.
Ray = ComputeRay(u,v)
Normal
r~~
vector L
i-th samq~le point
SampleL,ocation(Ray
Maximum Intensity Projection (MIP)
This is the pseudo-code for the MIP:
for every pixel u,v in I {
ray = ComputeRay(u,v)
for every sample point i along ray {
(x,y,z)= SampleLocation(ray, i )
if( I(u,v) < V(x,y,z) )
I'(u,v) ~ V(x,Y.z)
}
}
Density Volume Rendering.(DV'R)
This is the pseudo code for DVR (referred to as "Multicolor" in the IAP man
Pas).
for every pixel u,v in I {
ray ~. ComputeRay(u,v)
ray opacity = 0.0
for every sample point~i along ray ~
(x,y,z)= SampleLocation(ray, i )
alpha ~ ( 1.0 - ray opacity ) * op(V(x,y,z))
I(u,.v) += color(V(x,y,z)) * alpha
ray opacity += alpha
}
Cedars Software Corp. Page 11
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper ~~'
Shaded VoIW ne Rendering
This is the pseudo code for DVR (referred to as "Shaded Multicolor" in the
IAP man pages):
for every pixel u,v in I
ray = ComputeRay(u,v)
ray opacity = 0.0
for every sample point i along ray
(x,y;z)= SampleLocation(ray, i )
alpha = ( 1.0 - ray opacity ) * op(V(x,y,z))
shade = Normal(x,y,z)o L
I(u,v) += color(V(x,y,z)) * alpha * shade
ray opacity += alpha
Figure 9 shows the same dataset rendered with these three rendering modes.
Figure 9 - Rendering modes.
MIP
Object Model
For a generic description of the IAP object model, please refer to the PrS
White
Paper. In this section, we illustrate in detail the object involved in the
volume
rendering pipeline.
In Figure 10, you can see the IAP volume rendering pipeline. The original
slices
are kept in a number of raster objects (Ras). Those Ras objects are collected
in a
singe slice stack (Ss) and then passed to the interpolator object (Sirnerp).
Here,
the developer has the choice of several irnerpolation techniques ('~.e.,
cubic,
linear, nearest neighbor). The output of the Smtecp then goes into the view
independent preprocessor (Vol) and then finally to the projector object
(Cproj).
From that point on, the pipeline becomes purely rwo-dimensional If Cpro~ is
Page 12 Cedara Software Corp.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
used: in gray scale mode (no color table applied during the projection, like
in
Figure 9), the application can apply window/level to the-output image in V2.
If
Cpro~ is m color mode, V2 will ignore the window/level setting, 'The
application
can add windowllevel in color mode using a dynamic extension of the Filter2
object, as reported in Appendix C.
Figure 10 - Pipeline used for vohune rendering.
One of the keys to correct interpretation of a volume dataset is motion. The
amount of data in today's clinical dataset is so high that it is very
difficult to
perceive all the details in a static picture. I-fistorically, this has been
done using a
pre-recorded cine-loop, but now it can be done by uiteractively changing the
view position. Apart from the raw speed of the algorithm, it is also
convenient
to enhance the "perceived speed" of a volume rendering algorithm by using a
progressive refinement update. Progressive refinement can be implemented in
several ways in IAP but the easiest and most common way to do it for volume
rendering is to set up a second downsampled data-stream The two branches,
each processing the input data.at different resolutions, are rendered in
alternation.
Figure 11 shows how the pipeline has to be modified in order to achieve
progress refinement in IAP. Two volume rendering pipelines (Sinterp, Volume,
Cpro~) are set in parallel and are fed with the same stack and output m the
same
window. One of these pipelines, called High Speed, uses a down sampled
dataset (the downsampling is performed in Sii~terp}. This allows rendering at
a
very high rate (more than 10 frames per second) even on a general purpose PC.
The second pipeline is called High Quality. It renders the original dataset
usually interpolated with a linear or cubic kernel
Cedars Software Corp. Page 13
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
I-figh Speed and High Quality are j oined at the progressive refinement object
(Pref): The High Quality pipeline is always interrupttble to guarantee
application
responsiveness, while the I-fig~i. Speed pipecLine is .normally in atomic mode
(non-
intenuptible) and executes at a higher priority: The application developer has
control of all these options, and can alter the settings depending on the
requirements. For example, if a cine-loop has to be generated, the High Speed
pipeline is disconnected because the intermediate renderings are not used in
this
scenario. For rendering modes that require opacity and color control, a set of
Pvx objects speafying a pixel value transformation can be connected to Vol.
The transformation is specified independently from the.pixel type and can be a
simplified linear ramp or full piece-arse linear. Using the full piece-wise
linear
specification, the application writer can implement several different types of
opacity and color editors specific to each medical imaging modality. Another
common addition to the volume rendering pipeline is a cut plane object, r -~m
3
or Binary object. These objects specify the region of the stack that have to
be
projected and can be connected either to Sinterp or Vol.
The processing server can also render a multi-channel dataset together. As
discussed in "Clinical Scenarios" on page 1, this feature is very important in
modalities such as MR and ultrasound, which can -acqitire multiple datasets.
The
renderer will interleave the datasets during the projection, which guarantees
that
it is a real 3D fusion, rather than a 2D overlay. Examples of this
functionality is
in Figure 2 on page 5. Several stack are rendered together simply connecting
several Volume object to the same Cproj object, as shown in Figure 12.
As in the case of one Volume object the developer can use an High Speed
pipeline and High Quality pipeline to enhance the interactivity.
Page 14 Cedars Software Corp.
Figure 11- High-speed/high quality pipelines.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Advanced Features
As we have seen .in "Clinical Scenarios" on page 4, a renderer needs to have a
much higher level of sophistication to be effective in clinical practice. The
processing server provides functionality which is beyond the "standard" volume
rendering. The most relevant are presented here.
Clipping Tools
The user in 3D visualization is usually interested in the relationship between
the
anatomical structures. In order to investigate that relationship, it is
necessary
that the application be allowed to clip part of the dataset to reveal the
internal
structure. Our experience shows that there are two kinds of clipping.
Permanent - part of the dataset (normally irrelevant structures) is
permanently removed
Temporary - part of the dataset (normally relevant anatomy) is temporarily
removed to reveal internal structures.
To accommodate these requu~ements, and in order to maximize performance,
the processing server implements two kinds of clipping:
' ~PP~ ~ pre-processing time.
This feature.unplements "permanexlt"' clipping. The voxels clipped in this
stage are not included in the data structure used by the renderer, and are
not even computed, if possible.
'. CJippmg at rendering time.
This feature implements "temporary" clipping. The voxels are only skipped
during the rendering process and are kept m the rendering data structure.
Using clipping at pre-processing time optimizes rendering performance, but a~
changes to the clipping region will require a (potentially) expensive pre-
Cedars Software Corp. Page 15
Figure 12 - Rendering of a mufti-channel daxaset.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
processing which usually prevents using this feature interactively. The
clippnig
at rendering time instead allows the application to interactively clip the
dataset,
since the clipping is performed skipping some voxel during the projection and
so changing this regron if fully interactive.
Usually the application generates two types of clipping regions:
~ Geometrical Clipping regions (e.g., generated by a bounding box or oblique
plane).
~ ~' ~PP~ regrons (e.g., generated by outlining an anatomical
r~on).
The processing server supports these two kinds of clipping both at pre-
processing time and rendering time.
Preprocessing Rendering .
The object that 'The render. object,
performs Cproj, accepts
the pre-processing,a bounding box as input.
Vol, In the
accepts a boundingcase of a mufti-channel
box dataset,
for clippin
g. each volume. can be
Geometrical clipped
independently. Using
the of
object, it is also
possrble to clip
each volume with an
oblique
plane or a box rotated
with
respect to the orthogonal
axes.
The Vol object Any binary volume can
accepts a be used a
bitvol as input.clipping region. The
Vol will processing
preprocess only server also allows
the voxels to rnteractively
included in the
bit
L
vo translating this region.
~~ ~PPmg ~ r~~.ng time is very useful in several situations, but
P~Wing M'IP rendering. Normally several anatomical structures
overlap ru the MII' image so interactively moving a clipping region (e.g.,
rylinder; oblique slab, or sphere) can clarify the abilities on the rmage. See
Frgure 13 for an example.
Another very common way to clip the dataset is based on the density. For
example, on a CT scan the densities below the Hounsfield value of the water
are
background and hence have no diagnostic information. Even in this cask the
processing server supports the clipprtig, at pre-processing time (using the
Vol
object) and at rendering time (simply changing the opacity):
Page 16 , .. ... Cedara Software Corp.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 13 - Interactive clipping on the MIP view,
The user can interactively move the semi-sphere that clips the dataset and
identify the region of the AVM. Usually the clipping region is also overlaid
in the orthogonal MIP views.
Local Classification
The global classification is a very simple model and has limited
applicability. As
shown in Figure 7, the MR dataset does not allow the clinician to visualize
arty
specific tissue or organ. In the CTA study, it is not possible to distinguish
between contrast agent and bone. Both cases significantly reduce the ability
to
understand the anatomy. In order to overcome these problems, it is necessary
to use the local classification.
By definition; the local classification is a function that assigns color and
opacity
to each voxel in the dataset based on its density and location. In the reality
the
color is associate to the voxel depending on the anatomical suucriire to which
it
belongs. So it is possible to split the dataset in several regions
(representing the
anatonucal structures) and assigning a colomiap and opacity to each one of
them.
In other temys the local classification is implemented with a set of
classifications, each one applied to a different r~ion of the dataset. For
this
reason is very often referred as Multiple Classification.
The processing server supports Multiple Classification is a very easy and
efficient manner. The application.can create several binary objects (the
regions)
and apply for each one of them a different classification. The Multiple
Classification is dixectly supported by the MclVo1 object which allow to apply
several classifications to the output of the Vol object. Figure 14 shows the
case
in which two classifications are defined in the same stack.
Cedara Software Core. -. Page 17
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White papa
Figure 14 - Pipeline to use two classifications in the same volume.
Each classification (C1) is defined in the area of the bitvol ($v) object
associated
with it. The Bvf~ob~ect has beers added to the pipeline to allow it to
interactively
translate the area of the classification to obtain, for example, features as
shown
in Figure 13.
Since all the classifications are sharing the same data stn~cture (produced by
VoI) the amount of memory required is independent of the number of
classifications. This design also allows you to add/remove/move a
classification
with minimal pre-processing.
The MclVo1 object allows controlling of the classification in the area in
which
one or more regions are overlapping. This is implemented through a set of
policies that can be extended with run-time extension. This feature is a
powerful
tool for the application which knows in advance what each classification
represents, as shown in "Error! Reference source not found".
Figure 15 shows how this pipeline can produce the correct result on a GTA
study. The aorta is filled anth a contrast agent that has the same densities
as the
bone. Defining,one bitvol (shown in Figure 15a) which defines the area of the
aorta allows the application to apply the proper classification to the stack.
Note
that the bitvol does not have to exactly match the structure of interest, but
rather loosely contains it or excludes the obstructing anatomies. 'The direct
volume rendering process, by using an appropriate opacity transfer function,
will allow. for the visualization of all the details contained m the bitvols.
This approach can be applied to a variety of different modalities. In Figure 1
on
page 4, ~t is applied to an MR study. The main benefit of our approach is that
the creation of the bitvols is very time effective. Please refer to the "IAP
PrS
Segmentation Architecture" V~lhite Paper for a complete list of the
functionalities.
Page 18 . . . Cedara Software Corp.
s~ B~ ci
U
CA 02365045 2001-12-14
~-~9
prS 3D Rendering Architecture White Paper
Pigure 15 - Local classification applied to a CTA study.
In (a), Bitvol is used to define the region of the aorta. In (b), CTA of the
abdominal aorta is classified using local classification.
The processing server functionality for the creation/manipulation of bitvols
includes:
~ Shape Interpolation - Allows the clinician to reconsrxuct a smooth binary
obiect from a set of 2D ROIs. It is very effective for eictracting complex
organs that have an,irregluar shape lie the brain. Figure 16 shows how this
functionality can be applied
~ Extrusion:.- Allows the clinician to exude a. . 2D ROI along aa3r direction.
It
is very uiseful, for: eliminating structures that obscure relevant pans of the
anatomy, and is often used m ultrasound applications.
~ Seeding- 3D region growing in the binary object. This functionality very
naturally eliminates nose or relevant structures around the area of interest.
~ Disarticularion - A kind of region growing which allows the clinician to
separate objects which are loosely connected, and represent different
anatomical structures.
~ Dilation/Erosion - Each binary object can be dilated or eroded in 3D with
an arbitrary number of pixels for each axis.
~ Union, Intersection and complement of the binary objects.
~ Gray Level region garowing - Allows the clinician to segment the object
without requiring the knowledge of thresholds.
The processing~server'does not limit the manipulation of the binary object to
these tools. If the application has the knowledge of the region to define its
own
segmentation (e.g., using axi anatomical atlas can delineate the brain and
generate directly the bitvol used in Figure 16); it is straightforward
introducing
this bitvol in the rendering pipeline.
Cedars Software Corp. Page 19
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper ..--
Figure 16 - Example of shape interpolation.
To extract a complex organ, the user roughly outlines the brain contours on
a few slices and then shape interpolation reconstruction generates the
bitvol. The outlining of the brain does not have to be precise, but just
includes the region of interest. The volume rendering process, with a
proper opacity table, will extract the details on the brain.
The functionality supported by MclVol can also be used to arbitrarily cut the
dataset and texturewiap'the origiua~l gray level on the cutting surface, One
of
the main benefits of this feature is that the gray level from the MPRs can be
embedded in the 3D visualization. Figure 17 shows how this can be done in two
different ways: In Figure 17a, the MPR planes are visualized with the original
gray level information. This can be used to correlate the MPR shown by the
application in the dataset. In Figure 17b, a cut into the dataset shows the
depth
of the fracture on the bone.
Figure.18 shows another example of this feature: In this case, a spherical
classification is used to reveal the bone structures in a CT dataset (Figure
18a)
or the mening~oma in an MR dataset (Figure 18b).
Page 20 . .; i . . . , . Software Core.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper 0
Figure 17 - Example of embedding gray level from the MPRs.
The functionality of McIVoI allows you to embed the information from the
MPRs in the visualization. This can be used for showing the location of the
MPRs in the space of investigating the depth of some structure.
a. b.
Figure 18 - Example of cligping achievable with MclVol.
In (a), there are .two classifications in the CT dataset: the skin which is
applied to the whole dataset;. and the bone which is applied to the sphere.
In (b), there are, four different classifications (for details, see "Error!
Reference soaijice not, found;";).
..:~ . . - .
a_ ,~ ~rK., . ..,., ,. b
To minimize memory consumption, MclVol supports. mukiple doovnstreaal
connectionsso that several Cproj objects can share the same data structure. It
is
possible to associate a group~of classifications to a specific Cproj, so each
Cproj
object can display a different set of objects. Note that in order to minimiT.e
the
memory corisrimption, Volume keeps only one copy of the processed dataset.
Thus, if some Cproj objects connected to MclVol are using Shaded Volume
Rendering, while others are using MIP or Unshaded Volume Rendering, the
performances will be severely affected since Volume will remove the data
structure of the gradient when the MIP o* Unshaded Volume Rendering is
scheduled for comlnuation, and it will recoinpute it when the Shaded Volume
Cedars Software Corp. Page 21
CA 02365045 2001-12-14
PrS 3D.Rendering Architecture White Paper
Rendering is scheduled In this scenario, we suggest you use two Vols, and two
MclVols, one for Shaded and one for Unshaded Volume Rendering, and
connect the Cproj to the MclVo1 depending on the rendering mode used.
Page 22 Software Corp.
CA 02365045 2001-12-14
Additional Description C
"PrS 3D Rendering Architecture - Part 2"
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper ,.-!
..
Open M1VIR
There is a class of applications that requires anatomical data to be rendered
with
synthetic objects, usually defined by polygons. Typically, applications
oriented
for planning (i.e., radiotherapy or surgery) require this feature.
The processing. server supports this functionality in a very flexible way. The
application can provide to the renderer images with associated Z buffer that
has
to be embedded.uz the scene. The Z buffer stores for each pixel in the image
the distance from the polygon to the image plane. The Z buffer is widely used
in compuxer graphics and supported by several libraries. The application has
the
freedom to choose the 3D library, and, if necessary, write their oavn.
Figure 1 shows an example of this funaronality. The cone, which has been.
rendered using OpenGL, is embedded in the dataset. Notice that some voxels '
in front of the cone are semi transparent, while other voxels behind the cone
are not rendered at all. This functionality doesn't require any pre-processing
so
changing the geometry can be fully interactive.
The Z buffer is a very simple and effective way to embed geometry in the
Volume, but imposes some limitations. For each pixel in the image there must
be only one polygon projected onto it. This restricts the application to fully
opaque polygons or non-overlapping semi-transparent polygons.
Cedars Software Corp. Page 3
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 1- Geometry embedded in the volumetric dataset.
Opacity Modulation
One technique widely used in volume rendering to enhance the quality of the
image is changing the opacity of each voxel depending on the magnitude of the
gradient at the voxel's location. This technique, called opacity modulation,
allows the user to enhance the transition between the anatomical structures
(characterized by strong magnitude) and suppress homogeneous regions in the
dataset. It greatly improves the effect of translucenry because when the
homogeneous regions are rendered with low opacity, they tend to became dark
and suppress details. Using opacity modulation, the contribution of these
regions can be completely suppressed.
The processing server supports opacity modulation in a very flexible manner. A
modulation table, which defines a multiplication factor for the opacity of
each
voxel depending on its gradient, is defined for each range of densities. In
Figure
2, for example, the modulation has been applied only to the soft tissue
densities
in the CTA dataset.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 2 - Opacity modulation used to enhance translucency.
Suppressing the contn'bution of a homogeneous region, which usually
appears dark with low opacity, allows the user to visualize the inner details
of the data.
The effect of the modulation is also evident in Figure 3 which shows two
slices
of the dataset with the same settings used in Figure 2. The bone strucnxre is
the
same in both images, while in Figuie 20b, only the border of the soft tissue
is
visible (characterized by_strong magnitude), while the inner pan has been
removed (characterized .by low magnitude).
Figure 3 -,Cross-section of the dataset using the settings from Figure 20.
b.
As mentioned before, opacity modulation can also enhance details is the
dataset. In Figure 4; the vasculature rendered is enhanced by increasing the
opadty at the border of the vessels characterized by high gradient magnitude.
As you can see, the vessel in Figure ?.2b appears more solid and better
defined
Cedars Software Corp. Page 5
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 4 - Opacitymodulation used to enhance vasculature visualization.
a. b.
One more application of opacity modulation is to minimize the effect of "color
bleeding" due to the partial volume artifact in the CTA dataset. In this case,
the
opa«ty ion is applied to the densities of the contrast agent and the
strong magnitude is suppressed. Typically, the user tunes this value for each
specific dataset. Figure 5 shows an example of a CTA of the neck region.
;,,...t;, .
Figure 5 - Opacity modulation used to. sugpress "color bleeding".
In (b) the opacitymodulation is used to suppress the high gradient
magnitude associated~;witli thevegrttrast agent densities.
a. ° b.
In this situation, opacity modulation represents a good tradeoff between usei
intervention and image To use the local classification, the user has to
segment the carotid from~data~et; while in this case, the user has only to
change a slider to interactively adjust for the best image quality. Note: that
when
using this technique, it is not possible to measure the volume of the carotid,
since the system does not have .any information about the object. Measurement
requires the segmentation of the anatomical structure, and. hence mulriple
classification. Please refer to the "IAP PrS Segmentaxion Architecture" White
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Paper for a detailed description of the measurements and their relationship
with
the local classification.
The processing server allows the application to set a modulation table for
each
density range of each Volume object rendered in the scene. Currently, opacity
modulation is supported when the input rasters are 8 bit or 12 bit.
Mixing of Rendering Modes
The application usually chooses .the rendering method based on the feature
that
the user is looking for in the dataset. Typically, Ma' is used to show the
vasculature Lori an MR dataset, while Unshaded or Shaded Volume Rendering is
more appropriate to show the anatomy (like brain or tumor). In some
situations,
both these features have to be visualized together, hence different rendering
modes have to be mixed.
The processing server provides. this functionality because we have seen some
clinical benefit. For example, Figure 6 shows a CTA dataset in which two
regions have been classified; the cararid and the bone structure. In Figure
24a,
both regions are rendered with Shaded Volume Rendering. The user can
appreciate the shape of the anatomy but cannot see the calcification inside
the
carotid. In Figure 24b, the carotid is. rendered using MIP while the bone
using
Shaded Volume Rendering. In this image, the caldfication is easily visible.
The
processing server merges the two regions during the rendering, not as a 2D
overlay. This guarantees.that.the relative positions of the anatomical
structures
are correct.
Figure 6 - CTA study showing mixed rendering.
In (a), the, carotid and bone are both rendered with Shaded Volume
Rendering: In:(b), the carotid is rendered with MIP and the bone with
Shaded Volume Rendering. ,.
a: b.
Another situation in which this functionality can be used is with a muhi
channel
dataset. In Figure 2.21 an NM and CT dataset are rendered together. In this
Cedars Software Corp. ' Page 7
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
example the user is looking for the "hot spots" in the NM dataset and their
relationship with the surrounding anatomy. The hot spots are by their nature
visualized using Ma' rendering, while the anatomy in the CT dataset can be
properly rendered.using Shaded Volume Rendering. Figure 2.21B shows the
mixing rendering mode. The hot spots are very visible in the body, wht~e in
Figure 2.21A depicts both data set rendered using Shaded mode, in this case
the
, location of the "hot spot" in the body is not as clear.
Cun ently, the processing server.allows the fusion of only MIP with Shaded or
Unshaded Volume Rendering.
Figure 7 - Different modalities and rendering modes.
In (a), a NM dataset and a CT dataset are both rendered using Shaded
Volume Rendering.~In'(b);~ahe NM dataset is rendered using MIP and the
CT dataset is rendered' using Shaded Volume Rendering. Data provided by
Rambam Hospital, Israel:
.;. , , . : .::..
b:
Coordinate .Query. .
In the application, there is, often the need to correlate the 3D images with
the
2D MPRs: For example, Figure 8 shows that the user can see the stenosis in the
MlP vieavbutneeds the MPR to estimate the degree of stenosis.
. <~.. ~::. ::,~ ... :. ; : :.
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Since the voxel can .be semi-transparent in.volume rendezing, selecting a
point
in the 3D view does not'uniquely determine a location inside the stack; rather
it
determines a list,of voxels;whi'ch'contribute to the color of the pixel
selected.
Several algorithms can be usedto select the voxel in this list to be
considered as
the "selected:'~oint'" ~o~'example, the first non-transparent voxel can be
selected or~tri~;~~st~;opaque~oxel can be used instead. Our experience shows
that the a~gon;.thm described ixl "A method for specifying 3D interested
regions
on Volun~e.Rendeied hnages~ and its evaluation for Virtual Endoscopy System",
Tcryofumi~SAITO, Kensaku MORI, Yasuhito SLTENGA, jun-ichi
HASEGAWA;~jun-iehiro.TO>ZIWAKI, and Kazuhiro KATADA, CARS2000
San Francisco, works v~~r,",w~ell:.:The algorithm selects the voxel than
contribuxes
most to the color of the selected piitel: It is fully automatic (no threshold
requested) and it works in a very intuitive way. "Error! Reference source not
found." shows an implementatio~~of this algorithm
This functionality can be also used to create a 3D marker. When a specific
point
is selected in stack space, the processing server can track its position
across
rotations and the application can'query the list of voxels that contribute to
that
pixel. The application can then verify if the point is the one visible. Since
the
marker is rendered as a 2D_:overlay, the image thickness of the line does not
increase when the .image is z~med. See Figure 9.
,;;,.._. .;' ~:.: :.. ~.:.w:
..._ ~ .~'.n~.~,t_.:1.,.
:;v". ~.'., x
.-".: :.;;',... y. . , '.
. . , n ,vt:~~S:,i...
,.
Cedars Software Core. Page 9
Figure 8 - Localization of an anatomical point using a MIP view:
The user clicks on the stenosis visible on the MIP view (note the position
of the pointer). The application then moves the MPR planes to that
location to allow 'the user to estimate the degree of stenosis..
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 9 - ~Exainple: of a 3D marker.
A specific:point at the base of the orbit is tracked during the interactive
rotation of.the dataset. In (a), the point is visible to the user and hence
the
marker is rendered and drawn in red. In (b), the visibility of the point is
occluded by bony structure~; hence the marker is drawn in blue.
a. :w. .~_~.,::. , . . . b.
The processing server xeturns all ~of the voxels projected in the selected
pixel to
the application. The following values are returned to the application (for
details
on these parameters; see..".Error! Reference source not found." on page
Error! Bookmark not defined.):
...:..y v.y,..~.. :.. ',...~..,
~ The density. of the voxel: V (xyz)
~ The opaci of~the voi~el o~[V(xyz)]
~,:. ::.,f..'.F"' :. ';.,et:>a'... ,.~,Y..:
~ If coIpr'hiode is used, the .color of the voxel
;,x~y, ...
~ The accumulated opacity:~,ray; opacity
~ The accumulated color; if color mode is used, or accumulated density: I(xy)
During coordinate query, nearest neighbor interpolation is used. The
application
can then scan this list to determine the selected voxel using its own
algorithm
Note that the IAP processing se=ver also returns the voxels that do not
contribute to the final pixel. This is done on purpose so that the application
can,
for example, determuie the size/thickness of the object (e.g.; vessel or bone)
on
which the user has clicked.
" . .,.. , ..,., '::....
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Perspective Projection
In medical imaging, the term "3D image" is very often referred to as an image
generated using parallel projection. The word parallel indicates that the
projector rays (the rays described in the pseudo code of "Error! Reference
source not found:" on page Error! Bookmark not defined.) are all parallel.
This technique creates an impression of the three-dimensionality in the image
but does not simulate human vision accurately. In human vision, the rays afe
not parallel .but instead converge on a point, the eye of the viewer (more
specifically the. xetina).,'This rendering geometry is very often referred to
as
perspective, projection. Perspective projection allows the generation of more
realistic images.than parallel projectian. Figure 10 and Figure 11 illustrate
the
difference between these two projections methods.
Figure 10 - Parallel and Perspective Projections
Parallel Projection ~ Perspective Projection
Image Plane ~ nage Plane
,:,..
Focal Point
Cedara Software Core. Page 11
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Figure 11- Parallel projection and Perspective projection schemes. .
a. b:
There is an important implication of using perspective projection in medical
imaging. Iil perspective projection; different parts of the object
are~magnified
with different factors;. parts close to the eye look bigger than objects
further
away. This implies that on a perspective image, it is not possible to compare
object sizes or, distances. An ei~ample of this is shown in Figure 12.
Figure 12 -Perspective magnifies part of the dataset with differeat factors.
In (a), the image is rendered with parallel projection. The yellow marker
shows that the two vessels do cot have the same size. In (b), the image is
rendered with perspective projection. The red marker shows that. the same
two vessels appear to have the same size.
a. , . ... _...,......,..:::. h_
CA 02365045 2001-12-14
PrS 3D Rendering Architecture White Paper
Although not suitable for measurement, perspective projection is useful in
medical imaging for several reasons. It can simulate the view of the endoscope
and the radiographic acquisition.
The geometry of the acquisition of radiography is, by its nature, a
perspective
projection. Using a CT dataset;:it is theoretically possible to reconstruct
the
radiograph of a patient from, ariy position. This process is referred to as
DRR
(Digital Reconstructed Radiography) and is used in Radiotherapy Planning. Cane
of the technical difficulties of DRR is that the x-ray used in the CT scanner
has
a different energy (and hence different characteristic) compared to that used
in
x-ray radiography. The IAP processing server allows correction of those
differences using the opacity map as a x-ray absorption curve for each
specific
voxeL Figure 13 shows two examples of DRR
Figure 13 - DRRs of a CT dataset.
To correct for the different:x-tax absorptions between CT x-ray and
radiographic x-ray, the opacity' curve has been approximated to an
exponential function. The function is designed to highlight the bone voxels
in the dataset which in the radiography absorb more than in CT.
~~Y
The perspective projection can also simulate the .acquisition from an
endoscope.
This allows the fly,through of the dataset to perform; for example, virtual
colonoscopy. Figure 14 shows two images generated from inside the CTA
dataset.
.Ts: y
Cedars Software Corp. ' . . Page 13
CA 02365045 2001-12-14
PrS, 3D Rendering Architecture White Paper
Figure 14 - Perspective allows "fly-though" animations.
These images are frames from the CTA chest dataset. The white objects
are calcification in the aorta. .In (a), the aneurysm is shown from inside the
aorta. In (b), the branch to the iliac arteries is shown. Refer to the IAP
Image Gallery for the full animation.
b.
..., ..: . :. : . : ;.: ..:; . .
Since perspective projeetiou.involyes different optimizations than parallel
projection, ~t has been implemented as, a separate object, Pproj, which
supports
the same input and output connections as Cproj. The pipeline shown in this
white paper can be used in perspective mode by simply switching Cproj with
Pproj (although currently not all the functionality available in Cproj is
available
in Pproj).
Because of the sampling scheme implemented in the Perspective renderer, it is
possible to have a great level of detail even when the images are minified
several times.
Figure 15 - Perspective renderer allows a high level of detail.
The perspective rendering engine is released as a separate dll on win32. The
application can replace this dll with its own implementation as long as it is
compliant with the same interface and uses the same data structure.
