Note: Descriptions are shown in the official language in which they were submitted.
CA 02245175 1998-07-30
WO 97131336 PCT/IB97/00236
Automation methodology for 3 D image metrology systems
TECHNICAL FIELD
S The present invention relates to 3D-image metrology with
photogrammetric methods.
The invention is applied to 3D Image Metrology but can also
be applied to other electro-optical systems. 3D Image
Metrology is applicable to all areas where 3D geometric and
other information is to be extracted from imagery, imagery
being acguired through standard imaging systems or other
devices such as imaging radar or sonar. Typical application
areas are dimentional measurements to be performed in the
industrial process reaching from design, development,
production to quality control, in engineering but even in
areas such as art and medicine.
BACKGROUND OF THE INVENTION
The background of the invention is methodologies for the
automation of so far m~nl~l tasks in 3D image metrology
systems.
For exemple, robots today are programmed off line which
means a teaching of robots using a computer simulation
program. ~ue to technical limits in the mechanical
movements of a robot arm, there is a difference between the
programmed position of a robot arm and its real position.
This differecne can be measured with the robot calibration
system and correction polynoms for the different ankles can
be derived. Using such a correction for the simulation, the
accuracy of the real robot movement can be increased. Three
or four CCD-cameras are mounted in the vinicity of the
robot to observe the measurement volume. The cameras detect
auto-recognizable targets mounted on and in a specific
relation to the tool center point. According to these
1"~ J
Application No. PCT/IB97/00236 lY.u lYY~
- App]icant: Imetric S~A 0224~l75 Isss-07-30BP 97-001
.
detections a calibration process is activated and
corrections for the simulation is established.
Targets were designed to facilitate automatic detection and
identification. For example, the targets could be designed
as a white circle with a concentric ring on a black base.
Around the concentric rin~ there is another concentric ring
as a label area in which a ten bit target identi~ication is
coded. The central circle has a black dot in the middle to
facilitate manual measurement. Black and white are colours
chosen to get the maximum contrast.
Therefore, part of the invention, i.e. auto-recognisable
targets, is already used.
A large number of attempts to arrive at this goal of
automation have been undertaken. All attempts so far have
only achieved partial results and/or are not suited for
widespread use.
One approach of automatically identifying individual
targets was relying on different forms and shapes of the
targets. Targets which were made up of various geometric
shapes such as circles, rectangles and crosses were used to
differentiate between 2D patterns of targets. This approa~h
need relatively large targets and does not allow for a
large number of different patterns.
Another approach used targets with an additional linear
pattern area in the vicinity of circular targets. The
pattern was similar to typical bar codes. The code was such
not integrated with the target per se and needed relatively
much space. These targets were so called "coded" targets
and were developed by several groups at that time.
An example of these "coded" targets is described in the
document "Robust video object recognition and pose
A~1E~ D ~.
lP~ P
CA 02245175 1998-07-30
2b
determination using passive target labels"; SPIE Vol. 182~
Cooperative Intelligent Robotics in Space III (1992), pages
2 - 12. The document discloses a method of 3D image
metrology using four targets combined with one label area
s with the purpose to compute the orientation of images with
these targets which all must be located on one and the same~
plane.
Another approach with "coded" targets uses a bit pattern
around the target. The pattern was a binary bit pattern
- J~s
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97100236
which was designed to be independent o~ the rotation of the
taget. Two major disadvantages were common to these
targets. They had too small distance between target and
code when used with CCD-cameras and they could only use the
labels provided by the code, i.e. typically numbers like
1,2,3 but no alphanumeric labels. The insuf~icient amount
of space between target and code degraded the measurement
accuracy of the target. Groups with several hundred targets
were discernible with that approach. All codes provided
only to use the same label for the target as the label
which was encoded in the labeling area of the target.
Furthermore, groups of targets were used by some systems to
identify one particular group and through that group
individual targets of the group. These techniques rely on
the particular geometric relationship of all targets in a
group which can either be identified in an image and/or in
3D space.
Other approaches to identify an element was to use groups
of targets which lie in a particular geometric relationship
to eachother and which discern the group from other targets
within an image or even within the 3D space. Disadvantages
of these techniques ly in the fact that one always needs
several targets and that the technique does not allow for a
large number of different groups with resonable economic
effort and space requirements.
The above approches were furthermore combined with the use
of colours in order to increase the number of targets which
are discernible.
Another aspect of automation is the establishment of
correspondence of targets which are not discernible through
their particular individual characteristics such as a
label, a colour, a form. Two basic techniques were
developed to overcome this problem. One uses the geometric
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97~00236
relation between images from which geometric conditions can
be derived, i.e. the so called epipolar line. Using this
technique the correspondence between large numbers of
targets could be established automatically. A similar
approach where the geometric relation between images and
images and the object space is used is the Multiphoto
Geometrically Constrained Matching. In another approach the
3D relation of points was used directly. Both techniques
only adressed part of the problem as they required that the
exterior orientation and/or the relative orientation of
images had to be known a priori.
The establi~hment of the correspondence of homologous
points in images can be performed by the use of the
epipolar geometry, as well as using the 3D relation of
points.
These approaches of the known solutions to automatically
identify individual targets were rather limited and shows
disadvantages in the following ways:
l. The targets could not be measured with sufficient
accuracy (not ~or all techniques).
2. Only numeric labels were associated with the targets
although in many applications alphanumeric labels are
required.
3. Groups of targets require much more space than a target
with an associated label area.
4. Groups of targets are not sufficiently discernible from
other targets in applications with large numbers of
targets.
The automatically indentifiable targets have priviously
been designed with film based 3D image metrology systems in
mind. They did not take care of the imaging characteristics
of CCD-sensor or other digital imaging systems.
CA 0224~17~ 1998-07-30
WO~7/31336 PCT~B97/00236
The establishment of the correspondence did not use auto-
identifiable targets to establish the initial orientation
of the images. With respect to a complete system they did
not allow to:
Automatically identify the object.
1. Provide ~or a mechanism to automatically load all the
relevant data for a particular application.
2. Provide for highest measurement accuracy, ;~ nlty to
occlusions, and alph~nl~m~ic labels.
3. Provide for automatic identification of adapters
independent of knowledge on their spatial location.
Overall, the existing methods 60 far were only addressing
part of the problem.
OB~ECT OF THE Il~VENTION
The object of the invention, in parts or total, is to
provide to partial and/or complete automation for
dimentional measurements with image metrology. Most
important are systems where 3D image metrology is to
perform fully automatically, such as in automated
production systems where robots or other manufacturing
systems are to be guided, e.g. for machln;ng, fitting,
moving.
An other object of the invention is to provide for a higher
degree or even total automation in 3D image metrology
and/or other applications where such systems are used.
SUMMARY OF THE INVENTION
The invention applies to the following su~tasks:
- Finding targets for the purpose of automatically
determining the exterior orientation of images.
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97/00236
- Finding targets in several images for the purpose of
establishing homologous point in these images.
- Finding targets from different views in an electro-
optical system in order to establish their relation.
- Identifying targets which serve as references for the
definition of a coordinate system.
- Identifying measurement adapters in order to identi~y
the particular adapter and its geometric and/or other
characteristics.
Examples for such adapters are:
- Individual targets for which some of~sets and other
geometric parameters are known.
- Two or more targets forming a straight line with
another spatial location which they refer to.
- Three or more targets which are in a ~patial relation
to one or more other spatial locations.
Examples thereof are:
- Adapters consisting of three or more visible
targets and three or more half spheres which lie on
an object surface. The visible targets and the
other locations are in a known relative spatial
relation.
- Adapters consisting of three or more visible
targets which are in relation to one or more
locations. The visible targets and the other
locations are in a known relative spatial relation.
Physical examples are the DigiPen, which is a
touchprobe o~ the Imetrics systems, or a spindle
and the targets on the spindle holder in a TI2
~ystem.
- LUT
The invention will be part of a future improvement of a 3D
image metrology system. It is applicable to other systems.
CA 0224~17~ 1998-07-30
WO97131336 PCT~B97/00236
It will also be u~ed in an industrial manufacturing system.
DESCRIPTION OF THE INVENTION
The invention uses the following steps in the current
i~plementation either on an individual basis or as in any
combination of the individual techniques.
1. Measurement Accuracy: The design of the auto-
identifiable targets is such that the distance between
the central target and the label area, which is used to
identify the target, i.e. the area with a binary code or
other code, is sufficiently large in order not to
degrade the accuracy with which the central target can
be located. This distance was determined to be at least
equal to the diameter of the central target when the
central target was designed to provide for an optimum
measurement accuracy, i.e. to be imaged on typically 6
pixels in diameter.
2. Immunity to partial occlusions of label area: In many
applications, part of the area containing the label can
be occluded from other objects lying between the imaging
system and the label area. Techniques using a redundancy
in the code, additional areas such as a border around
the label, and a-priori knowledge on which auto-
identifiable targets were used, can be used individually
or in combinations to circumvent this problem.
3. Labeling Freedom: ~o overcome the problem of having to
apply particular targets on particular points which one
would li~e to label differently, a translation between
the label of the target and its physical label is used.
Such an auto-identifiable target with the label "11" can
be used as a target with a label "A". This can for
example be implemented using a look up table where a
"user" label is connected to each physical label of a
CA 0224~17~ 1998-07-30
WO97/31336 PCTnB97/00236
target. This look up table can be user de~inable or
~ixed. Furthermore a double look up table can be used,
i.e. a table where the label provided by the label area
i8 first translated into another label and later is
finally related to a label to be used in the further
processing. For example, the label on the target
contains the label Illl". On the target, the name "A" can
be used to identify it to a person, but in the
particular application the target is to be used as
target "X". This helps to reduce the number of auto-
identifiable targets physically required in order to
fullfill a large number o~ di~ferent applications.
4. Identification of the Measurement Object: Here, one or
l~ more labels of auto-identifiable targets are used to
identify a particular object. For example, the hood of
model A of a car will have auto-identifiable targets
with label "K" and the hood of model B of a car will
have auto-identifiable targets with label "Y". One or
more labels can be used. Based on this information, data
related to the particular object can be automatically
selected. Use of such an object identification as a safe
guard (for example in manufacturing, but also when
per~orming quality inspection) and/or to load data
related to the particular object.
5. Default Project Strategy: A methodology used in
connection with the auto-identifiable targets where the
data pertaining to a particular task (e.g. coordinates
o~ reference locations, compensation ~or adapters,
measurements to be performed) is collected in a "default
project". This can be a collection of ~iles containing
the pertinent data or a particular access to a data base
or some other technique to store and access the relevant
data.
6. Automated Orientation:
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97/00236
A procedure where in the setup of a measurement system
auto-identifiable target~ are used to automatically
compute the exterior orientation and/or relative
orie~tation of one or more cameras/images.
.,~;
A procedure where during measurements auto-identifiable
targets are used to automatically compute the exterior
orientation of one or more cameras/images in order to
account for eventual changes in the relation between the
object~s) on which the auto-identifiable targets are
fixed and the camera (8) and/or to perform other
measurements.
Any procedure where three or more auto-identifiable
targets are used to establish a particular coordinate
system.
In all these procedures, the 3D coordinates of at least
three auto-identifiable targets must be known.
7. Automated Measurement:
Procedures where auto-identifiable targets are used to
establish the geometric relation of images with respect
to a coordinate system related to the auto-identifiable
targets (exterior orientation) or to some arbitrary
coordinate system, for example related to one or more of
the cameras/images (relative orientation) in order to be
able to establish the correspondence between standard
targets in two or more images.
Determining the exterior orientation o~ individual
images using for at least three auto-identifiable
targets of which the 3D spatial coordinates in some 3D
coordinate system are known, either prior to start of
measurements or which were determined in the course of
the measurements.
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97/00236
Determining homologous points in images. This can for
example be used to compute the relative orientation of
these images.
Combination of auto-identifiable targets and standard
targets. ~his technique allows to establish the spatial
geometric relation of the images, i.e. either their
exterior orientation~ or their relative orientations,
through auto-identifiable targets and eventual other
targets which were already located in the images but not
identified (i.e. the relation between a particu~ar
target in an image and its origin in object space and/or
its corresponding image in another image is not known),
and to establish the correspondence between homologous
targets imaged in different images using the geometric
relation of these images (i.e. their exterior
orientations or their relative orientations) and/or the
3D geometry of the targets to be identified which
becomes computable using the geometric relation of the
images in which they were imaged.
Targets can be retro-reflective targets but also any other
feature of an object which can be located in an image or a
group of images.
8. Scale Bar: The use of one or more auto-identifiable
targets to identify a particular scale bar in order to
obtain its calibrated distance or distances (if more
than two points are located on the scale bar). The
following elements are part of the techni~ue:
1) Identify the particular scale bar
2) Help in the search for others targets on the scale bar
3) Use several known distances on one scale bar for the
purpose of improved accuracy and reliability.
They can be applied individually or in combinations.
q. Particular adapters or object are identified by using
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97/00236
either one or more of the labels. Using a single or a
group of adapters is then in turn used.
One or more labels are used to identify a particular
object. Such an object can be:
- An individal point ~ith or without some given
characteristics such as XYZ-coordinates for the
purpose of orientation. Using the label, the 3D
spatial coordinates of a point can be taken from a
data base in order to compute exterior orientations,
to compute spatial position of an object with respect
to some coordinate system through a spatial similarity
trans~ormation.
- An individual label serving to identify an adapter
with its characteristics. These can for example be the
offset of a button target, the vector distances for
reductions of double vector targets or hidden point
bars.
- A multi-point adapter where the label is used to find
the geometric relation between visible targets and
another point or points which are in particular
geometric relation to the targets.
- A particular scale bar.
- A particular touch probe or other device such as a
2~ spindle or other manufacturing device and other
objects which are to be identified.
Adapters are any technique where one or more auto-
identifiable targets on any sort of adapter i9 to:
- Identify the particular adapter to derive geometric
parameters of the adapter.
- Identi~y the particular adapter to retrieve its
J spatial coordinates from a data base or other storage
mechanism.
- Identify the particular adapter to control the
software such that it performs a particular
measurement sequence or other process.
CA 0224~17~ 1998-07-30
WO97131336 PCT~B97/00236
Examples:
a) The system finds a double vector target and
computes automatically the mechanical point
related to the two optical targets.
b) The system finds a particulat target group and
com~n~ a robot to go and grab the piece.
The following first defines a number of adapters and
then defines the use of the auto-identifiable targets
with these adapters.
Single Polnt Adapter: A "Single Point Adapter" is a
device consisting of one optically visible target, e.g.
a retro-reflective target or LED, and one or more
1~ mechanical points, e.g. the location of a shank, the
center of a sphere where the relation of the optical and
mechanical targets are known and usually fixed.
Typical examples are:
- A "Button Target" which consists of one optical target
sitting on a mechanical piece with a shank adapted to
tooling holes. The target is in the centre of the
shank axis but at a certain distance from a mechanical
interface.
- Target integrated into a partial sphere as for example
targets integrated into a Taylor Hobson sphere.
The auto-identifiable target is used to identify an
individual "Single Point Adapter" and to extract some
generally geometrical characteristics from a data base.
Such information can be the offset of the optical target
and the one or more mechanical points. This allows to
use adapters with various offsets in one measurement
task without requiring the user to identify the adapter
and without relying on applying one particular adapter
with a particular offset to specific locations.
CA 0224~17~ 1998-07-30
WO97/31336 pcTnss7/oo236
The auto-identi~iable target is used to ~ind the 3D
reference coordinates of the "Single Point Adapter",
i.e. either of the optical target or the reference
location.
A "Two Point Adapter" is a device consisting of two or
more optically visible targets and one or more
mechanical or virtual point~. The differentiation
between "Two Point Adapters" and "Multi-Point Adapters"
is the fact that the first one does not allow to
establish a 3D relation between the optically visible
targets and the mechanical or virtual points without
additional knowledge on the spatial location of the "Two
Point Adapter". In geometric terms, the points of the
"Two Points Adapter" cannot be used to determine a 3D
spatial similarity transformation with six parameters.
Usually the point(s) to be computed using a "Two Point
Adapter" are on a 3D line with two optical targets.
Typical examples of 'ITwo Point Adapters" are:
"Double Vector Targets". This adapter consists typically
of two optically visible targets. The mechanical or
virtual point of interest is on a spatial line with the
two optically visible targets and its (spatial) distance
from one or both targets is known or it may be in the
geometric centre of the two optically visible targets.
One or more auto-identifiable targets are used to
identify a particular "Two Point Adapter~ in order to:
- Identify the adapter and obtain its geometric
parameters and/or the naming convention for the one or
more points to be computed.
- Identify the adapter to retrieve the spatial
coordinates of one or more optical targets and/or one
or more re~erence locations of the adapter.
- Identify the adapter ~o control particular software
functions or to indicate certain actions by the
-
CA 0224~17~ 1998-07-30
WO97/31336 PCT~B97/00236
system.
A "Multi-Point Adapter" is a device consisting of three
or more optically visible targets which are in a
geometric relation to an object and/or one or more
mechanical or virtual points. The "Multi-Point Adapter"
can for example be used to determine the spatial
position and orientation of the adapter or an object
physically connected to the adapter. Furthermore, it can
be used to derive the spatial location of one or more
points on the adapter or another object physically
connected with the adapter.
Examples of "Multi-Point Adapters" are:
- V-Plates: A device consisting of three optical targets
and three half spheres.
- Robot Calibration Fixtures: Devices fixed to robots
for the purpose of calibrating robots.
- Touch Probes such as the DigiPen and other devices
consisting of three or more optical targets connected
to a CMM touch probe for the purpose of determ;n;ng
the 3D spatial coordinates of the probe.
- Tooling Fixtures: Any device on a tool which serves to
position a piece in a manufacturing production
process. Adapters can be built to position tooling
fixtures.
- Tool holders or Tools (drills, grippers, cutters and
other end effectors): These can be on manufacturing
systems such as robots or robot arms, CNC machines,
but also on hand-held devices.
The use of auto-identifiable target(s) for:
- Identi~ying the particular adapter to derieve its
geometric characteristics and/or to retrieve its
spatial reference coordinates.
- Identifying the particular adapter to initiate
particular actions or computations to be performed
CA 0224~17~ 1998-07-30
WO 97/31336 PCT/IB97/00236
such as to automatically compute the invisible
targets.
q
Notes:
- Targets to be mounted in tooling holes or other
places with a target that can automatically be
identified, labeled and measured by a computer
program.
- Automatically measuring targets for the purpose o~
determining the orientation of one or more cameras
automatically for the purpose of orienting cameras.
- Update orientation.
- Tool with three or four targets to automatically
orient cameras.
- Devices with three or more points that are
automatically identifiable and are used to derive
1. the position and/or attitude of the device that
they are attached to in space
2. the position of one or more other points
physically connected with these points in space.
Tool to perform an automatic orientation and to
automatically compute approximations for object points.
Method where points are labeled by indicating their
label in one image only.
Coded targets to position other devices such as robots,
CMMs, CNC machines.
r~ J~,
