Note: Descriptions are shown in the official language in which they were submitted.
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
Mobile Human Interface Robot
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This U.S. patent application claims priority under 35 U.S.C.
119(e) to U.S.
Provisional Application 61/428,717, filed on December 30, 2010; U.S.
Provisional
Application 61/428,734, filed on December 30, 2010; U.S. Provisional
Application
61/428,759, filed on December 30, 2010; U.S. Provisional Application
61/429,863, filed
on January 5, 2011, U.S. Provisional Application 61/445,408, filed on February
22, 2011;
U.S. Provisional Application 61/445,473, filed on February 22, 2011; U.S.
Provisional
Application 61/478,849, filed on April 25, 2011; and under 35 U.S.C. 120 to
U.S. Patent
Application 13/032,312, filed on February 22, 2011; and U.S. Patent
Application
13/032,228, filed on February 22, 2011. The disclosures of these prior
applications are
considered part of the disclosure of this application and are hereby
incorporated by
reference in their entireties.
TECHNICAL FIELD
[0002] This disclosure relates to mobile human interface robots.
BACKGROUND
[0003] A robot is generally an electro-mechanical machine guided by a
computer or
electronic programming. Mobile robots have the capability to move around in
their
environment and are not fixed to one physical location. An example of a mobile
robot
that is in common use today is an automated guided vehicle or automatic guided
vehicle
(AGV). An AGV is generally a mobile robot that follows markers or wires in the
floor,
or uses a vision system or lasers for navigation. Mobile robots can be found
in industry,
military and security environments. They also appear as consumer products, for
entertainment or to perform certain tasks like vacuum cleaning and home
assistance.
SUMMARY
[0004] One aspect of the disclosure provides a mobile robot that
includes a drive
system having a forward drive direction, a controller in communication with
the drive
i
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
system, and a volumetric point cloud imaging device supported above the drive
system
and directed to be capable of obtaining a point cloud from a volume of space
that
includes a floor plane in a direction of movement of the mobile robot. A dead
zone
sensor has a detection field arranged to detect an object in a volume of space
undetectable
by the volumetric point cloud imaging device. The controller receives point
cloud signals
from the imaging device and detection signals from the dead zone sensor and
issues drive
commands to the drive system based at least in part on the received point
cloud and
detection signals.
[0005] Implementations of the disclosure may include one or more of the
following
features. In some implementations, the dead zone sensor includes at least one
of a
volumetric point cloud imaging device, a sonar sensor, a camera, an ultrasonic
sensor,
LIDAR, LADAR, an optical sensor, and an infrared sensor. The detection field
of the
dead zone sensor may envelope a volume of space undetectable by the volumetric
point
cloud imaging device (i.e., a dead zone). In some examples, the volume of
space
undetectable by the volumetric point cloud imaging device is defined by a
first angle, a
second angle and a radius (e.g., 57 x 450 x 50 cm). The detection field of
the dead zone
sensor may be arranged between the volumetric point cloud imaging device and a
detection field of the volumetric point cloud imaging device. In some
examples, the dead
zone sensor has a field of view extending at least 3 meters outward from the
dead zone
sensor. In this example, the dead zone sensor can be dual-purposed for
relative short
range within the dead zone and as a long range sensor for detecting objects
relatively far
away for path planning and obstacle avoidance.
[0006] In some implementations, the robot includes an array of dead
zone sensors
with at least one dead zone sensor having its detection field arranged to
detect an object
in the volume of space undetectable by the volumetric point cloud imaging
device. Te
array of dead zone sensors may be arranged with their fields of view along the
forward
drive direction or evenly disbursed about a vertical center axis defined by
the robot.
[0007] The imaging device, in some examples, emits light onto a scene
about the
robot and captures images of the scene along the drive direction of the robot.
The images
include at least one of (a) a three-dimensional depth image, (b) an active
illumination
image, and (c) an ambient illumination image. The controller determines a
location of an
2
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
object in the scene based on the images and issues drive commands to the drive
system to
maneuver the robot in the scene based on the object location. The imaging
device may
determine a time-of-flight between emitting the light and receiving reflected
light from
the scene. The controller uses the time-of-flight for determining a distance
to the
reflecting surfaces of the object.
[0008] In some implementations, the imaging device includes a light
source for
emitting light onto the scene and an imager for receiving reflections of the
emitted light
from the scene. The light source may emit the light in intermittent pulses,
for example, at
a first, power saving frequency and upon receiving a sensor event emits the
light pulses at
a second, active frequency. The sensor event may include a sensor signal
indicative of
the presence of an object in the scene. The imager may include an array of
light detecting
pixels.
[0009] The imaging device may include first and second portions (e.g.,
portions of
one sensor or first and second imaging sensors). The first portion is arranged
to emit
light substantially onto the ground and receive reflections of the emitted
light from the
ground. The second portion is arranged to emit light into a scene
substantially above the
ground and receive reflections of the emitted light from the scene about the
robot.
[0010] In some implementations, the imaging device includes a speckle
emitter
emitting a speckle pattern of light onto a scene along a drive direction of
the robot and an
imager receiving reflections of the speckle pattern from an object in the
scene. The
controller stores reference images of the speckle pattern as reflected off a
reference object
in the scene. The reference images are captured at different distances from
the reference
object. The controller compares at least one target image of the speckle
pattern as
reflected off a target object in the scene with the reference images for
determining a
distance of the reflecting surfaces of the target object. In some instances,
the controller
determines a primary speckle pattern on the target object and computes at
least one of a
respective cross-correlation and a decorrelation between the primary speckle
pattern and
the speckle patterns of the reference images.
[0011] To increase a lateral field of view, the imaging sensor may scan
side-to-side
with respect to the forward drive direction. Similarly, to increase a vertical
field of view,
the imaging sensor may scan up-and-down.
3
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0012] In some implementations, the controller ceases use of the
received point cloud
signals after a threshold period of time after receipt for issuing drive
commands to the
drive system. The controller may suspend cessation of use of the received
point cloud
signals upon detemiining the presence of an object in the volume of space
undetectable
by the volumetric point cloud imaging device based on the received detection
signals
from the dead zone sensor. Moreover, the controller may continue ceasing use
of the
received point cloud signals after the threshold period of time after receipt
upon
determining that the volume of space undetectable by the volumetric point
cloud imaging
device is free of any objects, for example, based on the received detection
signals from
the dead zone sensor.
[0013] Another aspect of the disclosure provides a mobile robot that
includes a base
and a holonomic drive system supported by the base and defining a vertical
axis (Z). The
holonomic drive system maneuvers the robot over a work surface of a scene. The
robot
includes a controller in communication with the drive system, a leg extending
upward
from the base, and a torso supported by the leg. The torso rotates about the
vertical axis
with respect to the base. At least one imaging sensor (e.g., a volumetric
point cloud
imaging device) is disposed on the torso and captures a volumetric point cloud
(e.g.,
three-dimensional images) of the scene about the robot. The rotating torso
moves the
imaging sensor in a panning motion about the vertical axis providing up to a
3600 field of
view about the robot.
[0014] In some implementations, the at least one imaging sensor has an
imaging axis
arranged to aim downward along a forward drive direction of the drive system.
The at
least one imaging sensor may include a first imaging sensor having an imaging
axis
arranged to aim downward along a forward drive direction of the drive system
and a
second imaging sensor having an imaging axis arranged to aim away from the
torso
parallel to or above the work surface. Moreover, the at least one imaging
sensor may
scan side-to-side with respect to the forward drive direction to increase a
lateral field of
view and/or up-and-down to increase a vertical field of view of the imaging
sensor.
[0015] The at least one imaging sensor may include a speckle emitter
emitting a
speckle pattern of light onto the scene and an imager receiving reflections of
the speckle
pattern from an object in the scene. The controller stores reference images of
the speckle
4
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
pattern as reflected off a reference object in the scene. The reference images
are captured
at different distances from the reference object. The controller compares at
least one
target image of the speckle pattern as reflected off a target object in the
scene with the
reference images for determining a distance of the reflecting surfaces of the
target object.
The imaging sensor may capture images of the scene along a drive direction of
the robot.
The images include at least one of (a) a three-dimensional depth image, (b) an
active
illumination image, and (c) an ambient illumination image.
[0016] The controller may determine a location of an object in the
scene based on the
image comparison and issues drive commands to the drive system to maneuver the
robot
in the scene based on the object location. In some examples, the controller
determines a
primary speckle pattern on the target object and computes at least one of a
respective
cross-correlation and a decorrelation between the primary speckle pattern and
the speckle
patterns of the reference images.
[0017] The imaging sensor may be a volumetric point cloud imaging
device
positioned at a height of greater than 2 feet above the work surface and
directed to be
capable of obtaining a point cloud from a volume of space that includes a
floor plane in a
direction of movement of the robot. The imaging sensor may have a horizontal
field of
view of at least 45 degrees and a vertical field of view of at least 40
degrees and/or a
range of between about 1 meter and about 5 meters. In some examples, the
imaging
sensor has a latency of about 44 ms, and imaging output of the imaging sensor
may
receive a time stamp for compensating for latency.
[0018] In some implementations, the robot includes a dead zone sensor
having a
detection field arranged to detect an object in a volume of space undetectable
by the
volumetric point cloud imaging device. The dead zone sensor may include at
least one of
a volumetric point cloud imaging device, a sonar sensor, a camera, an
ultrasonic sensor,
LIDAR, LADAR, an optical sensor, and an infrared sensor. The detection field
of the
dead zone sensor may envelope a volume of space undetectable by the volumetric
point
cloud imaging device (i.e., a dead zone). In some examples, the volume of
space
undetectable by the volumetric point cloud imaging device is defined by a
first angle, a
second angle and a radius (e.g., 57 x 450 x 50 cm). The detection field of
the dead zone
sensor may be arranged between the volumetric point cloud imaging device and a
5
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
detection field of the volumetric point cloud imaging device. In some
examples, the dead
zone sensor has a field of view extending at least 3 meters outward from the
dead zone
sensor. In this example, the dead zone sensor can be dual-purposed for
relative short
range within the dead zone and as a long range sensor for detecting objects
relatively far
away for path planning and obstacle avoidance.
[0019] In some implementations, the robot includes an array of dead
zone sensors
with at least one dead zone sensor having its detection field arranged to
detect an object
in the volume of space undetectable by the volumetric point cloud imaging
device. Te
array of dead zone sensors may be arranged with their fields of view along the
forward
drive direction or evenly disbursed about a vertical center axis defined by
the robot.
[0020] In some implementations, the controller ceases use of received
point cloud
signals after a threshold period of time for issuing drive commands to the
drive system.
The controller may suspend cessation of use of the received point cloud
signals upon
determining the presence of an object in the volume of space undetectable by
the imaging
sensor based on received detection signals from the dead zone sensor.
Moreover, the
controller may continue ceasing use of the received point cloud signals after
the threshold
period of time upon determining that the volume of space undetectable by the
imaging
sensor is free of any objects, for example, based on the received detection
signals from
the dead zone sensor.
[0021] The torso may rotate with respect to the leg and/or the leg may
rotate with
respect with the base about the vertical axis. In some examples, the leg has a
variable
height.
[0022] In yet another aspect, a method of object detection for a mobile
robot includes
rotating an imaging sensor about a vertical axis of the robot. The imaging
sensor emits
light onto a scene about the robot and captures images of the scene. The
images include
at least one of (a) a three-dimensional depth image, (b) an active
illumination image, and
(c) an ambient illumination image. The method further includes determining a
location
of an object in the scene based on the images, assigning a confidence level
for the object
location, and maneuvering the robot in the scene based on the object location
and
corresponding confidence level.
6
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0023] In some implementations, the method includes constructing an
object
occupancy map of the scene. The confidence level of each object location may
be
degraded over time until updating the respective object location with a newly
determined
object location. The method may include maneuvering the robot to at least one
of: a)
contact the object and follow along a perimeter of the object, or b) avoid the
object.
[0024] In some examples, the method includes detecting an object in a
volume of
space undetectable by the imaging sensor, such as by using a dead zone sensor
having a
detection field arranged to detect an object in the volume of space
undetectable by the
imaging sensor, and ceasing degradation of the confidence level of the
detected object.
The method may include continuing degradation of the confidence level of the
detected
object upon detecting that the volume of space undetectable by the imaging
sensor is free
of that object.
[0025] The method may include emitting the light onto the scene in
intermittent
pulses, optionally altering a frequency of the emitted light pulses. The light
pulses may
be emitted at a first, power saving frequency and upon receiving a sensor
event, emitted
at a second, active frequency. The sensor event may include a sensor signal
indicative of
the presence of an object in the scene.
[0026] The method may include constructing the three-dimensional depth
image of
the scene by emitting a speckle pattern of light onto the scene, receiving
reflections of the
speckle pattern from the object in the scene, and storing reference images of
the speckle
pattern as reflected off a reference object in the scene. The reference images
are captured
at different distances from the reference object. The method further includes
capturing at
least one target image of the speckle pattern as reflected off a target object
in the scene
and comparing the at least one target image with the reference images for
determining a
distance of the reflecting surfaces of the target object. The method may
include
determining a primary speckle pattern on the target object and computing at
least one of a
respective cross-correlation and a decorrelation between the primary speckle
pattern and
the speckle patterns of the reference images. Moreover, the method may include
capturing frames of reflections of the emitted speckle pattern off surfaces of
the target
object at a frame rate, e.g., between about 10 Hz and about 90 Hz, and
optionally
7
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
resolving differences between speckle patterns captured in successive frames
for
identification of the target object.
[0027] The details of one or more implementations of the disclosure are
set forth in
the accompanying drawings and the description below. Other aspects, features,
and
advantages will be apparent from the description and drawings, and from the
claims.
DESCRIPTION OF DRAWINGS
[0028] FIG. 1 is a perspective view of an exemplary mobile human
interface robot.
[0029] FIG. 2 is a schematic view of an exemplary mobile human
interface robot.
[0030] FIG. 3 is an elevated perspective view of an exemplary mobile
human
interface robot.
[0031] FIG. 4A is a front perspective view of an exemplary base for a
mobile human
interface robot.
100321 FIG. 4B is a rear perspective view of the base shown in FIG 4A.
[0033] FIG. 4C is a top view of the base shown in FIG. 4A.
[0034] FIG. 5A is a front schematic view of an exemplary base for a mobile
human
interface robot.
[0035] FIG. 5B is a top schematic view of an exemplary base for a
mobile human
interface robot.
[0036] FIG. 6 is a front perspective view of an exemplary torso for a
mobile human
interface robot.
[0037] FIG. 7 is a front perspective view of an exemplary neck for a
mobile human
interface robot.
[0038] FIGS. 8A-8G are schematic views of exemplary circuitry for a
mobile human
interface robot.
[0039] FIG. 9 is a schematic view of an exemplary mobile human interface
robot.
100401 FIG. 10A is a perspective view of an exemplary mobile human
interface robot
having multiple sensors pointed toward the ground.
[0041] FIG. 10B is a perspective view of an exemplary mobile robot
having multiple
sensors pointed parallel with the ground.
8
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0042] FIG. 11 is a schematic view of an exemplary imaging sensor
sensing an object
in a scene.
[0043] FIG. 12 is a schematic view of an exemplary arrangement of
operations for
operating an imaging sensor.
[0044] FIG. 13 is a schematic view of an exemplary three-dimensional (3D)
speckle
camera sensing an object in a scene.
[0045] FIG. 14 is a schematic view of an exemplary arrangement of
operations for
operating a 3D speckle camera.
[0046] FIG. 15 is a schematic view of an exemplary 3D time-of-flight
(TOF) camera
sensing an object in a scene.
[0047] FIG. 16 is a schematic view of an exemplary arrangement of
operations for
operating a 3D TOF camera.
[0048] FIG. 17A is a schematic view of an exemplary occupancy map.
[0049] FIG. 17B is a schematic view of a mobile robot having a field of
view of a
scene in a working area.
[0050] FIG. 18 is a schematic view of a dead zone of an imaging sensor.
[0051] FIG 19 is a perspective view of an exemplary mobile robot having
a first
imaging sensor arranged to point downward along a forward drive direction and
a second
imaging sensor arranged to point outward above the ground.
[0052] FIG. 20 is a top view of an exemplary mobile robot having a torso
rotating
with respect to its base.
[0053] FIG. 21 is a schematic view of an exemplary imaging sensor
having a dead
zone and a dead zone sensor having a field of view enveloping the dead zone.
[0054] FIG 22 is a top view of an exemplary mobile robot having a dead
zone sensor
arranged to detect objects in a dead zone of an imaging sensor.
[0055] FIG. 23 is a top view of an exemplary mobile robot having an
array of dead
zone sensors.
[0056] FIG. 24 is a top view of an exemplary mobile robot having long
range sensors
arranged about a vertical axis of the robot.
9
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0058] FIG. 26A provides an exemplary schematic view of the local
perceptual space
of a mobile human interface robot while stationary.
[0059] FIG. 26B provides an exemplary schematic view of the local
perceptual space
of a mobile human interface robot while moving.
[0060] FIG. 26C provides an exemplary schematic view of the local
perceptual space
of a mobile human interface robot while stationary.
[0061] FIG. 26D provides an exemplary schematic view of the local
perceptual space
of a mobile human interface robot while moving.
[0062] Like reference symbols in the various drawings indicate like
elements.
DETAILED DESCRIPTION
[0063] Mobile robots can interact or interface with humans to provide a
number of
services that range from home assistance to commercial assistance and more. In
the
example of home assistance, a mobile robot can assist elderly people with
everyday tasks,
including, but not limited to, maintaining a medication regime, mobility
assistance,
communication assistance (e.g., video conferencing, telecommunications,
Internet access,
etc.), home or site monitoring (inside and/or outside), person monitoring,
and/or
providing a personal emergency response system (PERS). For commercial
assistance,
the mobile robot can provide videoconferencing (e.g., in a hospital setting),
a point of
sale terminal, interactive information/marketing terminal, etc.
[0064] Referring to FIGS. 1-2, in some implementations, a mobile robot 100
includes
a robot body 110 (or chassis) that defines a forward drive direction F. The
robot 100 also
includes a drive system 200, an interfacing module 300, and a sensor system
400, each
supported by the robot body 110 and in communication with a controller 500
that
coordinates operation and movement of the robot 100. A power source 105 (e.g.,
battery
or batteries) can be carried by the robot body 110 and in electrical
communication with,
and deliver power to, each of these components, as necessary. For example, the
controller 500 may include a computer capable of > 1000 MIPS (million
instructions per
second) and the power source 1058 provides a battery sufficient to power the
computer
for more than three hours.
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0065] The robot body 110, in the examples shown, includes a base 120,
at least one
leg 130 extending upwardly from the base 120, and a torso 140 supported by the
at least
one leg 130. The base 120 may support at least portions of the drive system
200. The
robot body 110 also includes a neck 150 supported by the torso 140. The neck
150
supports a head 160, which supports at least a portion of the interfacing
module 300. The
base 120 includes enough weight (e.g., by supporting the power source 105
(batteries) to
maintain a low center of gravity CGB of the base 120 and a low overall center
of gravity
CGR of the robot 100 for maintaining mechanical stability.
[0066] Referring to FIGS. 3 and 4A-4C, in some implementations, the
base 120
defines a trilaterally symmetric shape (e.g., a triangular shape from the top
view). For
example, the base 120 may include a base chassis 122 that supports a base body
124
having first, second, and third base body portions 124a, 124b, 124c
corresponding to each
leg of the trilaterally shaped base 120 (see e.g., FIG. 4A). Each base body
portion 124a,
124b, 124c can be movably supported by the base chassis 122 so as to move
independently with respect to the base chassis 122 in response to contact with
an object.
The trilaterally symmetric shape of the base 120 allows bump detection 360
around the
robot 100. Each base body portion 124a, 124b, 124c can have an associated
contact
sensor e.g., capacitive sensor, read switch, etc.) that detects movement of
the
corresponding base body portion 124a, 124b, 124c with respect to the base
chassis 122.
[0067] In some implementations, the drive system 200 provides omni-
directional
and/or holonomic motion control of the robot 100. As used herein the term
"omni-
directional" refers to the ability to move in substantially any planar
direction, i.e., side-to-
side (lateral), forward/back, and rotational. These directions are generally
referred to
herein as x, y, and Oz, respectively. Furthermore, the term "holonomic" is
used in a
manner substantially consistent with the literature use of the term and refers
to the ability
to move in a planar direction with three planar degrees of freedom, i.e., two
translations
and one rotation. Hence, a holonomic robot has the ability to move in a planar
direction
at a velocity made up of substantially any proportion of the three planar
velocities
(forward/back, lateral, and rotational), as well as the ability to change
these proportions in
a substantially continuous manner.
11
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0068] The robot 100 can operate in human environments (e.g.,
environments
typically designed for bipedal, walking occupants) using wheeled mobility. In
some
implementations, the drive system 200 includes first, second, and third drive
wheels
210a, 210b, 210c equally spaced (i.e., trilaterally symmetric) about the
vertical axis Z
(e.g., 120 degrees apart); however, other arrangements are possible as well,
such a four
wheel holonomic drive system. Referring to FIGS. 5A and 5B, the drive wheels
210a,
210b, 210c may define a transverse arcuate rolling surface (i.e., a curved
profile in a
direction transverse or perpendicular to the rolling direction DR), which may
aid
maneuverability of the holonomic drive system 200. Each drive wheel 210a,
210b, 210c
is coupled to a respective drive motor 220a, 220b, 220c that can drive the
drive wheel
210a, 210b, 210c in forward and/or reverse directions independently of the
other drive
motors 220a, 220b, 220c. Each drive motor 220a-c can have a respective encoder
212
(FIG. 8C), which provides wheel rotation feedback to the controller 500. In
some
examples, each drive wheels 210a, 210b, 210c is mounted on or near one of the
three
points of an equilateral triangle and having a drive direction (forward and
reverse
directions) that is perpendicular to an angle bisector of the respective
triangle end.
Driving the trilaterally symmetric holonomic base 120 with a forward driving
direction
F, allows the robot 100 to transition into non forward drive directions for
autonomous
escape from confinement or clutter and then rotating and/or translating to
drive along the
forward drive direction F after the escape has been resolved.
[0069] In the examples shown in FIGS. 3-5B, the first drive wheel 210a
is arranged
as a leading drive wheel along the forward drive direction F with the
remaining two drive
wheels 210b, 210c trailing behind. In this arrangement, to drive forward, the
controller
500 may issue a drive command that causes the second and third drive wheels
210b, 210c
to drive in a forward rolling direction at an equal rate while the first drive
wheel 210a
slips along the forward drive direction F. Moreover, this drive wheel
arrangement allows
the robot 100 to stop short (e.g., incur a rapid negative acceleration against
the forward
drive direction F). This is due to the natural dynamic instability of the
three wheeled
design. If the forward drive direction F were along an angle bisector between
two
forward drive wheels, stopping short would create a torque that would force
the robot 100
to fall, pivoting over its two "front" wheels. Instead, travelling with one
drive wheel
12
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
210a forward naturally supports or prevents the robot 100 from toppling over
forward, if
there is need to come to a quick stop. When accelerating from a stop, however,
the
controller 500 may take into account a moment of inertia I of the robot 100
from its
overall center of gravity CGR.
[0070] In some implementations of the drive system 200, each drive wheel
210a,
210b, 210 has a rolling direction DR radially aligned with a vertical axis Z,
which is
orthogonal to X and Y axes of the robot 100. The first drive wheel 210a can be
arranged
as a leading drive wheel along the forward drive direction F with the
remaining two drive
wheels 210b, 210c trailing behind. In this arrangement, to drive forward, the
controller
500 may issue a drive command that causes the first drive wheel 210a to drive
in a
forward rolling direction and the second and third drive wheels 210b, 210c to
drive at an
equal rate as the first drive wheel 210a, but in a reverse direction.
[0071] In other implementations, the drive system 200 can be arranged
to have the
first and second drive wheels 210a, 210b positioned such that an angle
bisector of an
angle between the two drive wheels 210a, 210b is aligned with the forward
drive
direction F of the robot 100. In this arrangement, to drive forward, the
controller 500
may issue a drive command that causes the first and second drive wheels 210a,
210b to
drive in a forward rolling direction and an equal rate, while the third drive
wheel 210c
drives in a reverse direction or remains idle and is dragged behind the first
and second
drive wheels 210a, 210b. To turn left or right while driving forward, the
controller 500
may issue a command that causes the corresponding first or second drive wheel
210a,
210b to drive at relatively quicker/slower rate. Other drive system 200
arrangements can
be used as well. The drive wheels 210a, 210b, 210c may define a cylindrical,
circular,
elliptical, or polygonal profile.
[0072] Referring again to FIGS. 1-3, the base 120 supports at least one leg
130
extending upward in the Z direction from the base 120. The leg(s) 130 may be
configured to have a variable height for raising and lowering the torso 140
with respect to
the base 120. In some implementations, each leg 130 includes first and second
leg
portions 132, 134 that move with respect to each other (e.g., telescopic,
linear, and/or
angular movement). Rather than having extrusions of successively smaller
diameter
telescopically moving in and out of each other and out of a relatively larger
base
13
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
extrusion, the second leg portion 134, in the examples shown, moves
telescopically over
the first leg portion 132, thus allowing other components to be placed along
the second
leg portion 134 and potentially move with the second leg portion 134 to a
relatively close
proximity of the base 120. The leg 130 may include an actuator assembly 136
(FIG. 8C)
for moving the second leg portion 134 with respect to the first leg portion
132. The
actuator assembly 136 may include a motor driver 138a in communication with a
lift
motor 138b and an encoder 138c, which provides position feedback to the
controller 500.
[0073] Generally, telescopic arrangements include successively smaller
diameter
extrusions telescopically moving up and out of relatively larger extrusions at
the base 120
in order to keep a center of gravity CGL of the entire leg 130 as low as
possible.
Moreover, stronger and/or larger components can be placed at the bottom to
deal with the
greater torques that will be experienced at the base 120 when the leg 130 is
fully
extended. This approach, however, offers two problems. First, when the
relatively
smaller components are placed at the top of the leg 130, any rain, dust, or
other
particulate will tend to run or fall down the extrusions, infiltrating a space
between the
extrusions, thus obstructing nesting of the extrusions. This creates a very
difficult sealing
problem while still trying to maintain full mobility/articulation of the leg
130. Second, it
may be desirable to mount payloads or accessories on the robot 100. One common
place
to mount accessories is at the top of the torso 140. If the second leg portion
134 moves
telescopically in and out of the first leg portion, accessories and components
could only
be mounted above the entire second leg portion 134, if they need to move with
the torso
140. Otherwise, any components mounted on the second leg portion 134 would
limit the
telescopic movement of the leg 130.
[0074] By having the second leg portion 134 move telescopically over
the first leg
portion 132, the second leg portion 134 provides additional payload attachment
points
that can move vertically with respect to the base 120. This type of
arrangement causes
water or airborne particulate to run down the torso 140 on the outside of
every leg portion
132, 134 (e.g., extrusion) without entering a space between the leg portions
132, 134.
This greatly simplifies sealing any joints of the leg 130. Moreover,
payload/accessory
mounting features of the torso 140 and/or second leg portion 134 are always
exposed and
available no matter how the leg 130 is extended.
14
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0075] Referring to FIGS. 3 and 6, the leg(s) 130 support the torso
140, which may
have a shoulder 142 extending over and above the base 120. In the example
shown, the
torso 140 has a downward facing or bottom surface 144 (e.g., toward the base)
forming at
least part of the shoulder 142 and an opposite upward facing or top surface
146, with a
side surface 148 extending therebetween. The torso 140 may define various
shapes or
geometries, such as a circular or an elliptical shape having a central portion
141
supported by the leg(s) 130 and a peripheral free portion 143 that extends
laterally
beyond a lateral extent of the leg(s) 130, thus providing an overhanging
portion that
defines the downward facing surface 144. In some examples, the torso 140
defines a
polygonal or other complex shape that defines a shoulder, which provides an
overhanging
portion that extends beyond the leg(s) 130 over the base 120.
[0076] The robot 100 may include one or more accessory ports 170 (e.g.,
mechanical
and/or electrical interconnect points) for receiving payloads. The accessory
ports 170 can
be located so that received payloads do not occlude or obstruct sensors of the
sensor
system 400 (e.g., on the bottom surface 144 and/or top surface 146 of the
torso 140, etc.).
In some implementations, as shown in FIG. 6, the torso 140 includes one or
more
accessory ports 170 on a rearward portion 149 of the torso 140 for receiving a
payload in
the basket 360, for example, and so as not to obstruct sensors on a forward
portion 147 of
the torso 140 or other portions of the robot body 110.
[0077] Referring again to FIGS. 1-3 and 7, the torso 140 supports the neck
150,
which provides panning and tilting of the head 160 with respect to the torso
140. In the
examples shown, the neck 150 includes a rotator 152 and a tilter 154. The
rotator 152
may provide a range of angular movement OR (e.g., about the Z axis) of between
about
90 and about 360 . Other ranges are possible as well. Moreover, in some
examples, the
rotator 152 includes electrical connectors or contacts that allow continuous
360 rotation
of the head 160 with respect to the torso 140 in an unlimited number of
rotations while
maintaining electrical communication between the head 160 and the remainder of
the
robot 100. The tilter 154 may include the same or similar electrical
connectors or
contacts allow rotation of the head 160 with respect to the torso 140 while
maintaining
electrical communication between the head 160 and the remainder of the robot
100. The
rotator 152 may include a rotator motor 152m coupled to or engaging a ring 153
(e.g., a
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
toothed ring rack). The tilter 154 may move the head at an angle OT (e.g.,
about the Y
axis) with respect to the torso 140 independently of the rotator 152. In some
examples
that tilter 154 includes a tilter motor 155, which moves the head 160 between
an angle OT
of 90 with respect to Z-axis. Other ranges are possible as well, such as
450, etc. The
robot 100 may be configured so that the leg(s) 130, the torso 140, the neck
150, and the
head 160 stay within a perimeter of the base 120 for maintaining stable
mobility of the
robot 100. In the exemplary circuit schematic shown in FIG. 8F, the neck 150
includes a
pan-tilt assembly 151 that includes the rotator 152 and a tilter 154 along
with
corresponding motor drivers 156a, 156b and encoders 158a, 158b.
[0078] FIGS. 8A-8G provide exemplary schematics of circuitry for the robot
100.
FIGS. 8A-8C provide exemplary schematics of circuitry for the base 120, which
may
house the proximity sensors, such as the sonar proximity sensors 410 and the
cliff
proximity sensors 420, contact sensors 430, the laser scanner 440, the sonar
scanner 460,
and the drive system 200. The base 120 may also house the controller 500, the
power
source 105, and the leg actuator assembly 136. The torso 140 may house a
microcontroller 145, the microphone(s) 330, the speaker(s) 340, an imaging
sensor 450
(such as a scanning 3-D image sensor 450a), and a torso touch sensor system
480, which
allows the controller 500 to receive and respond to user contact or touches
(e.g., as by
moving the torso140 with respect to the base 120, panning and/or tilting the
neck 150,
and/or issuing commands to the drive system 200 in response thereto). The neck
150
may house a pan-tilt assembly 151 that may include a pan motor 152 having a
corresponding motor driver 156a and encoder 158a, and a tilt motor 154 having
a
corresponding motor driver 156b and encoder 158b. The head 160 may house one
or
more web pads 310 (e.g., capable of being a remote computing device in
communication
with the robot 100) and a camera 320.
[0079] The web pad 310 may executes a software application (e.g., a
tablet-based UI
component/application) that allows a remote user to visualize an environment
or scene 10
about the robot 100 and remotely control the robot 100. The software
application may
use hardware such as an Apple iPad 2 and/or a Motorola Xoom for a web pad 310
and a
PrimeSensor camera (available from PrimeSense, 28 Habarzel St., 4th floor, Tel-
Aviv,
69710, Israel) for the imaging sensor 450 or any other suitable hardware. The
software
16
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
application may use Apple iOS 4.x, Android 3.0 (a.k.a. Honeycomb), OpenGL ES
2Ø, or
any other suitable operating system or program.
[0080] Referring to FIGS. 1-4C and 9, to achieve reliable and robust
autonomous
movement, the sensor system 400 may include several different types of sensors
which
can be used in conjunction with one another to create a perception of the
robot's
environment sufficient to allow the robot 100 to make intelligent decisions
about actions
to take in that environment. The sensor system 400 may include one or more
types of
sensors supported by the robot body 110, which may include obstacle detection
obstacle
avoidance (ODOA) sensors, communication sensors, navigation sensors, etc. For
example, these sensors may include, but not limited to, proximity sensors,
contact
sensors, three-dimensional (3D) imaging / depth map sensors, a camera (e.g.,
visible light
and/or infrared camera), sonar, radar, LIDAR (Light Detection And Ranging,
which can
entail optical remote sensing that measures properties of scattered light to
find range
and/or other information of a distant target), LADAR (Laser Detection and
Ranging), etc.
In some implementations, the sensor system 400 includes ranging sonar sensors
410 (e.g.,
nine about a perimeter of the base 120), proximity cliff detectors 420,
contact sensors
430, a laser scanner 440, one or more 3-D imaging/depth sensors 450, and an
imaging
sonar 450.
[0081] There are several challenges involved in placing sensors on a
robotic platfoini.
First, the sensors need to be placed such that they have maximum coverage of
areas of
interest around the robot 100. Second, the sensors may need to be placed in
such a way
that the robot 100 itself causes an absolute minimum of occlusion to the
sensors; in
essence, the sensors cannot be placed such that they are "blinded" by the
robot itself
Third, the placement and mounting of the sensors should not be intrusive to
the rest of the
industrial design of the platform. In terms of aesthetics, it can be assumed
that a robot
with sensors mounted inconspicuously is more "attractive" than otherwise. In
terms of
utility, sensors should be mounted in a manner so as not to interfere with
normal robot
operation (snagging on obstacles, etc.).
[0082] In some implementations, the sensor system 400 includes a set or
an array of
proximity sensors 410, 420 in communication with the controller 500 and
arranged in one
or more zones or portions of the robot 100 (e.g., disposed on or near the base
body
17
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
portion 124a, 124b, 124c of the robot body 110) for detecting any nearby or
intruding
obstacles. The proximity sensors 410, 420 may be converging infrared (IR)
emitter-
sensor elements, sonar sensors, ultrasonic sensors, and/or imaging sensors
(e.g., 3D depth
map image sensors) that provide a signal to the controller 500 when an object
is within a
given range of the robot 100.
[0083] In the example shown in FIGS. 4A-4C, the robot 100 includes an
array of
sonar-type proximity sensors 410 disposed (e.g., substantially equidistant)
around the
base body 120 and arranged with an upward field of view. First, second, and
third sonar
proximity sensors 410a, 410b, 410c are disposed on or near the first (forward)
base body
portion 124a, with at least one of the sonar proximity sensors near a radially
outer-most
edge 125a of the first base body 124a. Fourth, fifth, and sixth sonar
proximity sensors
410d, 410e, 410f are disposed on or near the second (right) base body portion
124b, with
at least one of the sonar proximity sensors near a radially outer-most edge
125b of the
second base body 124b. Seventh, eighth, and ninth sonar proximity sensors
410g, 410h,
410i are disposed on or near the third (right) base body portion 124c, with at
least one of
the sonar proximity sensors near a radially outer-most edge 125c of the third
base body
124c. This configuration provides at least three zones of detection.
[0084] In some examples, the set of sonar proximity sensors 410 (e.g.,
410a-410i)
disposed around the base body 120 are arranged to point upward (e.g.,
substantially in the
Z direction) and optionally angled outward away from the Z axis, thus creating
a
detection curtain 412 around the robot 100. Each sonar proximity sensor 410a-
410i may
have a shroud or emission guide 414 that guides the sonar emission upward or
at least not
toward the other portions of the robot body 110 (e.g., so as not to detect
movement of the
robot body 110 with respect to itself). The emission guide 414 may define a
shell or half
shell shape. In the example shown, the base body 120 extends laterally beyond
the leg
130, and the sonar proximity sensors 410 (e.g., 410a-410i) are disposed on the
base body
120 (e.g., substantially along a perimeter of the base body 120) around the
leg 130.
Moreover, the upward pointing sonar proximity sensors 410 are spaced to create
a
continuous or substantially continuous sonar detection curtain 412 around the
leg 130.
The sonar detection curtain 412 can be used to detect obstacles having
elevated lateral
protruding portions, such as table tops, shelves, etc.
18
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0085] The upward looking sonar proximity sensors 410 provide the
ability to see
objects that are primarily in the horizontal plane, such as table tops. These
objects, due to
their aspect ratio, may be missed by other sensors of the sensor system, such
as the laser
scanner 440 or imaging sensors 450, and as such, can pose a problem to the
robot 100.
The upward viewing sonar proximity sensors 410 arranged around the perimeter
of the
base 120 provide a means for seeing or detecting those type of
objects/obstacles.
Moreover, the sonar proximity sensors 410 can be placed around the widest
points of the
base perimeter and angled slightly outwards, so as not to be occluded or
obstructed by the
torso 140 or head 160 of the robot 100, thus not resulting in false positives
for sensing
portions of the robot 100 itself In some implementations, the sonar proximity
sensors
410 are arranged (upward and outward) to leave a volume about the torso 140
outside of
a field of view of the sonar proximity sensors 410 and thus free to receive
mounted
payloads or accessories, such as the basket 460. The sonar proximity sensors
410 can be
recessed into the base body 124 to provide visual concealment and no external
features to
snag on or hit obstacles.
[0086] The sensor system 400 may include or more sonar proximity
sensors 410
(e.g., a rear proximity sensor 410j) directed rearward (e.g., opposite to the
forward drive
direction F) for detecting obstacles while backing up. The rear sonar
proximity sensor
410j may include an emission guide 414 to direct its sonar detection field
412.
Moreover, the rear sonar proximity sensor 410j can be used for ranging to
determine a
distance between the robot 100 and a detected object in the field of view of
the rear sonar
proximity sensor 410j (e.g., as "back-up alert"). In some examples, the rear
sonar
proximity sensor 410j is mounted recessed within the base body 120 so as to
not provide
any visual or functional irregularity in the housing form.
[0087] Referring to FIGS. 3 and 4B, in some implementations, the robot 100
includes
cliff proximity sensors 420 arranged near or about the drive wheels 210a,
210b, 210c, so
as to allow cliff detection before the drive wheels 210a, 210b, 210c encounter
a cliff (e.g.,
stairs). For example, a cliff proximity sensors 420 can be located at or near
each of the
radially outer-most edges 125a-c of the base bodies 124a-c and in locations
therebetween.
In some cases, cliff sensing is implemented using infrared (IR) proximity or
actual range
sensing, using an infrared emitter 422 and an infrared detector 424 angled
toward each
19
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
other so as to have an overlapping emission and detection fields, and hence a
detection
zone, at a location where a floor should be expected. IR proximity sensing can
have a
relatively narrow field of view, may depend on surface albedo for reliability,
and can
have varying range accuracy from surface to surface. As a result, multiple
discrete
sensors can be placed about the perimeter of the robot 100 to adequately
detect cliffs
from multiple points on the robot 100. Moreover, IR proximity based sensors
typically
cannot discriminate between a cliff and a safe event, such as just after the
robot 100
climbs a threshold.
[0088] The cliff proximity sensors 420 can detect when the robot 100
has
encountered a falling edge of the floor, such as when it encounters a set of
stairs. The
controller 500 (executing a control system) may execute behaviors that cause
the robot
100 to take an action, such as changing its direction of travel, when an edge
is detected.
In some implementations, the sensor system 400 includes one or more secondary
cliff
sensors (e.g., other sensors configured for cliff sensing and optionally other
types of
sensing). The cliff detecting proximity sensors 420 can be arranged to provide
early
detection of cliffs, provide data for discriminating between actual cliffs and
safe events
(such as climbing over thresholds), and be positioned down and out so that
their field of
view includes at least part of the robot body 110 and an area away from the
robot body
110. In some implementations, the controller 500 executes cliff detection
routine that
identifies and detects an edge of the supporting work surface (e.g., floor),
an increase in
distance past the edge of the work surface, and/or an increase in distance
between the
robot body 110 and the work surface. This implementation allows: 1) early
detection of
potential cliffs (which may allow faster mobility speeds in unknown
environments); 2)
increased reliability of autonomous mobility since the controller 500 receives
cliff
imaging information from the cliff detecting proximity sensors 420 to know if
a cliff
event is truly unsafe or if it can be safely traversed (e.g., such as climbing
up and over a
threshold); 3) a reduction in false positives of cliffs (e.g., due to the use
of edge detection
versus the multiple discrete IR proximity sensors with a narrow field of
view).
Additional sensors arranged as "wheel drop" sensors can be used for redundancy
and for
detecting situations where a range-sensing camera cannot reliably detect a
certain type of
cliff.
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0089] Threshold and step detection allows the robot 100 to effectively
plan for either
traversing a climb-able threshold or avoiding a step that is too tall. This
can be the same
for random objects on the work surface that the robot 100 may or may not be
able to
safely traverse. For those obstacles or thresholds that the robot 100
determines it can
climb, knowing their heights allows the robot 100 to slow down appropriately,
if deemed
needed, to allow for a smooth transition in order to maximize smoothness and
minimize
any instability due to sudden accelerations. In some implementations,
threshold and step
detection is based on object height above the work surface along with geometry
recognition (e.g., discerning between a threshold or an electrical cable
versus a blob, such
as a sock). Thresholds may be recognized by edge detection. The controller 500
may
receive imaging data from the cliff detecting proximity sensors 420 (or
another imaging
sensor on the robot 100), execute an edge detection routine, and issue a drive
command
based on results of the edge detection routine. The controller 500 may use
pattern
recognition to identify objects as well. Threshold detection allows the robot
100 to
change its orientation with respect to the threshold to maximize smooth step
climbing
ability.
[0090] The proximity sensors 410, 420 may function alone, or as an
alternative, may
function in combination with one or more contact sensors 430 (e.g., bump
switches) for
redundancy. For example, one or more contact or bump sensors 430 on the robot
body
110 can detect if the robot 100 physically encounters an obstacle. Such
sensors may use
a physical property such as capacitance or physical displacement within the
robot 100 to
detel __ mine when it has encountered an obstacle. In some implementations,
each base
body portion 124a, 124b, 124c of the base 120 has an associated contact sensor
430 (e.g.,
capacitive sensor, read switch, etc.) that detects movement of the
corresponding base
body portion 124a, 124b, 124c with respect to the base chassis 122 (see e.g.,
FIG. 4A).
For example, each base body 124a-c may move radially with respect to the Z
axis of the
base chassis 122, so as to provide 3-way bump detection.
[0091] Referring to FIGS. 1-4C, 9 and 10A, in some implementations, the
sensor
system 400 includes a laser scanner 440 mounted on a forward portion of the
robot body
110 and in communication with the controller 500. In the examples shown, the
laser
scanner 440 is mounted on the base body 120 facing forward (e.g., having a
field of view
21
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
along the forward drive direction F) on or above the first base body 124a
(e.g., to have
maximum imaging coverage along the drive direction F of the robot). Moreover,
the
placement of the laser scanner on or near the front tip of the triangular base
120 means
that the external angle of the robotic base (e.g., 300 degrees) is greater
than a field of
view 442 of the laser scanner 440 (e.g., ¨285 degrees), thus preventing the
base 120 from
occluding or obstructing the detection field of view 442 of the laser scanner
440. The
laser scanner 440 can be mounted recessed within the base body 124 as much as
possible
without occluding its fields of view, to minimize any portion of the laser
scanner sticking
out past the base body 124 (e.g., for aesthetics and to minimize snagging on
obstacles).
[0092] The laser scanner 440 scans an area about the robot 100 and the
controller
500, using signals received from the laser scanner 440, creates an environment
map or
object map of the scanned area. The controller 500 may use the object map for
navigation, obstacle detection, and obstacle avoidance. Moreover, the
controller 500 may
use sensory inputs from other sensors of the sensor system 400 for creating
object map
and/or for navigation.
[0093] In some examples, the laser scanner 440 is a scanning LIDAR,
which may use
a laser that quickly scans an area in one dimension, as a "main" scan line,
and a time-of-
flight imaging element that uses a phase difference or similar technique to
assign a depth
to each pixel generated in the line (returning a two dimensional depth line in
the plane of
scanning). In order to generate a three dimensional map, the LIDAR can perform
an
"auxiliary" scan in a second direction (for example, by "nodding" the
scanner). This
mechanical scanning technique can be complemented, if not supplemented, by
technologies such as the "Flash" LIDAR/LADAR and "Swiss Ranger" type focal
plane
imaging element sensors, techniques which use semiconductor stacks to permit
time of
flight calculations for a full 2-D matrix of pixels to provide a depth at each
pixel, or even
a series of depths at each pixel (with an encoded illuminator or illuminating
laser).
[0094] The sensor system 400 may include one or more three-dimensional
(3-D)
image sensors 450 in communication with the controller 500. If the 3-D image
sensor
450 has a limited field of view, the controller 500 or the sensor system 400
can actuate
the 3-D image sensor 450a in a side-to-side scanning manner to create a
relatively wider
field of view to perform robust ODOA.
22
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[0095] Referring again to FIG. 2 and 4A-4C, the sensor system 400 may
include an
inertial measurement unit (IMU) 470 in communication with the controller 500
to
measure and monitor a moment of inertia of the robot 100 with respect to the
overall
center of gravity CGR of the robot 100.
[0096] The controller 500 may monitor any deviation in feedback from the
IMU 470
from a threshold signal corresponding to normal unencumbered operation. For
example,
if the robot begins to pitch away from an upright position, it may be "clothes
lined" or
otherwise impeded, or someone may have suddenly added a heavy payload. In
these
instances, it may be necessary to take urgent action (including, but not
limited to, evasive
maneuvers, recalibration, and/or issuing an audio/visual warning) in order to
assure safe
operation of the robot 100.
[0097] Since robot 100 may operate in a human environment, it may
interact with
humans and operate in spaces designed for humans (and without regard for robot
constraints). The robot 100 can limit its drive speeds and accelerations when
in a
congested, constrained, or highly dynamic environment, such as at a cocktail
party or
busy hospital. However, the robot 100 may encounter situations where it is
safe to drive
relatively fast, as in a long empty corridor, but yet be able to decelerate
suddenly, as
when something crosses the robots' motion path.
[0098] When accelerating from a stop, the controller 500 may take into
account a
moment of inertia of the robot 100 from its overall center of gravity CGR to
prevent robot
tipping. The controller 500 may use a model of its pose, including its current
moment of
inertia. When payloads are supported, the controller 500 may measure a load
impact on
the overall center of gravity CGR and monitor movement of the robot moment of
inertia.
For example, the torso 140 and/or neck 150 may include strain gauges to
measure strain.
If this is not possible, the controller 500 may apply a test torque command to
the drive
wheels 210 and measure actual linear and angular acceleration of the robot
using the IMU
470, in order to experimentally determine safe limits.
[0099] During a sudden deceleration, a commanded load on the second and
third
drive wheels 210b, 210c (the rear wheels) is reduced, while the first drive
wheel 210a
(the front wheel) slips in the forward drive direction and supports the robot
100. If the
loading of the second and third drive wheels 210b, 210c (the rear wheels) is
23
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
asymmetrical, the robot 100 may "yaw" which will reduce dynamic stability. The
IMU
470 (e.g., a gyro) can be used to detect this yaw and command the second and
third drive
wheels 210b, 210c to reorient the robot 100.
[00100] Referring to FIGS. 1-3, 9 and 10A, in some implementations, the
robot 100
includes a scanning 3-D image sensor 450a mounted on a forward portion of the
robot
body 110 with a field of view along the forward drive direction F (e.g., to
have maximum
imaging coverage along the drive direction F of the robot). The scanning 3-D
image
sensor 450a can be used primarily for obstacle detection/obstacle avoidance
(ODOA). In
the example shown, the scanning 3-D image sensor 450a is mounted on the torso
140
underneath the shoulder 142 or on the bottom surface 144 and recessed within
the torso
140 (e.g., flush or past the bottom surface 144), as shown in FIG. 3, for
example, to
prevent user contact with the scanning 3-D image sensor 450a. The scanning 3-D
image
sensor 450 can be arranged to aim substantially downward and away from the
robot body
110, so as to have a downward field of view 452 in front of the robot 100 for
obstacle
detection and obstacle avoidance (ODOA) (e.g., with obstruction by the base
120 or other
portions of the robot body 110). Placement of the scanning 3-D image sensor
450a on or
near a forward edge of the torso 140 allows the field of view of the 3-D image
sensor 450
(e.g., ¨285 degrees) to be less than an external surface angle of the torso
140 (e.g., 300
degrees) with respect to the 3-D image sensor 450, thus preventing the torso
140 from
occluding or obstructing the detection field of view 452 of the scanning 3-D
image sensor
450a. Moreover, the scanning 3-D image sensor 450a (and associated actuator)
can be
mounted recessed within the torso 140 as much as possible without occluding
its fields of
view (e.g., also for aesthetics and to minimize snagging on obstacles). The
distracting
scanning motion of the scanning 3-D image sensor 450a is not visible to a
user, creating a
less distracting interaction experience. Unlike a protruding sensor or
feature, the recessed
scanning 3-D image sensor 450a will not tend to have unintended interactions
with the
environment (snagging on people, obstacles, etc.), especially when moving or
scanning,
as virtually no moving part extends beyond the envelope of the torso 140.
[00101] In some implementations, the sensor system 400 includes additional 3-D
image sensors 450 disposed on the base body 120, the leg 130, the neck 150,
and/or the
head 160. In the example shown in FIG. 1, the robot 100 includes 3-D image
sensors 450
24
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No. 225899-318442
on the base body 120, the torso 140, and the head 160. In the example shown in
FIG. 2,
the robot 100 includes 3-D image sensors 450 on the base body 120, the torso
140, and
the head 160. In the example shown in FIG. 9, the robot 100 includes 3-D image
sensors
450 on the leg 130, the torso 140, and the neck 150. Other configurations are
possible as
well. One 3-D image sensor 450 (e.g., on the neck 150 and over the head 160)
can be
used for people recognition, gesture recognition, and/or videoconferencing,
while another
3-D image sensor 450 (e.g., on the base 120 and/or the leg 130) can be used
for
navigation and/or obstacle detection and obstacle avoidance.
[00102] A forward facing 3-D image sensor 450 disposed on the neck 150 and/or
the
head 160 can be used for person, face, and/or gesture recognition of people
about the
robot 100. For example, using signal inputs from the 3-D image sensor 450 on
the head
160, the controller 500 may recognize a user by creating a three-dimensional
map of the
viewed/captured user's face and comparing the created three-dimensional map
with
known 3-D images of people's faces and determining a match with one of the
known 3-D
facial images. Facial recognition may be used for validating users as
allowable users of
the robot 100. Moreover, one or more of the 3-D image sensors 450 can be used
for
determining gestures of person viewed by the robot 100, and optionally
reacting based on
the determined gesture(s) (e.g., hand pointing, waving, and or hand signals).
For
example, the controller 500 may issue a drive command in response to a
recognized hand
point in a particular direction.
[00103] FIG. 10B provides a schematic view of a robot 900 having a camera 910,
sonar sensors 920, and a laser range finder 930 all mounted on a robot body
905 and each
having a field of view parallel or substantially parallel to the ground G.
This arrangement
allows detection of objects at a distance. In the example, a laser range
finder 930 detects
objects close to the ground G, a ring of ultrasonic sensors (sonars) 920
detect objects
further above the ground G, and the camera 910 captures a large portion of the
scene
from a high vantage point. The key feature of this design is that the sensors
910, 920,
930 are all oriented parallel to the ground G. One advantage of this
arrangement is that
computation can be simplified, in the sense that a distance to an object
determined by the
using one or more of the sensors 910, 920, 930 is also the distance the robot
900 can
travel before it contacts an object in a corresponding given direction. A
drawback of this
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
arrangement is that to get good coverage of the robot's surroundings, many
levels of
sensing are needed. This can be prohibitive from a cost or computation
perspective,
which often leads to large gaps in a sensory field of view of all the sensors
910, 920, 930
of the robot 900.
[00104] In some implementations, the robot includes a sonar scanner 460 for
acoustic
imaging of an area surrounding the robot 100. In the examples shown in FIGS. 1
and 3,
the sonar scanner 460 is disposed on a forward portion of the base body 120.
[00105] Referring to FIGS. 1, 3B and 10A, in some implementations, the robot
100
uses the laser scanner or laser range finder 440 for redundant sensing, as
well as a rear-
facing sonar proximity sensor 410j for safety, both of which are oriented
parallel to the
ground G. The robot 100 may include first and second 3-D image sensors 450a,
450b
(depth cameras) to provide robust sensing of the environment around the robot
100. The
first 3-D image sensor 450a is mounted on the torso 140 and pointed downward
at a fixed
angle to the ground G. By angling the first 3-D image sensor 450a downward,
the robot
100 receives dense sensor coverage in an area immediately forward or adjacent
to the
robot 100, which is relevant for short-term travel of the robot 100 in the
forward
direction. The rear-facing sonar 410j provides object detection when the robot
travels
backward. If backward travel is typical for the robot 100, the robot 100 may
include a
third 3D image sensor 450 facing downward and backward to provide dense sensor
coverage in an area immediately rearward or adjacent to the robot 100.
[00106] The second 3-D image sensor 450b is mounted on the head 160, which can
pan and tilt via the neck 150. The second 3-D image sensor 450b can be useful
for
remote driving since it allows a human operator to see where the robot 100 is
going. The
neck 150 enables the operator tilt and/or pan the second 3-D image sensor 450b
to see
both close and distant objects. Panning the second 3-D image sensor 450b
increases an
associated horizontal field of view. During fast travel, the robot 100 may
tilt the second
3-D image sensor 450b downward slightly to increase a total or combined field
of view
of both 3-D image sensors 450a, 450b, and to give sufficient time for the
robot 100 to
avoid an obstacle (since higher speeds generally mean less time to react to
obstacles). At
slower speeds, the robot 100 may tilt the second 3-D image sensor 450b upward
or
substantially parallel to the ground G to track a person that the robot 100 is
meant to
26
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
follow. Moreover, while driving at relatively low speeds, the robot 100 can
pan the
second 3-D image sensor 450b to increase its field of view around the robot
100. The
first 3-D image sensor 450a can stay fixed (e.g., not moved with respect to
the base 120)
when the robot is driving to expand the robot's perceptual range.
[00107] The 3-D image sensors 450 may be capable of producing the following
types
of data: (i) a depth map, (ii) a reflectivity based intensity image, and/or
(iii) a regular
intensity image. The 3-D image sensors 450 may obtain such data by image
pattern
matching, measuring the flight time and/or phase delay shift for light emitted
from a
source and reflected off of a target.
[00108] In some implementations, reasoning or control software, executable
on a
processor (e.g., of the robot controller 500), uses a combination of
algorithms executed
using various data types generated by the sensor system 400. The reasoning
software
processes the data collected from the sensor system 400 and outputs data for
making
navigational decisions on where the robot 100 can move without colliding with
an
obstacle, for example. By accumulating imaging data over time of the robot's
surroundings, the reasoning software can in turn apply effective methods to
selected
segments of the sensed image(s) to improve depth measurements of the 3-D image
sensors 450. This may include using appropriate temporal and spatial averaging
techniques.
[00109] The reliability of executing robot collision free moves may be
based on: (i) a
confidence level built by high level reasoning over time and (ii) a depth-
perceptive sensor
that accumulates three major types of data for analysis - (a) a depth image,
(b) an active
illumination image and (c) an ambient illumination image. Algorithms cognizant
of the
different types of data can be executed on each of the images obtained by the
depth-
perceptive imaging sensor 450. The aggregate data may improve the confidence
level a
compared to a system using only one of the kinds of data.
[00110] The 3-D image sensors 450 may obtain images containing depth and
brightness data from a scene about the robot 100 (e.g., a sensor view portion
of a room or
work area) that contains one or more objects. The controller 500 may be
configured to
determine occupancy data for the object based on the captured reflected light
from the
scene. Moreover, the controller 500, in some examples, issues a drive command
to the
27
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
drive system 200 based at least in part on the occupancy data to
circumnavigate obstacles
(i.e., the object in the scene). The 3-D image sensors 450 may repeatedly
capture scene
depth images for real-time decision making by the controller 500 to navigate
the robot
100 about the scene without colliding into any objects in the scene. For
example, the
speed or frequency in which the depth image data is obtained by the 3-D image
sensors
450 may be controlled by a shutter speed of the 3-D image sensors 450. In
addition, the
controller 500 may receive an event trigger (e.g., from another sensor
component of the
sensor system 400, such as proximity sensor 410, 420, notifying the controller
500 of a
nearby object or hazard. The controller 500, in response to the event trigger,
can cause
the 3-D image sensors 450 to increase a frequency at which depth images are
captured
and occupancy information is obtained.
[00111] Referring to FIG. 11, in some implementations, the 3-D imaging sensor
450
includes a light source 1172 that emits light onto a scene 10, such as the
area around the
robot 100 (e.g., a room). The imaging sensor 450 may also include an imager
1174 (e.g.,
an array of light-sensitive pixels 1174p) which captures reflected light from
the scene 10,
including reflected light that originated from the light source 1172 (e.g., as
a scene depth
image). In some examples, the imaging sensor 450 includes a light source lens
1176
and/or a detector lens 1178 for manipulating (e.g., speckling or focusing) the
emitted and
received reflected light, respectively. The robot controller 500 or a sensor
controller (not
shown) in communication with the robot controller 500 receives light signals
from the
imager 1174 (e.g., the pixels 1174p) to determine depth information for an
object 12 in
the scene 10 based on image pattern matching and/or a time-of-flight
characteristic of the
reflected light captured by the imager 1174.
[00112] FIG. 12 provides an exemplary arrangement 1200 of operations for
operating
the imaging sensor 450. With additional reference to FIG. 10A, the operations
include
emitting 1202 light onto a scene 10 about the robot 100 and receiving 1204
reflections of
the emitted light from the scene 10 on an imager (e.g., array of light-
sensitive pixels).
The operations further include the controller 500 receiving 1206 light
detection signals
from the imager, detecting 1208 one or more features of an object 12 in the
scene 10
using image data derived from the light detection signals, and tracking 1210 a
position of
the detected feature(s) of the object 12 in the scene 10 using image depth
data derived
28
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
from the light detection signals. The operations may include repeating 1212
the
operations of emitting 1202 light, receiving 1204 light reflections, receiving
1206 light
detection signals, detecting 1208 object feature(s), and tracking 12010 a
position of the
object feature(s) to increase a resolution of the image data or image depth
data, and/or to
provide a confidence level.
[00113] The repeating 1212 operation can be performed at a relatively
slow rate (e.g.,
slow frame rate) for relatively high resolution, an intermediate rate, or a
high rate with a
relatively low resolution. The frequency of the repeating 1212 operation may
be
adjustable by the robot controller 500. In some implementations, the
controller 500 may
raise or lower the frequency of the repeating 1212 operation upon receiving an
event
trigger. For example, a sensed item in the scene may trigger an event that
causes an
increased frequency of the repeating 1212 operation to sense an possibly
eminent object
12 (e.g., doorway, threshold, or cliff) in the scene 10. In additional
examples, a lapsed
time event between detected objects 12 may cause the frequency of the
repeating 1212
operation to slow down or stop for a period of time (e.g., go to sleep until
awakened by
another event). In some examples, the operation of detecting 1208 one or more
features
of an object 12 in the scene 10 triggers a feature detection event causing a
relatively
greater frequency of the repeating operation 1212 for increasing the rate at
which image
depth data is obtained. A relatively greater acquisition rate of image depth
data can allow
for relatively more reliable feature tracking within the scene.
[00114] The operations also include outputting 1214 navigation data for
circumnavigating the object 12 in the scene 10. In some implementations, the
controller
500 uses the outputted navigation data to issue drive commands to the drive
system 200
to move the robot 100 in a manner that avoids a collision with the object 12.
[00115] In some implementations, the sensor system 400 detects multiple
objects 12
within the scene 10 about the robot 100 and the controller 500 tracks the
positions of each
of the detected objects 12. The controller 500 may create an occupancy map of
objects
12 in an area about the robot 100, such as the bounded area of a room. The
controller
500 may use the image depth data of the sensor system 400 to match a scene 10
with a
portion of the occupancy map and update the occupancy map with the location of
tracked
objects 12.
29
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00116] Referring to FIG. 13, in some implementations, the 3-D image
sensor 450
includes a three-dimensional (3D) speckle camera 1300, which allows image
mapping
through speckle decorrelation. The speckle camera 1300 includes a speckle
emitter 1310
(e.g., of infrared, ultraviolet, and/or visible light) that emits a speckle
pattern into the
scene 10 (as a target region) and an imager 1320 that captures images of the
speckle
pattern on surfaces of an object 12 in the scene 10.
[00117] The speckle emitter 1310 may include a light source 1312, such
as a laser,
emitting a beam of light into a diffuser 1314 and onto a reflector 1316 for
reflection, and
hence projection, as a speckle pattern into the scene 10. The imager 1320 may
include
objective optics 1322, which focus the image onto an image sensor 1324 having
an array
of light detectors 1326, such as a CCD or CMOS-based image sensor. Although
the
optical axes of the speckle emitter 1310 and the imager 1320 are shown as
being
collinear, in a decorrelation mode for example, the optical axes of the
speckle emitter
1310 and the imager 1320 may also be non-collinear, while in a cross-
correlation mode
for example, such that an imaging axis is displaced from an emission axis.
[00118] The speckle emitter 1310 emits a speckle pattern into the scene 10 and
the
imager 1320 captures reference images of the speckle pattern in the scene 10
at a range of
different object distances Zn from the speckle emitter 1310 (e.g., where the Z-
axis can be
defined by the optical axis of imager 1320). In the example shown, reference
images of
the projected speckle pattern are captured at a succession of planes at
different, respective
distances from the origin, such as at the fiducial locations marked Z1, Z2,
Z3, and so on.
The distance between reference images, AZ, can be set at a threshold distance
(e.g., 5
mm) or adjustable by the controller 500 (e.g., in response to triggered
events). The
speckle camera 1300 archives and indexes the captured reference images to the
respective
emission distances to allow decorrelation of the speckle pattern with distance
from the
speckle emitter 1310 to perform distance ranging of objects 12 captured in
subsequent
images. Assuming AZ to be roughly equal to the distance between adjacent
fiducial
distances Z15 Z25 Z35 , the speckle pattern on the object 12 at location ZA
can be
correlated with the reference image of the speckle pattern captured at Z2, for
example.
On the other hand, the speckle pattern on the object 12 at ZB can be
correlated with the
reference image at Z3, for example. These correlation measurements give the
CA 02824 60 6 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
approximate distance of the object 12 from the origin. To map the object 12 in
three
dimensions, the speckle camera 1300 or the controller 500 receiving
information from the
speckle camera 1300 can use local cross-correlation with the reference image
that gave
the closest match.
[00119] Other details and features on 3D image mapping using speckle ranging,
via
speckle cross-correlation using triangulation or decorrelation, for example,
which may
combinable with those described herein, can be found in PCT Patent Application
PCT/IL2006/000335; the contents of which is hereby incorporated by reference
in its
entirety.
[00120] FIG. 14 provides an exemplary arrangement 1400 of operations for
operating
the speckle camera 1300. The operations include emitting 1402 a speckle
pattern into the
scene 10 and capturing 1404 reference images (e.g., of a reference object 12)
at different
distances from the speckle emitter 1310. The operations further include
emitting 1406 a
speckle pattern onto a target object 12 in the scene 10 and capturing 1408
target images
of the speckle pattern on the object 12. The operations further include
comparing 1410
the target images (of the speckled object) with different reference images to
identify a
reference pattern that correlates most strongly with the speckle pattern on
the target
object 12 and deterniining 1412 an estimated distance range of the target
object 12 within
the scene 10. This may include determining a primary speckle pattern on the
object 12
and finding a reference image having speckle pattern that correlates most
strongly with
the primary speckle pattern on the object 12. The distance range can be
determined from
the corresponding distance of the reference image.
[00121] The operations optionally include constructing 1414 a 3D map of the
surface
of the object 12 by local cross-correlation between the speckle pattern on the
object 12
and the identified reference pattern, for example, to determine a location of
the object 12
in the scene. This may include determining a primary speckle pattern on the
object 12
and finding respective offsets between the primary speckle pattern on multiple
areas of
the object 12 in the target image and the primary speckle pattern in the
identified
reference image so as to derive a three-dimensional (3D) map of the object.
The use of
solid state components for 3D mapping of a scene provides a relatively
inexpensive
solution for robot navigational systems.
31
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00122] Typically, at least some of the different, respective distances
are separated
axially by more than an axial length of the primary speckle pattern at the
respective
distances. Comparing the target image to the reference images may include
computing a
respective cross-correlation between the target image and each of at least
some of the
reference images, and selecting the reference image having the greatest
respective cross-
correlation with the target image.
[00123] The operations may include repeating 1416 operations 1402-1412
or
operations 1406-1412, and optionally operation 1414, (e.g., continuously) to
track motion
of the object 12 within the scene 10. For example, the speckle camera 1300 may
capture
a succession of target images while the object 12 is moving for comparison
with the
reference images.
[00124] Other details and features on 3D image mapping using speckle ranging,
which
may combinable with those described herein, can be found in U.S. Patent
7,433,024; U.S.
Patent Application Publication No. 2008/0106746, entitled "Depth-varying light
fields
for three dimensional sensing"; U.S. Patent Application Publication No.
2010/0118123,
entitled "Depth Mapping Using Projected Patterns"; U.S. Patent Application
Publication
No. 2010/0034457, Entitled "Modeling Of Humanoid Forms From Depth Maps"; U.S.
Patent Application Publication No. 2010/0020078, Entitled "Depth Mapping Using
Multi-Beam Illumination"; U.S. Patent Application Publication No.
2009/0185274,
Entitled "Optical Designs For Zero Order Reduction"; U.S. Patent Application
Publication No. 2009/0096783, Entitled "Three-Dimensional Sensing Using
Speckle
Patterns"; U.S. Patent Application Publication No. 2008/0240502, Entitled
"Depth
Mapping Using Projected Patterns"; and U.S. Patent Application Publication No.
2008/0106746, Entitled "Depth-Varying Light Fields For Three Dimensional
Sensing";
the contents of which are hereby incorporated by reference in their
entireties.
1001251 Referring to FIG. 15, in some implementations, the 3-D imaging
sensor 450
includes a 3D time-of-flight (TOF) camera 1500 for obtaining depth image data.
The 3D
TOF camera 1500 includes a light source 1510, a complementary metal oxide
semiconductor (CMOS) sensor 1520 (or charge-coupled device (CCD)), a lens
1530, and
control logic or a camera controller 1540 having processing resources (and/or
the robot
controller 500) in communication with the light source 1510 and the CMOS
sensor 1520.
32
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
The light source 1510 may be a laser or light-emitting diode (LED) with an
intensity that
is modulated by a periodic signal of high frequency. In some examples, the
light source
1510 includes a focusing lens 1512. The CMOS sensor 1520 may include an array
of
pixel detectors 1522, or other arrangement of pixel detectors 1522, where each
pixel
detector 1522 is capable of detecting the intensity and phase of photonic
energy
impinging upon it. In some examples, each pixel detector 1522 has dedicated
detector
circuitry 1524 for processing detection charge output of the associated pixel
detector
1522. The lens 1530 focuses light reflected from a scene 10, containing one or
more
objects 12 of interest, onto the CMOS sensor 1520. The camera controller 1540
provides
a sequence of operations that formats pixel data obtained by the CMOS sensor
1520 into
a depth map and a brightness image. In some examples, the 3D TOF camera 1500
also
includes inputs / outputs (10) 1550 (e.g., in communication with the robot
controller
500), memory 1560, and/or a clock 1570 in communication with the camera
controller
1540 and/or the pixel detectors 1522 (e.g., the detector circuitry 1524).
[00126] FIG. 16 provides an exemplary arrangement 1600 of operations for
operating
the 3D TOF camera 1500. The operations include emitting 1602 a light pulse
(e.g.,
infrared, ultraviolet, and/or visible light) into the scene 10 and commencing
1604 timing
of the flight time of the light pulse (e.g., by counting clock pulses of the
clock 1570).
The operations include receiving 1606 reflections of the emitted light off one
or more
surfaces of an object 12 in the scene 10. The reflections may be off surfaces
of the object
12 that are at different distances 41 from the light source 1510. The
reflections are
received though the lens 1530 and onto pixel detectors 1522 of the CMOS sensor
1520.
The operations include receiving 1608 time-of-flight for each light pulse
reflection
received on each corresponding pixel detector 1522 of the CMOS sensor 1520.
During
the roundtrip time of flight (TOF) of a light pulse, a counter of the detector
circuitry 1523
of each respective pixel detector 1522 accumulates clock pulses. A larger
number of
accumulated clock pulses represents a longer TOF, and hence a greater distance
between
a light reflecting point on the imaged object 12 and the light source 1510.
The operations
further include determining 1610 a distance between the reflecting surface of
the object
12 for each received light pulse reflection and optionally constructing 1612 a
three-
dimensional object surface. In some implementations, the operations include
repeating
33
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
1614 operations 1602-1610 and optionally 1612 for tracking movement of the
object 12
in the scene 10.
[00127] Other details and features on 3D time-of-flight imaging, which may
combinable with those described herein, can be found in U.S. Patent No.
6,323,942,
entitled "CMOS Compatible 3-D Image Sensor"; U.S. Patent No. 6,515,740,
entitled
"Methods for CMOS-Compatible Three-Dimensional Image Sensing Using Quantum
Efficiency Modulation"; and PCT Patent Application PCT/US02/16621, entitled
"Method
and System to Enhance Dynamic Range Conversion Usable with CMOS Three-
Dimensional Imaging", the contents of which are hereby incorporated by
reference in
their entireties.
[00128] In some implementations, the 3-D imaging sensor 450 provides three
types of
information: (1) depth information (e.g., from each pixel detector 1522 of the
CMOS
sensor 1520 to a corresponding location on the scene 12); (2) ambient light
intensity at
each pixel detector location; and (3) the active illumination intensity at
each pixel
detector location. The depth information enables the position of the detected
object 12 to
be tracked over time, particularly in relation to the object's proximity to
the site of robot
deployment. The active illumination intensity and ambient light intensity are
different
types of brightness images. The active illumination intensity is captured from
reflections
of an active light (such as provided by the light source 1510) reflected off
of the target
object 12. The ambient light image is of ambient light reflected off of the
target object
12. The two images together provide additional robustness, particularly when
lighting
conditions are poor (e.g., too dark or excessive ambient lighting).
[00129] Image segmentation and classification algorithms may be used to
classify and
detect the position of objects 12 in the scene 10. Information provided by
these
algorithms, as well as the distance measurement information obtained from the
imaging
sensor 450, can be used by the robot controller 500 or other processing
resources. The
imaging sensor 450 can operate on the principle of time-of-flight, and more
specifically,
on detectable phase delays in a modulated light pattern reflected from the
scene 10,
including techniques for modulating the sensitivity of photodiodes for
filtering ambient
light.
34
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00130] The robot 100 may use the imaging sensor 450 for 1) mapping,
localization &
navigation; 2) object detection & object avoidance (ODOA); 3) object hunting
(e.g., to
find a person); 4) gesture recognition (e.g., for companion robots); 5) people
& face
detection; 6) people tracking; 7) monitoring manipulation of objects by the
robot 100;
and other suitable applications for autonomous operation of the robot 100.
[00131] In some implementations, at least one of 3-D image sensors 450 can be
a
volumetric point cloud imaging device (such as a speckle or time-of-flight
camera)
positioned on the robot 100 at a height of greater than 1 or 2 feet above the
ground and
directed to be capable of obtaining a point cloud from a volume of space
including a floor
plane in a direction of movement of the robot (via the omni-directional drive
system
200). In the examples shown in FIGS. 1 and 3, the first 3-D image sensor 450a
can be
positioned on the base 120 at height of greater than 1 or 2 feet above the
ground (or at a
height of about 1 or 2 feet above the ground) and aimed along the forward
drive direction
F to capture images (e.g., volumetric point cloud) of a volume including the
floor while
driving (e.g., for obstacle detection and obstacle avoidance). The second 3-D
image
sensor 450b is shown mounted on the head 160 (e.g., at a height greater than
about 3 or 4
feet above the ground), so as to be capable of obtaining skeletal recognition
and
definition point clouds from a volume of space adjacent the robot 100. The
controller
500 may execute skeletal/digital recognition software to analyze data of the
captured
volumetric point clouds.
[00132] Properly sensing objects 12 using the imaging sensor 450, despite
ambient
light conditions can be important. In many environments the lighting
conditions cover a
broad range from direct sunlight to bright fluorescent lighting to dim
shadows, and can
result in large variations in surface texture and basic reflectance of objects
12. Lighting
can vary within a given location and from scene 10 to scene 10 as well. In
some
implementations, the imaging sensor 450 can be used for identifying and
resolving
people and objects 12 in all situations with relatively little impact from
ambient light
conditions (e.g., ambient light rejection).
[00133] In some implementations, VGA resolution of the imaging sensor 450 is
640
horizontal by 480 vertical pixels; however, other resolutions are possible as
well, such.
320 x 240 (e.g., for short range sensors).
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00134] The imaging sensor 450 may include a pulse laser and camera iris to
act as a
bandpass filter in the time domain to look at objects 12 only within a
specific range. A
varying iris of the imaging sensor 450 can be used to detect objects 12 a
different
distances. Moreover, a pulsing higher power laser can be used for outdoor
applications.
[00135] Table 1 and Table 2 (below) provide exemplary features, parameters,
and/or
specifications of imaging sensors 450 for various applications. Sensor 1 can
be used as a
general purpose imaging sensor 450. Sensors 2 and 3 could be used on a human
interaction robot, and sensors 4 and 5 could be used on a coverage or cleaning
robot.
36
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
Unit Sensor 1 Sensor 2 Sensor 3 Sensor 4 Sensor 5
Long Short Long Short
Range Range Range Range
Dimensions
Width _ cm 18 <=18< 14 <14<= 6 <= 6 <= 6
Height cm 2.5 <2.5<4 <4<= 1.2 <= 1.2 <= 1.2
Depth cm 3.5 <=3.5<5 < 5<= .6 <=.6 <=.6
Operating Temp
Minimum C 5 5 5 5 5
Maximum C 40 40 40 40 40
Comm Port
Data interface USB 2.0 USB 2.0 USB 2.0 SPI SPI
Field-of-View
Horizontal deg 57.5 >=57.5 >70 >70 >70
Vertical deg 45 >=45 >=45 >=45 >40
Diagonal deg 69
Slatial Resolution
640 x
Depth image size _ 480 640 x 480
gl5cm mm
g20cm mm
A40cm mm
g80cm mm
A lm mm 1.7 1.7
g2m mm 3.4 3.4
@3m mm 5.1 5.1
g3.5m mm 6 6
Downsampling
QVGA pixels 320x240 320x240 320x240 320x240 320x240
QQVGA pixels 160x120 160x120 160x120 160x120 160x120
Table 1
37
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
Unit Sensor 1 Sensor 2 Sensor 3 Sensor 4 Sensor 5
Long Short Long Short
Range Range Range Range
Depth Resolution
cm 0.57
A2m cm 2.31
@3m cm 5.23
@3.5m cm 7.14
Minimum Object Size
cm 2.4 <=2.4 0.2
@2m cm 4.8 <=4.8
@3m cm , 7.2 <=7.2
@3.5m cm _8.4 <=8.4
Throughput
Frame rate _ fps 30 30 30 30 30
VGA depth image ms _ 44 <-44 <-44 <=44 <=44
QVGA depth
image ms 41 <-41 <-41 <-41 <-41
Range
0.25- 0.25- 0.15 -
In Spec. range m 0.8 - 3.5 0.8 - 3.5 1.50 1.50
1.0
0.15- 0.15- 0.10-
-
Observed range m 0.3 - 5 0.3 - 5 2.00 2.00 1.5
Color Image
Color camera CMOS N/R N/R N/R N/R
1280 x
1024
Audio
Built-in
microphones 2 N/R N/R N/R N/R
Data format 16
Sample rate 17746
External digital
audio inputs 4
Power
Power supply USB 2.0 USB 2.0 USB 2.0
Current
consumption 0.45
Max power
consumption 2.25 0.5
Table 2
38
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00136] Minimal sensor latency assures that objects 12 can be seen quickly
enough to
be avoided when the robot 100 is moving. Latency of the imaging sensor 450 can
be a
factor in reacting in real time to detected and recognized user gestures. In
some
examples, the imaging sensor 450 has a latency of about 44 ms. Images captured
by the
imaging sensor 450 can have an attributed time stamp, which can be used for
determining
at what robot pose an image was taken while translating or rotating in space.
[00137] A Serial Peripheral Interface Bus (SPI ) in communication with
the controller
500 may be used for communicating with the imaging sensor 450. Using an SPI
interface for the imaging sensor 450 does not limit its use for multi-node
distributed
sensor/actuator systems, and allows connection with an Ethernet enabled device
such as a
microprocessor or a field-programmable gate array (FPGA), which can then make
data
available over Ethernet and an EtherIO system, as described in U.S. Patent
Application
Serial No. 61/305,069, filed on February 16, 2010 and titled "Mobile Robot
Communication System," which is hereby incorporate by reference in its
entirety.
[00138] Since SPI is a limited protocol, an interrupt pin may be available
on the
interface to the imaging sensor 450 that would strobe or transition when an
image capture
is executed. The interrupt pin allows communication to the controller 500 of
when a
frame is captured. This allows the controller 500 to know that data is ready
to be read.
Additionally, the interrupt pin can be used by the controller 500 to capture a
timestamp
which indicates when the image was taken. Imaging output of the imaging sensor
450
can be time stamped (e.g., by a global clock of the controller 500), which can
be
referenced to compensate for latency. Moreover, the time stamped imaging
output from
multiple imaging sensors 450 (e.g., of different portions of the scene 10) can
be
synchronized and combined (e.g., stitched together). Over an EtherIO system,
an
interrupt time (on the interrupt pin) can be captured and made available to
higher level
devices and software on the EtherIO system. The robot 100 may include a multi-
node
distributed sensor/actuator systems that implements a clock synchronization
strategy,
such as 1EEE1588, which we can be applied to data captured from the imaging
sensor
450.
[00139] Both the SPI interface and EtherIO can be memory-address driven
interfaces.
Data in the form of bytes/words/double-words, for example, can be read from
the
39
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
imaging sensor 450 over the SPI interface, and made available in a memory
space of the
EtherIO system. For example, local registers and memory, such as direct memory
access
(DMA) memory, in an FPGA, can be used to control an EtherIO node of the
EtherIO
system.
[00140] In some cases, the robot 100 may need to scan the imaging sensor 450
from
side to side and/or up and down (e.g., to view an object 12 or around an
occlusion 16
(FIG. 17A)). For a differentially steered robot 100, this may involve rotating
the robot
100 in place with the drive system 200; or rotating a mirror, prism, variable
angle micro-
mirror, or MEMS mirror array associated with the imaging sensor 450.
[00141] The field of view 452 of the imaging sensor 450 having a view angle ev
less
than 360 can be enlarged to 360 degrees by optics, such as omni-directional,
fisheye,
catadioptric (e.g., parabolic mirror, telecentric lens), panamorph mirrors and
lenses.
Since the controller 500 may use the imaging sensor 450 for distance ranging,
inter alia,
but not necessarily for human-viewable images or video (e.g., for human
communications), distortion (e.g., warping) of the illumination of the light
source 1172
and/or the image capturing by the imager 1174 (FIG. 11) through optics is
acceptable for
distance ranging (e.g., as with the 3D speckle camera 1300 and/or the 3D TOF
camera
1500).
[00142] In some instances, the imaging sensor 450 may have difficulties
recognizing
and ranging black objects 12, surfaces of varied albedo, highly reflective
objects 12,
strong 3D structures, self-similar or periodic structures, or objects at or
just beyond the
field of view 452 (e.g., at or outside horizontal and vertical viewing field
angles). In such
instances, other sensors of the sensor system 400 can be used to supplement or
act as
redundancies to the imaging sensor 450.
[00143] In some implementations, the light source 1172 (e.g., of the 3D
speckle
camera 1300 and/or the 3D TOF camera 1500) includes an infrared (IR) laser, IR
pattern
illuminator, or other IR illuminator. A black object, especially black fabric
or carpet,
may absorb IR and fail to return a strong enough reflection for recognition by
the imager
1174. In this case, either a secondary mode of sensing (such as sonar) or a
technique for
self calibrating for surface albedo differences may be necessary to improve
recognition of
black objects.
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00144] A highly reflective object 12 or an object 12 with significant
specular
highlights (e.g., cylindrical or spherical) may make distance ranging
difficult for the
imaging sensor 450. Similarly, objects 12 that are extremely absorptive in the
wavelength of light for which the imaging sensor 450 is sensing, can pose
problems as
well. Objects 12, such as doors and window, which are made of glass can be
highly
reflective and, when ranged, either appear as if they are free space (infinite
range) or else
range as the reflection to the first non-specularly-reflective surface. This
may cause the
robot 100 to not see the object 12 as an obstacle, and, as a result, may
collide with the
window or door, possibly causing damage to the robot or to the object 12. In
order to
avoid this, the controller 500 may execute one or more algorithms that look
for
discontinuities in surfaces matching the size and shape (rectilinear) of a
typical window
pane or doorway. These surfaces can then be inferred as being obstacles and
not free
space. Another implementation for detecting reflective objects in the path of
the robot
includes using a reflection sensor that detects its own reflection. Upon
careful approach
of the obstacle or object 12, the reflection sensor can be used determine
whether there is a
specularly reflective object ahead, or if the robot can safely occupy the
space.
[00145] In the case of the 3D speckle camera 1300, the light source 1310 may
fail to
form a pattern recognizable on the surface of a highly reflective object 12 or
the imager
1320 may fail to recognize a speckle reflection from the highly reflective
object 12. In
the case of the 3D TOF camera 1500, the highly reflective object 12 may create
a multi-
path situation where the 3D TOF camera 1500 obtains a range to another object
12
reflected in the object 12 (rather than to the object itself). To remedy IR
failure modes,
the sensor system 400 may employ acoustic time of flight, millimeter wave
radar, stereo
or other vision techniques able to use even small reflections in the scene 10.
[00146] Mesh objects 12 may make distance ranging difficult for the imaging
sensor
450. If there are no objects 12 immediately behind mesh of a particular
porosity, the
mesh will appear as a solid obstacle 12. If an object 12 transits behind the
mesh,
however, and, in the case of the 3D speckle camera 1300, the speckles are able
to reflect
off the object 12 behind the mesh, the object will appear in the depth map
instead of the
mesh, even though it is behind it. If information is available about the
points that had
previously contributed to the identification of the mesh (before an object 12
transited
41
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
behind it), such information could be used to register the position of the
mesh in future
occupancy maps. By receiving information about the probabilistic correlation
of the
received speckle map at various distances, the controller 500 may determine
the locations
of multiple porous or mesh-like objects 12 in line with the imaging sensor
450.
[00147] The controller 500 may use imaging data from the imaging sensor 450
for
color/size/dimension blob matching. Identification of discrete objects 12 in
the scene 10
allows the robot 100 to not only avoid collisions, but also to search for
objects 12. The
human interface robot 100 may need to identify humans and target objects 12
against the
background of a home or office environment. The controller 500 may execute one
or
more color map blob-finding algorithms on the depth map(s) derived from the
imaging
data of the imaging sensor 450 as if the maps were simple grayscale maps and
search for
the same "color" (that is, continuity in depth) to yield continuous objects 12
in the scene
10. Using color maps to augment the decision of how to segment objects 12
would
further amplify object matching, by allowing segmentation in the color space
as well as
in the depth space. The controller 500 may first detect objects 12 by depth,
and then
further segment the objects 12 by color. This allows the robot 100 to
distinguish between
two objects 12 close to or resting against one another with differing optical
qualities.
[00148] In implementations where the sensor system 400 includes only one
imaging
sensor 450 (e.g., camera) for object detection, the imaging sensor 450 may
have problems
imaging surfaces in the absence of scene texture and may not be able to
resolve the scale
of the scene. Moreover, mirror and/or specular highlights of an object 12 can
cause
saturation in a group of pixels 1174p of the imager 1174 (e.g., saturating a
corresponding
portion of a captured image); and in color images, the specular highlights can
appear
differently from different viewpoints, thereby hampering image matching, as
for the
speckle camera 1300.
[00149] Using or aggregating two or more sensors for object detection can
provide a
relatively more robust and redundant sensor system 400. For example, although
flash
LADARs generally have low dynamic range and rotating scanners generally have
long
inspection times, these types of sensor can be useful for object detection. In
some
implementations, the sensor system 400 include a flash LADAR and/or a rotating
scanner
in addition to the imaging sensor 450 (e.g., the 3D speckle camera 1300 and/or
the 3D
42
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
TOF camera 1500) in communication with the controller 500. The controller 500
may
use detection signals from the imaging sensor 450 and the flash ladar and/or a
rotating
scanner to identify objects 12, determine a distance of objects 12 from the
robot 100,
construct a 3D map of surfaces of objects 12, and/or construct or update an
occupancy
map 1700. The 3D speckle camera 1300 and/or the 3D TOF camera 1500 can be used
to
address any color or stereo camera weaknesses by initializing a distance
range, filling in
areas of low texture, detecting depth discontinuities, and/or anchoring scale.
1001501 In examples using the 3D speckle camera 1300, the speckle pattern
emitted by
the speckle emitter 1310 may be rotation-invariant with respect to the imager
1320.
Moreover, an additional camera 1300 (e.g., color or stereo camera) co-
registered with the
3D speckle camera 1300 and/or the 3D TOF camera 1500 may employ a feature
detector
that is some or fully scale-rotation-affine invariant to handle ego rotation,
tilt,
perspective, and/or scale (distance). Scale-invariant feature transform (or
SIFT) is an
algorithm for detecting and/or describing local features in images. SIFT can
be used by
the controller 500 (with data from the sensor system 400) for object
recognition, robotic
mapping and navigation, 3D modeling, gesture recognition, video tracking, and
match
moving. SIFT, as a scale-invariant, rotation-invariant transform, allows
placement of a
signature on features in the scene 10 and can help reacquire identified
features in the
scene 10 even if they are farther away or rotated. For example, the
application of SIFT
on ordinary images allows recognition of a moved object 12 (e.g., a face or a
button or
some text) be identifying that the object 12 has the same luminance or color
pattern, just
bigger or smaller or rotated. Other of transforms may be employed that are
affine-
invariant and can account for skew or distortion for identifying objects 12
from an angle.
The sensor system 400 and/or the controller 500 may provide scale-invariant
feature
recognition (e.g., with a color or stereo camera) by employing SIFT, RIFT,
Affine SIFT,
RIFT, G-RIF, SURF, PCA-SIFT, GLOH. PCA-SIFT, SIFT w/FAST corner detector
and/or Scalable Vocabulary Tree, and/or SIFT w/ Irregular Orientation
Histogram
Binning.
1001511 In some implementations, the controller 500 executes a program or
routine
that employs SIFT and/or other transforms for object detection and/or
identification. The
controller 500 may receive image data from an image sensor 450, such as a
color, black
43
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
and white, or IR camera. In some examples, the image sensor 450 is a 3D
speckle IR
camera that can provide image data without the speckle illumination to
identify features
without the benefit of speckle ranging. The controller 500 can identify or tag
features or
objects 12 previously mapped in the 3D scene from the speckle ranging. The
depth map
can be used to filter and improve the recognition rate of SIFT applied to
features imaged
with a camera, and/or simplify scale invariance (because both motion and
change in
range are known and can be related to scale). SIFT-like transforms may be
useful with
depth map data normalized and/or shifted for position variation from frame to
frame,
which robots with inertial tracking, odometry, proprioception, and/or beacon
reference
may be able to track. For example, a transform applied for scale and rotation
invariance
may still be effective to recognize a localized feature in the depth map if
the depth map is
indexed by the amount of movement in the direction of the feature.
[00152] Other details and features on SIFT-like or other feature
descriptors to 3D data,
which may combinable with those described herein, can be found in Se, S.;
Lowe, David
G.; Little, J. (2001). "Vision-based mobile robot localization and mapping
using scale-
invariant features". Proceedings of the IEEE International Conference on
Robotics and
Automation (ICRA). 2. pp. 2051; or Rothganger, F; S. Lazebnik, C. Schmid, and
J. Ponce:
2004. 3D Object Modeling and Recognition Using Local Affine-Invariant Image
Descriptors and Multi-View Spatial Constraints, ICCV; or Iryna Gordon and
David G.
Lowe, "What and where: 3D object recognition with accurate pose," Toward
Category-
Level Object Recognition, (Springer-Verlag, 2006), pp. 67-82; the contents of
which are
hereby incorporated by reference in their entireties.
[00153] Other details and features on techniques suitable for 3D SIFT in human
action
recognition, including falling, can be found in Laptev, Ivan and Lindeberg,
Tony (2004).
"Local descriptors for spatio-temporal recognition". ECCV'04 Workshop on
Spatial
Coherence for Visual Motion Analysis, Springer Lecture Notes in Computer
Science,
Volume 3667. pp. 91-103; Ivan Laptev, Barbara Caputo, Christian Schuldt and
Tony
Lindeberg (2007). "Local velocity-adapted motion events for spatio-temporal
recognition". Computer Vision and Image Understanding 108: 207-229; Scovanner,
Paul; Ali, S; Shah, M (2007). "A 3-dimensional sift descriptor and its
application to
action recognition". Proceedings of the 15th International Conference on
Multimedia. pp.
44
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
357-360; Niebles, J. C. Wang, H. and Li, Fei-Fei (2006). "Unsupervised
Learning of
Human Action Categories Using Spatial-Temporal Words". Proceedings of the
British
Machine Vision Conference (BMVC). Edinburgh; the contents of which are hereby
incorporated by reference in their entireties.
[00154] The controller 500 may use the imaging sensor 450 (e.g., a depth map
sensor)
when constructing a 3D map of the surface of and object 12 to fill in holes
from depth
discontinuities and to anchor a metric scale of a 3D model. Structure-from-
motion,
augmented with depth map sensor range data, may be used to estimate sensor
poses. A
typical structure-from-motion pipeline may include viewpoint-invariant feature
estimation, inter-camera feature matching, and a bundle adjustment.
[00155] A software solution combining features of color/stereo cameras with
the
imaging sensor 450 (e.g., the 3D speckle camera 1300, and/or the TOF camera
1500)
may include (1) sensor pose estimation, (2) depth map estimation, and (3) 3D
mesh
estimation. In sensor pose estimation, the position and attitude of the sensor
package of
each image capture is determined. In depth map estimation, a high-resolution
depth map
is obtained for each image. In 3D mesh estimation, sensor pose estimates and
depth
maps can be used to identify objects of interest.
[00156] In some implementations, a color or stereo camera 320 (FIG. 9) and the
3D
speckle 1300 or the 3D TOF camera 1500 may be co-registered. A stand-off
distance of
1 meter and 45-degree field of view 452 may give a reasonable circuit time and
overlap
between views. If at least two pixels are needed for 50-percent detection, at
least a 1
mega pixel resolution color camera may be used with a lens with a 45-degree
field of
view 452, with proportionately larger resolution for a 60 degree or wider
field of view
452.
[00157] Although a depth map sensor may have relatively low resolution and
range
accuracy, it can reliably assign collections of pixels from the color/stereo
image to a
correct surface. This allows reduction of stereo vision errors due to lack of
texture, and
also, by bounding range to, e.g., a 5 cm interval, can reduce the disparity
search range,
and computational cost.
[00158] Referring again to FIG. 10A, the first and second 3-D image sensors
450a,
450b can be used to improve mapping of the robot's environment to create a
robot map,
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
as the first 3-D image sensor 450a can be used to map out nearby objects and
the second
3-D image sensor 450b can be used to map out distant objects.
[00159] Referring to FIGS. 17A and 17B, in some circumstances, the robot 100
receives an occupancy map 1700 of objects 12 in a scene 10 and/or work area 5,
or the
robot controller 500 produces (and may update) the occupancy map 1700 based on
image
data and/or image depth data received from an imaging sensor 450 (e.g., the
second 3-D
image sensor 450b) over time. In addition to localization of the robot 100 in
the scene 10
(e.g., the environment about the robot 100), the robot 100 may travel to other
points in a
connected space (e.g., the work area 5) using the sensor system 400. The robot
100 may
include a short range type of imaging sensor 450a (e.g., mounted on the
underside of the
torso 140, as shown in FIGS. 1 and 3) for mapping a nearby area about the
robot 110 and
discerning relatively close objects 12, and a long range type of imaging
sensor 450b (e.g.,
mounted on the head 160, as shown in FIGS. 1 and 3) for mapping a relatively
larger area
about the robot 100 and discerning relatively far away objects 12. The robot
100 can use
the occupancy map 1700 to identify known objects 12 in the scene 10 as well as
occlusions 16 (e.g., where an object 12 should or should not be, but cannot be
confirmed
from the current vantage point). The robot 100 can register an occlusion 16 or
new
object 12 in the scene 10 and attempt to circumnavigate the occlusion 16 or
new object
12 to verify the location of new object 12 or any objects 12 in the occlusion
16.
Moreover, using the occupancy map 1700, the robot 100 can determine and track
movement of an object 12 in the scene 10. For example, the imaging sensor 450,
450a,
450b may detect a new position 12' of the object 12 in the scene 10 while not
detecting a
mapped position of the object 12 in the scene 10. The robot 100 can register
the position
of the old object 12 as an occlusion 16 and try to circumnavigate the
occlusion 16 to
verify the location of the object 12. The robot 100 may compare new image
depth data
with previous image depth data (e.g., the map 1700) and assign a confidence
level of the
location of the object 12 in the scene 10. The location confidence level of
objects 12
within the scene 10 can time out or degrade after a threshold period of time.
The sensor
system 400 can update location confidence levels of each object 12 after each
imaging
cycle of the sensor system 400. In some examples, a detected new occlusion 16
(e.g., a
missing object 12 from the occupancy map 1700) within an occlusion detection
period
46
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
(e.g., less than ten seconds), may signify a "live" object 12 (e.g., a moving
object 12) in
the scene 10.
[00160] In some implementations, a second object 12b of interest,
located behind a
detected first object 12a in the scene 10, may be initially undetected as an
occlusion 16 in
the scene 10. An occlusion 16 can be area in the scene 10 that is not readily
detectable or
viewable by the imaging sensor 450, 450a, 450b. In the example shown, the
sensor
system 400 (e.g., or a portion thereof, such as imaging sensor 450, 450a,
450b) of the
robot 100 has a field of view 452 with a viewing angle ev (which can be any
angle
between 0 degrees and 360 degrees) to view the scene 10. In some examples, the
imaging sensor 450 includes omni-directional optics for a 360 degree viewing
angle ev;
while in other examples, the imaging sensor 450, 450a, 450b has a viewing
angle ev of
less than 360 degrees (e.g., between about 45 degrees and 180 degrees). In
examples,
where the viewing angle Ov is less than 360 degrees, the imaging sensor 450,
450a, 450b
(or components thereof) may rotate with respect to the robot body 110 to
achieve a
viewing angle Ov of 360 degrees. The imaging sensor 450, 450a, 450b may have a
vertical viewing angle ev_v the same as or different from a horizontal viewing
angle OV-1-1.
For example, the imaging sensor 450, 450a, 450b may have a a horizontal field
of view
Ov_i_i of at least 45 degrees and a vertical field of view ev_v of at least 40
degrees. In some
implementations, the imaging sensor 450, 450a, 450b or portions thereof, can
move with
respect to the robot body 110 and/or drive system 200. Moreover, in order to
detect the
second object 12b, the robot 100 may move the imaging sensor 450, 450a, 450b
by
driving about the scene 10 in one or more directions (e.g., by translating
and/or rotating
on the work surface 5) to obtain a vantage point that allows detection of the
second object
10b. Robot movement or independent movement of the imaging sensor 450, 450a,
450b,
or portions thereof, may resolve monocular difficulties as well.
[00161] A confidence level may be assigned to detected locations or
tracked
movements of objects 12 in the working area 5. For example, upon producing or
updating the occupancy map 1700, the controller 500 may assign a confidence
level for
each object 12 on the map 1700. The confidence level can be directly
proportional to a
probability that the object 12 actually located in the working area 5 as
indicated on the
map 1700. The confidence level may be determined by a number of factors, such
as the
47
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
number and type of sensors used to detect the object 12. For example, the
contact sensor
430 may provide the highest level of confidence, as the contact sensor 430
senses actual
contact with the object 12 by the robot 100. The imaging sensor 450 may
provide a
different level of confidence, which may be higher than the proximity sensor
430. Data
received from more than one sensor of the sensor system 400 can be aggregated
or
accumulated for providing a relatively higher level of confidence over any
single sensor.
[00162] Odometry is the use of data from the movement of actuators to estimate
change in position over time (distance traveled). In some examples, an encoder
is
disposed on the drive system 200 for measuring wheel revolutions, therefore a
distance
traveled by the robot 100. The controller 500 may use odometry in assessing a
confidence level for an object location. In some implementations, the sensor
system 400
includes an odometer and/or an angular rate sensor (e.g., gyroscope or the IMU
470) for
sensing a distance traveled by the robot 100. A gyroscope is a device for
measuring or
maintaining orientation, based on the principles of conservation of angular
momentum.
The controller 500 may use odometry and/or gyro signals received from the
odometer
and/or angular rate sensor, respectively, to determine a location of the robot
100 in a
working area 5 and/or on an occupancy map 1700. In some examples, the
controller 500
uses dead reckoning. Dead reckoning is the process of estimating a current
position
based upon a previously determined position, and advancing that position based
upon
known or estimated speeds over elapsed time, and course. By knowing a robot
location
in the working area 5 (e.g., via odometry, gyroscope, etc.) as well as a
sensed location of
one or more objects 12 in the working area 5 (via the sensor system 400), the
controller
500 can assess a relatively higher confidence level of a location or movement
of an object
12 on the occupancy map 1700 and in the working area 5 (versus without the use
of
odometry or a gyroscope).
[00163] Odometry based on wheel motion can be electrically noisy. The
controller
500 may receive image data from the imaging sensor 450 of the environment or
scene 10
about the robot 100 for computing robot motion, independently of wheel based
odometry
of the drive system 200, through visual odometry. Visual odometry may entail
using
optical flow to determine the motion of the imaging sensor 450. The controller
500 can
use the calculated motion based on imaging data of the imaging sensor 450 for
correcting
48
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
any errors in the wheel based odometry, thus allowing for improved mapping and
motion
control. Visual odometry may have limitations with low-texture or low-light
scenes 10,
if the imaging sensor 450 cannot track features within the captured image(s).
[00164] Other details and features on odometry and imaging systems, which may
combinable with those described herein, can be found in U.S. Patent 7,158,317
(describing a "depth-of field" imaging system), and U.S. Patent 7,115,849
(describing
wavefront coding interference contrast imaging systems), the contents of which
are
hereby incorporated by reference in their entireties.
[00165] Referring to FIG. 18, in some implementations, the imaging sensor 450
has an
imaging dead zone 453, which is a volume of space about the imaging sensor 450
in
which objects are not detected. In some examples, the imaging dead zone 453
includes
volume of space defined by a first angle a by a second angle 13 and by a
radius Rs of
about 570 x 450 x 50 cm, respectively, immediately proximate the imaging
sensor 450 and
centered about an imaging axis 455. The dead zone 453 is positioned between
the
imaging sensor 450 and a detection field 457 of the imaging sensor 450 within
the field
of view 452.
[00166] In the example shown in FIG. 19, the robot 100 includes a first
and second
imaging sensors 450a, 450b (e.g., 3D depth imaging sensors) disposed on the
torso 140.
Both imaging sensors 450a, 450b are arranged to have field of view 452 along
the
forward drive direction F. The first imaging sensor 450a is arranged to aim
its imaging
axis 455 substantially downward and away from the robot 100 (e.g., to view an
area on
the ground and/or about a lower portion of the robot) to detect objects before
contact with
the base 120 or leg 130. By angling the first imaging sensor 450a downward,
the robot
100 receives dense sensor coverage in an area immediately forward or adjacent
to the
robot 100, which is relevant for short-term travel of the robot 100 in the
forward
direction. The second imaging sensor 450b is arranged with its imaging axis
455
pointing substantially parallel with the ground along the forward drive
direction F (e.g.,
to detect objects approaching a mid and/or upper portion of the robot 100). In
other
examples, the second imaging sensor 450b is arranged with its imaging axis 455
pointing
above the ground or even upward away from the ground.
49
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
1001671 If the imaging sensors 450a, 450b have dead zones 453, there is
a possibility
of failing to detect an object proximate or adjacent the robot 100. In the
example shown
in FIG. 10A, the robot 100 includes an imaging sensor 450 mounted on the head
160,
which can pan and tilt via the neck 150. As a result, the robot 100 can move
the imaging
sensor 450 on the head to view the dead zones 453 of the other imaging sensors
450a,
450b, thus providing complete or substantially complete fields of view 452
about the
robot 100 for object detection. When placement of an imaging sensor 450 on the
head
160 is not possible or if an imaging sensor 450 cannot be moved to view the
dead zones
453, other techniques may be employed to view the dead zones 453. In addition
to dead
zones 453, some objects within the field of view 452 of the imaging sensor 450
can be
difficult to detect, due to size, shape, reflectivity, and/or color. For
example, sometimes
highly reflective or specular objects can be difficult to detect. In other
examples, very
dark or black objects can be difficult to detect. Moreover, slender objects
(i.e., having a
very thin profile) may be difficult to detect. Hard to detect objects may be
become
relatively more detectable when viewed from multiple angles or sensed from
multiple
sensors.
1001681 In the example shown in FIGS. 1, 4C and 10A, the robot includes one or
more
sonar proximity sensors 410 (e.g., 410a-410i) disposed around the base body
120 are
arranged to point upward (e.g., substantially in the Z direction) and
optionally angled
outward away from the Z axis, thus creating a detection curtain 412 around the
robot 100.
The sonar proximity sensors 410 can be arranged and aimed to sense objects
within the
dead zone 453 of each imaging sensor 450.
1001691 In some implementations, the robot 100 (via the controller 500 or the
sensor
system 400) moves or pans the imaging sensors 450, 450a, 450b to gain view-
ability of
the corresponding dead zones 453. An imaging sensor 450 can be pointed in any
direction 360 (+/- 180 ) by moving its associated imaging axis 455.
1001701 In some examples, the robot 100 maneuvers itself on the ground to move
the
imaging axis 455 and corresponding field of view 452 of each imaging sensor
450 to gain
perception of the volume of space once in a dead zone 453. For example, the
robot 100
may pivot in place, holonomically move laterally, move forward or backward, or
a
combination thereof. In additional examples, if the imaging sensor 450 has a
limited
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No. 225899-318442
field of view 452 and/or detection field 457, the controller 500 or the sensor
system 400
can actuate the imaging sensor 450 in a side-to-side and/or up and down
scanning manner
to create a relatively wider and/or taller field of view to perfot ______ in
robust ODOA. Panning
the imaging sensor 450 (by moving the imaging axis 455) increases an
associated
horizontal and/or vertical field of view, which may allow the imaging sensor
450 to view
not only all or a portion of its dead zone 453, but the dead zone 453 of
another imaging
sensor 450 on the robot 100.
[00171] In some examples, each imaging sensor 450 may have an associated
actuator
(not shown) moving the imaging sensor 450 in the scanning motion. In
additional
examples, the imaging sensor 450 includes an associated rotating a mirror,
prism,
variable angle micro-mirror, or MEMS mirror array to increase the field of
view 452
and/or detection field 457 of the imaging sensor 450.
[00172] In the example shown in FIG. 20, the torso 140 pivots about the Z-axis
on the
leg 130, allowing the robot 100 to move an imaging sensor 450 disposed on the
torso 140
with respect to the forward drive direction F defined by the base 120. In some
examples,
the leg 130 pivots about the Z-axis, thus moving the torso 140 about the Z-
axis. In either
example, an actuator 138 (such as a rotary actuator) in communication wit the
controller
500 rotates the torso 140 with respect to the base 120 (e.g., by either
rotating the torso
140 with respect to the leg 130 and/or rotating the leg 130 with respect to
the base 120).
The rotating torso 140 moves the imaging sensor 450 in a panning motion about
the Z-
axis providing up to a 360 field of view 452 about the robot 100. The robot
100 may
pivot the torso 140 in a continuous 360 or +/- an angle 180 with respect to
the
forward drive direction F.
[00173] Referring to FIG. 21, in some implementations, the robot 100 includes
a dead
zone sensor 490 associated with each imaging sensor 450 and arranged to sense
objects
within the dead zone 453 of the associated imaging sensor 450. The dead zone
sensor
490 may be a sonar sensor, camera, ultrasonic sensor, LIDAR, LADAR, optical
sensor,
infrared sensor, etc. In the example shown, the dead zone sensor 490 is
arranged to have
field of view 492 enveloping or substantially enveloping the dead zone 453.
FIG. 22
provides a top of view of a robot 100 having a dead zone sensor 490 disposed
on the
torso 140 adjacent the imaging sensor 450 and arranged to have its field view
492
51
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
extending into the dead zone 453. In the example shown the dead zone field of
view 492
is substantially centered within the dead zone 453; however, other
arrangements are
possible as well (e.g., off-center).
[00174] FIG. 23 illustrates an exemplary robot 100 having an array of dead
zone
sensors 490 disposed on a forward portion 147 of the torso 140. The array of
dead zone
sensors 490 not only provide coverage of the dead zone 453 shown, but also
additional
areas about the robot 100 not previously within the field of view of a sensor
(e.g., the
areas on each side of the field of view 452 of the imaging sensor 450). This
allows the
robot 100 to sense nearby objects before moving or turning into them.
[00175] In the example shown in FIG. 24, the robot 100 includes at least
one long
range sensor 2190 arranged and configured to detect an object 12 relatively
far away
from the robot 100 (e.g., > 3 meters). The long range sensor 2190 may be an
imaging
sensor 450 (e.g., having optics or a zoom lens configured for relatively long
range
detection). In additional examples, the long range sensor 2190 is a camera
(e.g., with a
zoom lens), a laser range finder, LIDAR, RADAR, etc. In the example shown, the
robot
100 includes four long range sensors 2190 arranged with corresponding fields
of view
2192 along forward, aft, right, and left drive directions. Other arrangements
are possible
as well.
[00176] Detection of far off objects allows the robot 100 (via the
controller 500) to
execute navigational routines to avoid the object, if viewed as an obstacle,
or approach
the object, if viewed as a destination (e.g., for approaching a person for
executing a video
conferencing session). Awareness of objects outside of the field of view of
the imaging
sensor(s) 450 on the robot 100, allows the controller 500 to avoid movements
that may
place the detected object 12 in a dead zone 453. Moreover, in person following
routines,
when a person moves out of the field of view of an imaging sensor 450, the
long range
sensor 2190 may detect the person and allow the robot 100 to maneuver to
regain
perception of the person in the field of view 452 of the imaging sensor 450.
[00177] Referring to FIG. 25, in some implementations, the controller
500 executes a
control system 510, which includes a control arbitration system 510a and a
behavior
system 510b in communication with each other. The control arbitration system
510a
allows applications 520 to be dynamically added and removed from the control
system
52
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
510, and facilitates allowing applications 520 to each control the robot 100
without
needing to know about any other applications 520. In other words, the control
arbitration
system 510a provides a simple prioritized control mechanism between
applications 520
and resources 530 of the robot 100. The resources 530 may include the drive
system 200,
the sensor system 400, and/or any payloads or controllable devices in
communication
with the controller 500.
[00178] The applications 520 can be stored in memory of or communicated to the
robot 100, to run concurrently on (e.g., a processor) and simultaneously
control the robot
100. The applications 520 may access behaviors 600 of the behavior system
510b. The
io independently deployed applications 520 are combined dynamically at
runtime and to
share robot resources 530 (e.g., drive system 200, arm(s), head(s), etc.) of
the robot 100.
A low-level policy is implemented for dynamically sharing the robot resources
530
among the applications 520 at run-time. The policy determines which
application 520
has control of the robot resources 530 required by that application 520 (e.g.
a priority
hierarchy among the applications 520). Applications 520 can start and stop
dynamically
and run completely independently of each other. The control system 510 also
allows for
complex behaviors 600 which can be combined together to assist each other.
[00179] The control arbitration system 510a includes one or more resource
controllers
540, a robot manager 550, and one or more control arbiters 560. These
components do
not need to be in a common process or computer, and do not need to be started
in any
particular order. The resource controller 540 component provides an interface
to the
control arbitration system 510a for applications 520. There is an instance of
this
component for every application 520. The resource controller 540 abstracts and
encapsulates away the complexities of authentication, distributed resource
control
arbiters, command buffering, and the like. The robot manager 550 coordinates
the
prioritization of applications 520, by controlling which application 520 has
exclusive
control of any of the robot resources 530 at any particular time. Since this
is the central
coordinator of information, there is only one instance of the robot manager
550 per robot.
The robot manager 550 implements a priority policy, which has a linear
prioritized order
of the resource controllers 540, and keeps track of the resource control
arbiters 560 that
provide hardware control. The control arbiter 560 receives the commands from
every
53
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
application 520 and generates a single command based on the applications'
priorities and
publishes it for its associated resources 530. The control arbiter 560 also
receives state
feedback from its associated resources 530 and sends it back up to the
applications 520.
The robot resources 530 may be a network of functional modules (e.g.
actuators, drive
systems, and groups thereof) with one or more hardware controllers. The
commands of
the control arbiter 560 are specific to the resource 530 to carry out specific
actions.
[00180] A dynamics model 570 executable on the controller 500 can be
configured to
compute the center for gravity (CG), moments of inertia, and cross products of
inertia of
various portions of the robot 100 for the assessing a current robot state. The
dynamics
model 570 may also model the shapes, weight, and/or moments of inertia of
these
components. In some examples, the dynamics model 570 communicates with an
inertial
moment unit 470 (IMU) or portions of one (e.g., accelerometers and/or gyros)
disposed
on the robot 100 and in communication with the controller 500 for calculating
the various
center of gravities of the robot 100. The dynamics model 570 can be used by
the
controller 500, along with other programs 520 or behaviors 600 to determine
operating
envelopes of the robot 100 and its components.
[00181] Each application 520 has an action selection engine 580 and a
resource
controller 540, one or more behaviors 600 connected to the action selection
engine 580,
and one or more action models 590 connected to action selection engine 580.
The
behavior system 510b provides predictive modeling and allows the behaviors 600
to
collaboratively decide on the robot's actions by evaluating possible outcomes
of robot
actions. In some examples, a behavior 600 is a plug-in component that provides
a
hierarchical, state-full evaluation function that couples sensory feedback
from multiple
sources with a-priori limits and information into evaluation feedback on the
allowable
actions of the robot. Since the behaviors 600 are pluggable into the
application 520 (e.g.,
residing inside or outside of the application 520), they can be removed and
added without
having to modify the application 520 or any other part of the control system
510. Each
behavior 600 is a standalone policy. To make behaviors 600 more powerful, it
is possible
to attach the output of multiple behaviors 600 together into the input of
another so that
you can have complex combination functions. The behaviors 600 are intended to
implement manageable portions of the total cognizance of the robot 100.
54
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00182] The action selection engine 580 is the coordinating element of
the control
system 510 and runs a fast, optimized action selection cycle
(prediction/correction cycle)
searching for the best action given the inputs of all the behaviors 600. The
action
selection engine 580 has three phases: nomination, action selection search,
and
completion. In the nomination phase, each behavior 600 is notified that the
action
selection cycle has started and is provided with the cycle start time, the
current state, and
limits of the robot actuator space. Based on internal policy or external
input, each
behavior 600 decides whether or not it wants to participate in this action
selection cycle.
During this phase, a list of active behavior primitives is generated whose
input will affect
the selection of the commands to be executed on the robot 100.
[00183] In the action selection search phase, the action selection
engine 580 generates
feasible outcomes from the space of available actions, also referred to as the
action space.
The action selection engine 580 uses the action models 590 to provide a pool
of feasible
commands (within limits) and corresponding outcomes as a result of simulating
the
action of each command at different time steps with a time horizon in the
future. The
action selection engine 580 calculates a preferred outcome, based on the
outcome
evaluations of the behaviors 600, and sends the corresponding command to the
control
arbitration system 510a and notifies the action model 590 of the chosen
command as
feedback.
[00184] In the completion phase, the commands that correspond to a
collaborative best
scored outcome are combined together as an overall command, which is presented
to the
resource controller 540 for execution on the robot resources 530. The best
outcome is
provided as feedback to the active behaviors 600, to be used in future
evaluation cycles.
[00185] Received sensor signals from the sensor system 400 can cause
interactions
with one or more behaviors 600 to execute actions. For example, using the
control
system 510, the controller 500 selects an action (or move command) for each
robotic
component (e.g., motor or actuator) from a corresponding action space (e.g., a
collection
of possible actions or moves for that particular component) to effectuate a
coordinated
move of each robotic component in an efficient manner that avoids collisions
with itself
and any objects about the robot 100, which the robot 100 is aware of. The
controller 500
can issue a coordinated command over robot network, such as the EtherIO
network.
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
[00186] The control system 510 may provide adaptive speed/acceleration of the
drive
system 200 (e.g., via one or more behaviors 600) in order to maximize
stability of the
robot 100 in different configurations/positions as the robot 100 maneuvers
about an area.
[00187] In some implementations, the controller 500 issues commands to the
drive
system 200 that propels the robot 100 according to a heading setting and a
speed setting.
One or behaviors 600 may use signals received from the sensor system 400 to
evaluate
predicted outcomes of feasible commands, one of which may be elected for
execution
(alone or in combination with other commands as an overall robot command) to
deal with
obstacles. For example, signals from the proximity sensors 410 may cause the
control
system 510 to change the commanded speed or heading of the robot 100. For
instance, a
signal from a proximity sensor 410 due to a nearby wall may result in the
control system
510 issuing a command to slow down. In another instance, a collision signal
from the
contact sensor(s) due to an encounter with a chair may cause the control
system 510 to
issue a command to change heading. In other instances, the speed setting of
the robot
100 may not be reduced in response to the contact sensor; ancUor the heading
setting of
the robot 100 may not be altered in response to the proximity sensor 410.
[00188] The behavior system 510b may include a mapping behavior 600a for
producing an occupancy map 1700, an object detection obstacle avoidance (ODOA)
behavior 600b, a speed behavior 600c (e.g., a behavioral routine executable on
a
processor) configured to adjust the speed setting of the robot 100 and a
heading behavior
600d configured to alter the heading setting of the robot 100. The speed and
heading
behaviors 600c, 600d may be configured to execute concurrently and mutually
independently. For example, the speed behavior 600c may be configured to poll
one of
the sensors (e.g., the set(s) of proximity sensors 410, 420), and the heading
behavior 600d
may be configured to poll another sensor (e.g., the kinetic bump sensor).
[00189] Referring to FIGS. 25 and 26A-26D, in some implementations, to
navigate to
a destination location, the robot 100 may rely on its ability to discern its
local perceptual
space 2100 (i.e., the space around the robot 100 as perceived through the
sensor system
400) and execute an object detection obstacle avoidance (ODOA) strategy. The
sensor
system 400 may provide sensor data including three-dimensional depth image
data
provided by a volumetric point cloud imaging device 450 positioned on the
robot 100 to
56
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
be capable of obtaining a point cloud from a volume of space adjacent the
robot 100. For
example, the volumetric point cloud imaging device 450 may be positioned on
the robot
100 at a height of greater than 2 feet above the ground and directed to be
capable of
obtaining a point cloud from a volume of space that includes a floor plane G
in a
direction of movement of the robot 100.
[00190] The robot 100 (e.g., the control system 510 shown in FIG. 25)
may classify its
local perceptual space 2100 into three categories: obstacles (black) 2102,
unknown (gray)
2104, and known free (white) 2106. Obstacles 2102 are observed (i.e., sensed)
points
above the ground G that are below a height of the robot 100 and observed
points below
the ground G (e.g., holes, steps down, etc.). Known free 2106 corresponds to
areas where
the 3-D image sensors 450 can see the ground G. Data from some or all sensors
in the
sensor system 400 can be combined into a discretized 3-D voxel grid. The 3-D
grid can
then be analyzed and converted into a 2-D grid 2101 with the three local
perceptual space
classifications. FIG. 26A provides an exemplary schematic view of the local
perceptual
space 2100 of the robot 100 while stationary. The information in the 3-D voxel
grid has
persistence, but decays over time if it is not reinforced (e.g., by fresh
sensor data). When
the robot 100 is moving, it has more known free area 2106 to navigate in
because of
persistence. The volumetric point cloud data of the 3-D imaging sensor 450 may
time-
out after a threshold period of time, such as milliseconds to seconds, so that
transient or
slightly older objects in the environment (e.g., people walking, sensor
artifacts, etc.) are
not used for local path planning.
[00191] When the 3-D imaging sensor 450 has a dead zone 453 (FIGS. 20-24) and
the
robot 100 maneuvers itself immediately next to a non-transient object 12
entirely within
the dead zone 453 (FIG. 26A), the control system 510 may allow the sensor data
associated with that non-transient object 12 to time-out and hence no longer
recognize the
object 12 as an obstacle 2102 for navigation purposes. Although other sensors
of the
sensor system 400, such as the dead zone sensor(s) 490, may be able to detect
the object
12, the control system 510 may execute an ODOA strategy that suspends the data
time-
out for sensor data associated with that object 12 in the dead zone 453.
[00192] For example, the control system 510 may suspend the data time-out for
sensor
data, which normally times out after a threshold period of time and is
associated with
57
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
obstacles 2102 (e.g., objects 12) in the local perceptual 2100, when the
obstacle 2102 is
perceived as residing in the dead zone 453 or an area immediately adjacent the
robot 100.
The control system 510 may determine the presence of an object 12
corresponding to the
obstacle 2102 in the dead zone 453, using one or more dead zone sensors 490 or
other
sensor(s) of the sensor system 400 as near sensors. The control system 510 may
allow
the sensor data associated with that object 12 to decay or time-out again only
after the
robot 100 has moved away from that location and/or the dead zone sensor(s) 490
detect
that the object 12 has moved out of the dead zone 453. This allows the control
system
510 to execute object detection obstacle avoidance (ODOA) navigation
strategies that
consider the possibility of an obstacle in the dead zone 453 of the robot 100.
1001931 An object detection obstacle avoidance (ODOA) navigation
strategy for the
control system 510 may include either accepting or rejecting potential robot
positions that
would result from commands. Potential robot paths 2110 can be generated many
levels
deep with different commands and resulting robot positions at each level. FIG.
26B
provides an exemplary schematic view of the local perceptual space 2100 of the
robot
100 while moving. An ODOA behavior 600b (FIG. 25) can evaluate each predicted
robot path 2110. These evaluations can be used by the action selection engine
580 to
detetinine a preferred outcome and a corresponding robot command. For example,
for
each robot position 2120 in the robot path 2110, the ODOA behavior 600b can
execute a
method for object detection and obstacle avoidance that includes identifying
each cell
2103 in the grid 2101 that is in a bounding shape 2107 (e.g., collision box,
triangle, or
circle) around a corresponding position 2120 of the robot 100, receiving a
classification
of each cell 2103. For each cell 2103 classified as an obstacle 2102 or
unknown 2104,
retrieving a grid point 2105 corresponding to the cell 2103 and executing a
collision
check by determining if the grid point 2105 is within a bounding shape 2107
(e.g.,
collision circle) about a location 2120 of the robot 100. If the grid point
2105 is within
the bounding shape 2107, the method further includes executing a triangle test
of whether
the grid point 2105 is within a bounding shape 2107 shaped as a triangle
(e.g., the robot
100 can be modeled as triangle). If the grid point 2105 is within the
collision triangle
2107, the method includes rejecting the grid point 2105. If the robot position
2120 is
inside of a sensor system field of view 405 of parent grid points 2105 on the
robot path
58
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
2110, then the "unknown" grid points 2105 are ignored because it is assumed
that by the
time the robot 100 reaches those grid points 2105, they will be known.
[00194] The method may include determining whether any obstacle collisions are
present within a robot path area (e.g., as modeled by a rectangle) between
successive
robot positions 2120 in the robot path 2110, to prevent robot collisions
during the
transition from one robot position 2120 to the next.
[00195] FIG. 26C provides a schematic view of the local perceptual space 2100
of the
robot 100 and a sensor system field of view 405 (the control system 510 may
use only
certain sensor, such as the first and second 3-D image sensors 450a, 450b, for
robot path
determination). Taking advantage of the holonomic mobility of the drive system
200, the
robot 100 can use the persistence of the known ground G to allow it to drive
in directions
where the sensor system field of view 405 does not actively cover. For
example, if the
robot 100 has been sitting still with the first and second 3-D image sensors
450a, 450b
pointing forward, although the robot 100 is capable of driving sideways, the
control
system 510 will reject the proposed move, because the robot 100 does not know
what is
to its side, as illustrated in the example shown in FIG. 26C, which shows an
unknown
classified area to the side of the robot 100. If the robot 100 is driving
forward with the
first and second 3-D image sensors 450a, 450b pointing forward, then the
ground G next
to the robot 100 may be classified as known free 2106, because both the first
and second
3-D image sensors 450a, 450b can view the ground G as free as the robot 100
drives
forward and persistence of the classification has not decayed yet. (See e.g.,
FIG. 26B.) In
such situations the robot 100 can drive sideways.
[00196] Referring to FIG. 26D, in some examples, given a large number of
possible
trajectories with holonomic mobility, the ODOA behavior 600b may cause robot
to
choose trajectories where it will (although not currently) see where it is
going. For
example, the robot 100 can anticipate the sensor field of view orientations
that will allow
the control system 510 to detect objects. Since the robot can rotate while
translating, the
robot can increase the sensor field of view 405 while driving.
[00197] Various implementations of the systems and techniques described here
can be
realized in digital electronic circuitry, integrated circuitry, specially
designed ASICs
(application specific integrated circuits), computer hardware, firmware,
software, and/or
59
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
combinations thereof. These various implementations can include implementation
in one
or more computer programs that are executable and/or interpretable on a
programmable
system including at least one programmable processor, which may be special or
general
purpose, coupled to receive data and instructions from, and to transmit data
and
instructions to, a storage system, at least one input device, and at least one
output device.
[00198] These computer programs (also known as programs, software, software
applications or code) include machine instructions for a programmable
processor, and can
be implemented in a high-level procedural and/or object-oriented programming
language,
and/or in assembly/machine language. As used herein, the terms "machine-
readable
medium" and "computer-readable medium" refer to any computer program product,
apparatus and/or device (e.g., magnetic discs, optical disks, memory,
Programmable
Logic Devices (PLDs)) used to provide machine instructions and/or data to a
programmable processor, including a machine-readable medium that receives
machine
instructions as a machine-readable signal. The term "machine-readable signal"
refers to
any signal used to provide machine instructions and/or data to a programmable
processor.
[00199] Implementations of the subject matter and the functional operations
described
in this specification can be implemented in digital electronic circuitry, or
in computer
software, firmware, or hardware, including the structures disclosed in this
specification
and their structural equivalents, or in combinations of one or more of them.
Embodiments of the subject matter described in this specification can be
implemented as
one or more computer program products, i.e., one or more modules of computer
program
instructions encoded on a computer readable medium for execution by, or to
control the
operation of, data processing apparatus. The computer readable medium can be a
machine-readable storage device, a machine-readable storage substrate, a
memory device,
a composition of matter effecting a machine-readable propagated signal, or a
combination
of one or more of them. The term "data processing apparatus" encompasses all
apparatus, devices, and machines for processing data, including by way of
example a
programmable processor, a computer, or multiple processors or computers. The
apparatus can include, in addition to hardware, code that creates an execution
environment for the computer program in question, e.g., code that constitutes
processor
firmware, a protocol stack, a database management system, an operating system,
or a
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
combination of one or more of them. A propagated signal is an artificially
generated
signal, e.g., a machine-generated electrical, optical, or electromagnetic
signal, that is
generated to encode information for transmission to suitable receiver
apparatus.
[00200] A computer program (also known as a program, software, software
application, script, or code) can be written in any form of programming
language,
including compiled or interpreted languages, and it can be deployed in any
form,
including as a stand alone program or as a module, component, subroutine, or
other unit
suitable for use in a computing environment. A computer program does not
necessarily
correspond to a file in a file system. A program can be stored in a portion of
a file that
holds other programs or data (e.g., one or more scripts stored in a markup
language
document), in a single file dedicated to the program in question, or in
multiple
coordinated files (e.g., files that store one or more modules, sub programs,
or portions of
code). A computer program can be deployed to be executed on one computer or on
multiple computers that are located at one site or distributed across multiple
sites and
interconnected by a communication network.
[00201] The processes and logic flows described in this specification
can be perfoi ined
by one or more programmable processors executing one or more computer programs
to
perform functions by operating on input data and generating output. The
processes and
logic flows can also be performed by, and apparatus can also be implemented
as, special
purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an
ASIC
(application specific integrated circuit).
[00202] Processors suitable for the execution of a computer program include,
by way
of example, both general and special purpose microprocessors, and any one or
more
processors of any kind of digital computer. Generally, a processor will
receive
instructions and data from a read only memory or a random access memory or
both. The
essential elements of a computer are a processor for performing instructions
and one or
more memory devices for storing instructions and data. Generally, a computer
will also
include, or be operatively coupled to receive data from or transfer data to,
or both, one or
more mass storage devices for storing data, e.g., magnetic, magneto optical
disks, or
optical disks. However, a computer need not have such devices. Moreover, a
computer
can be embedded in another device, e.g., a mobile telephone, a personal
digital assistant
61
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
(PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to
name just
a few. Computer readable media suitable for storing computer program
instructions and
data include all forms of non volatile memory, media and memory devices,
including by
way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash
memory devices; magnetic disks, e.g., internal hard disks or removable disks;
magneto
optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can
be supplemented by, or incorporated in, special purpose logic circuitry.
[00203] Implementations of the subject matter described in this specification
can be
implemented in a computing system that includes a back end component, e.g., as
a data
server, or that includes a middleware component, e.g., an application server,
or that
includes a front end component, e.g., a client computer having a graphical
user interface
or a web browser through which a user can interact with an implementation of
the subject
matter described is this specification, or any combination of one or more such
back end,
middleware, or front end components. The components of the system can be
interconnected by any form or medium of digital data communication, e.g., a
communication network. Examples of communication networks include a local area
network ("LAN") and a wide area network ("WAN"), e.g., the Internet.
[00204] The computing system can include clients and servers. A client and
server are
generally remote from each other and typically interact through a
communication
network. The relationship of client and server arises by virtue of computer
programs
running on the respective computers and having a client-server relationship to
each other.
[00205] While this specification contains many specifics, these should
not be
construed as limitations on the scope of the invention or of what may be
claimed, but
rather as descriptions of features specific to particular implementations of
the invention.
Certain features that are described in this specification in the context of
separate
implementations can also be implemented in combination in a single
implementation.
Conversely, various features that are described in the context of a single
implementation
can also be implemented in multiple implementations separately or in any
suitable sub-
combination. Moreover, although features may be described above as acting in
certain
combinations and even initially claimed as such, one or more features from a
claimed
62
CA 02824606 2013-06-26
WO 2012/091807
PCT/US2011/059980
Attorney Docket No: 225899-318442
combination can in some cases be excised from the combination, and the claimed
combination may be directed to a sub-combination or variation of a sub-
combination.
[00206] Similarly, while operations are depicted in the drawings in a
particular order,
this should not be understood as requiring that such operations be performed
in the
particular order shown or in sequential order, or that all illustrated
operations be
performed, to achieve desirable results. In certain circumstances, multi-
tasking and
parallel processing may be advantageous. Moreover, the separation of various
system
components in the embodiments described above should not be understood as
requiring
such separation in all embodiments, and it should be understood that the
described
program components and systems can generally be integrated together in a
single
software product or packaged into multiple software products.
[00207] A number of implementations have been described. Nevertheless, it will
be
understood that various modifications may be made without departing from the
spirit and
scope of the disclosure. Accordingly, other implementations are within the
scope of the
following claims. For example, the actions recited in the claims can be
performed in a
different order and still achieve desirable results.
63
