Note: Descriptions are shown in the official language in which they were submitted.
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
Mobile Human Interface Robot
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This U.S. patent application claims priority under 35 U.S.C. 119(e) to
U.S.
Provisional Application 61/346,612, filed on May 20, 2010; U.S. Provisional
Application
61/356,910, filed on June 21, 2010; U.S. Provisional Application 61/428,717,
filed on
December 30, 2010; U.S. Provisional Application 61/428,734, filed on December
30,
2010; U.S. Provisional Application 61/428,759, filed on December 30, 2010;
U.S.
Provisional Application 61/429,863, filed on January 5, 2011, and U.S.
Provisional
Application 61/445,473, filed on February 22, 2011. The disclosures of these
prior
applications are considered part of the disclosure of this application and are
hereby
incorporated by reference in their entireties.
TECHNICAL FIELD
[0002] This disclosure relates to mobile human interface robots.
BACKGROUND
[0003] A robot is generally an electro-mechanical machine guided by a computer
or
electronic programming. Mobile robots have the capability to move around in
their
environment and are not fixed to one physical location. An example of a mobile
robot
that is in common use today is an automated guided vehicle or automatic guided
vehicle
(AGV). An AGV is generally a mobile robot that follows markers or wires in the
floor,
or uses a vision system or lasers for navigation. Mobile robots can be found
in industry,
military and security environments. They also appear as consumer products, for
entertainment or to perform certain tasks like vacuum cleaning and home
assistance.
SUMMARY
[0004] One aspect of the disclosure provides a human interface robot that
includes a
controller, a camera in communication with the controller, and a display in
communication with the controller. The controller displays received image data
on the
1
CA 02800372 2012-11-26
display as an image, identifies at least one shape in the image, and displays
a shape specific label
on the image at least near the shape, the shape specific label identifying the
at least one shape.
[0005] Implementations of the disclosure may include one or more of the
following features. In
some implementations, the controller includes a segmentor, a size filter, and
a shape filter. The
segmentor receives three-dimensional image data from the camera and segments
the image data
into objects. The size filter processes the objects into right-sized objects
by rejecting objects
having a size less than a first threshold size or greater than a second
threshold size, the first and
second threshold sizes conforming to a target object size. The shape filter
rendersright-sized
objects that conform to a target shape into person data. The size filter may
reject objects having a
height greater than 8 feet or less than 3 feet. The label may include at least
one of identification
information, a hyper text markup language link, an email address, a web page
address, and a text
entry field.
[0006] In some implementations, the controller receives voice signals from a
user and alters the
voice signals. For example, the controller may alter a volume level of the
voice signals and/or
identify a first language corresponding to the voice signals and produce
translated signals
corresponding to a second language. In some examples, the controller processes
the voice signals
and transcribes dictation corresponding to the voice signals.
[0007] The controller may communicate with a cloud computing service. In some
examples, the
controller communicates with a cloud computing and a remote computing device
in
communication with the cloud computing service. The remote computing device
communicates
with the robot through the cloud computing service. The remote computing
device may execute
an application providing remote teleoperation of the robot. The application
may provide controls
for at least one of driving the robot, altering a pose of the robot, viewing
video from a camera of
the robot, and operating a camera of the robot. In some examples, the remote
computing device
executes an application providing video conferencing between a user of the
computing device
and a third party within view of a camera of the robot.
[0008] In some implementations, the mobile human interface robot includes a
mediating security
device controlling communications between the controller and the computing
device. The
mediating security device converts communications between a computing device
communication
protocol of the computing device and a robot communication protocol of the
robot. The
2
CA 02800372 2012-11-26
mediating security device may include an authorization chip for authorizing
communication
traffic between the computing device in the robot.
[0009] The display may be a tablet computer detachably supported by the robot
and in wircless
communication with the controller. The tablet computer may include a touch
screen having a
display area of at least 150 square inches. Moreover, the tablet computer can
be movable with at
least one degree of freedom while attached to the robot. In some
implementations, the robot
includes a monitor in electric communication with the controller. The tablet
computer is
detachably receivable over the monitor. The monitor has an inactive state when
the tablet
computer is received over the monitor and an active state when the tablet
computer is detached
from the monitor.
[0010] In some examples, the camera is movable within at least one degree of
freedom
separately from the display. The camera may comprise a volumetric point cloud
imaging device
positioned to be capable of obtaining a point cloud from a volume of space
adjacent the robot.
For example, the camera may be a volumetric point cloud imaging device
positioned at a height
of greater than about one foot above a ground surface and directed to be
capable of obtaining a
point cloud from a volume of space that includes a floor plane in a direction
of movement of the
robot.
[0011] The robot may include a holonomic drive system in communication with
the controller.
The holonomic drive system has first, second, and third driven drive wheels,
each drive wheel
trilaterally spaced about a vertical center axis of the base and each having a
drive direction
perpendicular to a radial axis with respect to the vertical center axis. The
robot may include a
base defining a vertical center axis and supporting the controller, an
extendable leg extending
upward from the base, and a torso supported by the leg. Actuation of the leg
causes a change in
elevation of the torso. The display is supported above the torso. The robot
may also include a
neck supported by the torso and a head supported by the neck. The neck is
capable of panning
and tilting the head with respect to the torso. The head detachably supports
the display.
[0012] Another aspect of the disclosure provides a method of operating a
robot. The method
includes receiving image data from a camera mounted on the robot corresponding
to an image,
identifying at least one shape in the image, and displaying on a display of
the robot a shape
specific label on the image at least near the shape, the shape specific label
identifying the at least
3
CA 02800372 2012-11-26
one shape.
[0013] In some implementations, the method includes segmenting the image data
into objects,
filtering the objects into right-sized objects by rejecting objects having a
size less than a first
threshold size or greater than a second threshold size, the first and second
threshold sized
conforming to a target object size, and rendering right-sized objects that
conform to a target
shape into person data. The filtering may include rejecting objects having a
height greater than 8
feet or less than 3 feet. The method may include identifying multiple people
corresponding to the
filtered objects.
[0014] In some examples, the method includes receiving the three-dimensional
image data from
a volumetric point cloud imaging device positioned at a height of greater than
1 or 2 feet above a
ground surface and directed to be capable of obtaining a point cloud from a
volume of space that
includes a floor plane in a direction of movement of the robot. The method may
include
receiving the three-dimensional image data from a volumetric point cloud
imaging device
positioned to be capable of obtaining a point cloud from a volume of space
adjacent the robot.
[0015] In some examples the label includes at least one of identification
information, a hyper text
markup language link, an email address, a web page address, and a text entry
field.
[0016] The method may include receiving voice signals from a user and altering
the voice
signals. For example, the method may include altering a volume level of the
voice signals and/or
identifying a first language corresponding to the voice signals and producing
translated signals
corresponding to a second language. Moreover, the method may include
processing the voice
signals and transcribing dictation corresponding to the voice signals.
[0017] In some implementations, the method includes communicating with a cloud
computing
service. For example, the method may include communicating with a remote
computing device
in communication with the cloud computing service. The remote computing device
communicating with the robot through the cloud computing service.
[0018] The details of one or more implementations of the disclosure are set
forth in the
accompanying drawings and the description below. Other aspects, features, and
advantages will
be apparent from the description and drawings, and from the claims.
4
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
DESCRIPTION OF DRAWINGS
[0019] FIG. 1 is a perspective view of an exemplary mobile human interface
robot.
[0020] FIG. 2 is a schematic view of an exemplary mobile human interface
robot.
[0021] FIG. 3 is an elevated perspective view of an exemplary mobile human
interface robot.
[0022] FIG. 4A is a front perspective view of an exemplary base for a mobile
human
interface robot.
[0023] FIG. 4B is a rear perspective view of the base shown in FIG. 4A.
[0024] FIG. 4C is a top view of the base shown in FIG. 4A.
[0025] FIG. 5A is a front schematic view of an exemplary base for a mobile
human
interface robot.
[0026] FIG. 5B is a top schematic view of an exemplary base for a mobile human
interface robot.
[0027] FIG. 5C is a front view of an exemplary holonomic wheel for a mobile
human
interface robot.
[0028] FIG. 5D is a side view of the wheel shown in FIG. 5C.
[0029] FIG. 6 is a front perspective view of an exemplary torso for a mobile
human
interface robot.
[0030] FIG. 7 is a front perspective view of an exemplary neck for a mobile
human
interface robot.
[0031] FIGS. 8A-8G are schematic views of exemplary circuitry for a mobile
human
interface robot.
[0032] FIG. 9 is a perspective view of an exemplary mobile human interface
robot
having detachable web pads.
[0033] FIGS. IOA-IOE perspective views of people interacting with an exemplary
mobile human interface robot.
[0034] FIG. 11 is a schematic view of an exemplary mobile human interface
robot.
[0035] FIG. 12 is a perspective view of an exemplary mobile human interface
robot
having multiple sensors pointed toward the ground.
[0036] FIG. 13 provides an exemplary telephony schematic for initiating and
conducting communication with a mobile human interface robot.
5
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[0037] FIGS. 14A-14C provide schematic views of exemplary robot system
architectures.
[0038] FIG. 15 is a schematic view of an exemplary control system executed by
a
controller of a mobile human interface robot.
[0039] FIG. 16A is a schematic view of an exemplary mobile human interface
robot
displaying an image having labels.
[0040] FIG. 16B is a schematic view of an exemplary robot system architecture.
[0041] FIG. 17 is a schematic view of an exemplary person detection system.
[0042] FIG. 18A is a schematic view of an exemplary occupancy map.
[0043] FIG. 18B is a schematic view of a mobile robot having a field of view
of a
scene in a working area.
[0044] Like reference symbols in the various drawings indicate like elements.
DETAILED DESCRIPTION
[0045] Mobile robots can interact or interface with humans to provide a number
of
services that range from home assistance to commercial assistance and more. In
the
example of home assistance, a mobile robot can assist elderly people with
everyday tasks,
including, but not limited to, maintaining a medication regime, mobility
assistance,
communication assistance (e.g., video conferencing, telecommunications,
Internet access,
etc.), home or site monitoring (inside and/or outside), person monitoring,
and/or
providing a personal emergency response system (PERS). For commercial
assistance,
the mobile robot can provide videoconferencing (e.g., in a hospital setting),
a point of
sale terminal, interactive information/marketing terminal, etc.
[0046] Referring to FIGS. 1-2, in some implementations, a mobile robot 100
includes
a robot body 110 (or chassis) that defines a forward drive direction F. The
robot 100 also
includes a drive system 200, an interfacing module 300, and a sensor system
400, each
supported by the robot body 110 and in communication with a controller 500
that
coordinates operation and movement of the robot 100. A power source 105 (e.g.,
battery
or batteries) can be carried by the robot body 110 and in electrical
communication with,
and deliver power to, each of these components, as necessary. For example, the
controller 500 may include a computer capable of > 1000 MIPS (million
instructions per
6
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
second) and the power source 105 provides a battery sufficient to power the
computer for
more than three hours.
[0047] The robot body 110, in the examples shown, includes a base 120, at
least one
leg 130 extending upwardly from the base 120, and a torso 140 supported by the
at least
one leg 130. The base 120 may support at least portions of the drive system
200. The
robot body 110 also includes a neck 150 supported by the torso 140. The neck
150
supports a head 160, which supports at least a portion of the interfacing
module 300. The
base 120 includes enough weight (e.g., by supporting the power source 105
(batteries) to
maintain a low center of gravity CGB of the base 120 and a low overall center
of gravity
CGR of the robot 100 for maintaining mechanical stability.
[0048] Referring to FIGS. 3 and 4A-4C, in some implementations, the base 120
defines a trilaterally symmetric shape (e.g., a triangular shape from the top
view). For
example, the base 120 may include a base chassis 122 that supports a base body
124
having first, second, and third base body portions 124a, 124b, 124c
corresponding to each
leg of the trilaterally shaped base 120 (see e.g., FIG. 4A). Each base body
portion 124a,
124b, 124c can be movably supported by the base chassis 122 so as to move
independently with respect to the base chassis 122 in response to contact with
an object.
The trilaterally symmetric shape of the base 120 allows bump detection 360
around the
robot 100. Each base body portion 124a, 124b, 124c can have an associated
contact
sensor e.g., capacitive sensor, read switch, etc.) that detects movement of
the
corresponding base body portion 124a, 124b, 124c with respect to the base
chassis 122.
[0049] In some implementations, the drive system 200 provides omni-directional
and/or holonomic motion control of the robot 100. As used herein the term
"omni-
directional" refers to the ability to move in substantially any planar
direction, i.e., side-to-
side (lateral), forward/back, and rotational. These directions are generally
referred to
herein as x, y, and Oz, respectively. Furthermore, the term "holonomic" is
used in a
manner substantially consistent with the literature use of the term and refers
to the ability
to move in a planar direction with three planar degrees of freedom, i.e., two
translations
and one rotation. Hence, a holonomic robot has the ability to move in a planar
direction
at a velocity made up of substantially any proportion of the three planar
velocities
7
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
(forward/back, lateral, and rotational), as well as the ability to change
these proportions in
a substantially continuous manner.
[0050] The robot 100 can operate in human environments (e.g., environments
typically designed for bipedal, walking occupants) using wheeled mobility. In
some
implementations, the drive system 200 includes first, second, and third drive
wheels
210a, 2l Ob, 210c equally spaced (i.e., trilaterally symmetric) about the
vertical axis Z
(e.g., 120 degrees apart); however, other arrangements are possible as well.
Referring to
FIGS. 5A and 5B, the drive wheels 210a, 2l Ob, 210c may define a transverse
arcuate
rolling surface (i.e., a curved profile in a direction transverse or
perpendicular to the
rolling direction DR), which may aid maneuverability of the holonomic drive
system 200.
Each drive wheel 210a, 2l Ob, 210c is coupled to a respective drive motor
220a, 220b,
220c that can drive the drive wheel 210a, 2l Ob, 210c in forward and/or
reverse directions
independently of the other drive motors 220a, 220b, 220c. Each drive motor
220a-c can
have a respective encoder 212 (FIG. 8C), which provides wheel rotation
feedback to the
controller 500. In some examples, each drive wheels 210a, 210b, 210c is
mounted on or
near one of the three points of an equilateral triangle and having a drive
direction
(forward and reverse directions) that is perpendicular to an angle bisector of
the
respective triangle end. Driving the trilaterally symmetric holonomic base 120
with a
forward driving direction F, allows the robot 100 to transition into non
forward drive
directions for autonomous escape from confinement or clutter and then rotating
and/or
translating to drive along the forward drive direction F after the escape has
been resolved.
[0051] Referring to FIGS. 5C and 5D, in some implementations, each drive wheel
210 includes inboard and outboard rows 232, 234 of rollers 230, each have a
rolling
direction Dr perpendicular to the rolling direction DR of the drive wheel 210.
The rows
232, 234 of rollers 230 can be staggered (e.g., such that one roller 230 of
the inboard row
232 is positioned equally between two adjacent rollers 230 of the outboard row
234. The
rollers 230 provide infinite slip perpendicular to the drive direction the
drive wheel 210.
The rollers 230 define an arcuate (e.g., convex) outer surface 235
perpendicular to their
rolling directions Dr, such that together the rollers 230 define the circular
or substantially
circular perimeter of the drive wheel 210. The profile of the rollers 230
affects the
overall profile of the drive wheel 210. For example, the rollers 230 may
define arcuate
8
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
outer roller surfaces 235 that together define a scalloped rolling surface of
the drive
wheel 210 (e.g., as treads for traction). However, configuring the rollers 230
to have
contours that define a circular overall rolling surface of the drive wheel 210
allows the
robot 100 to travel smoothly on a flat surface instead of vibrating vertically
with a wheel
tread. When approaching an object at an angle, the staggered rows 232, 234 of
rollers
230 (with radius r) can be used as treads to climb objects as tall or almost
as tall as a
wheel radius R of the drive wheel 210.
[0052] In the examples shown in FIGS. 3-5B, the first drive wheel 210a is
arranged
as a leading drive wheel along the forward drive direction F with the
remaining two drive
wheels 21 Ob, 21 Oc trailing behind. In this arrangement, to drive forward,
the controller
500 may issue a drive command that causes the second and third drive wheels 2l
Ob, 210c
to drive in a forward rolling direction at an equal rate while the first drive
wheel 210a
slips along the forward drive direction F. Moreover, this drive wheel
arrangement allows
the robot 100 to stop short (e.g., incur a rapid negative acceleration against
the forward
drive direction F). This is due to the natural dynamic instability of the
three wheeled
design. If the forward drive direction F were along an angle bisector between
two
forward drive wheels, stopping short would create a torque that would force
the robot 100
to fall, pivoting over its two "front" wheels. Instead, travelling with one
drive wheel
210a forward naturally supports or prevents the robot 100 from toppling over
forward, if
there is need to come to a quick stop. When accelerating from a stop, however,
the
controller 500 may take into account a moment of inertia I of the robot 100
from its
overall center of gravity CGR.
[0053] In some implementations of the drive system 200, each drive wheel 210a,
2l Ob, 210 has a rolling direction DR radially aligned with a vertical axis Z,
which is
orthogonal to X and Y axes of the robot 100. The first drive wheel 210a can be
arranged
as a leading drive wheel along the forward drive direction F with the
remaining two drive
wheels 210b, 210c trailing behind. In this arrangement, to drive forward, the
controller
500 may issue a drive command that causes the first drive wheel 210a to drive
in a
forward rolling direction and the second and third drive wheels 210b, 210c to
drive at an
equal rate as the first drive wheel 210a, but in a reverse direction.
9
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[0054] In other implementations, the drive system 200 can be arranged to have
the
first and second drive wheels 210a, 210b positioned such that an angle
bisector of an
angle between the two drive wheels 210a, 210b is aligned with the forward
drive
direction F of the robot 100. In this arrangement, to drive forward, the
controller 500
may issue a drive command that causes the first and second drive wheels 210a,
210b to
drive in a forward rolling direction and an equal rate, while the third drive
wheel 210c
drives in a reverse direction or remains idle and is dragged behind the first
and second
drive wheels 210a, 210b. To turn left or right while driving forward, the
controller 500
may issue a command that causes the corresponding first or second drive wheel
210a,
21 Ob to drive at relatively quicker/slower rate. Other drive system 200
arrangements can
be used as well. The drive wheels 210a, 2l Ob, 210c may define a cylindrical,
circular,
elliptical, or polygonal profile.
[0055] Referring again to FIGS. 1-3, the base 120 supports at least one leg
130
extending upward in the Z direction from the base 120. The leg(s) 130 may be
configured to have a variable height for raising and lowering the torso 140
with respect to
the base 120. In some implementations, each leg 130 includes first and second
leg
portions 132, 134 that move with respect to each other (e.g., telescopic,
linear, and/or
angular movement). Rather than having extrusions of successively smaller
diameter
telescopically moving in and out of each other and out of a relatively larger
base
extrusion, the second leg portion 134, in the examples shown, moves
telescopically over
the first leg portion 132, thus allowing other components to be placed along
the second
leg portion 134 and potentially move with the second leg portion 134 to a
relatively close
proximity of the base 120. The leg 130 may include an actuator assembly 136
(FIG. 8C)
for moving the second leg portion 134 with respect to the first leg portion
132. The
actuator assembly 136 may include a motor driver 138a in communication with a
lift
motor 138b and an encoder 138c, which provides position feedback to the
controller 500.
[0056] Generally, telescopic arrangements include successively smaller
diameter
extrusions telescopically moving up and out of relatively larger extrusions at
the base 120
in order to keep a center of gravity CGL of the entire leg 130 as low as
possible.
Moreover, stronger and/or larger components can be placed at the bottom to
deal with the
greater torques that will be experienced at the base 120 when the leg 130 is
fully
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
extended. This approach, however, offers two problems. First, when the
relatively
smaller components are placed at the top of the leg 130, any rain, dust, or
other
particulate will tend to run or fall down the extrusions, infiltrating a space
between the
extrusions, thus obstructing nesting of the extrusions. This creates a very
difficult sealing
problem while still trying to maintain full mobility/articulation of the leg
130. Second, it
may be desirable to mount payloads or accessories on the robot 100. One common
place
to mount accessories is at the top of the torso 140. If the second leg portion
134 moves
telescopically in and out of the first leg portion, accessories and components
could only
be mounted above the entire second leg portion 134, if they need to move with
the torso
140. Otherwise, any components mounted on the second leg portion 134 would
limit the
telescopic movement of the leg 130.
[0057] By having the second leg portion 134 move telescopically over the first
leg
portion 132, the second leg portion 134 provides additional payload attachment
points
that can move vertically with respect to the base 120. This type of
arrangement causes
water or airborne particulate to run down the torso 140 on the outside of
every leg portion
132, 134 (e.g., extrusion) without entering a space between the leg portions
132, 134.
This greatly simplifies sealing any joints of the leg 130. Moreover,
payload/accessory
mounting features of the torso 140 and/or second leg portion 134 are always
exposed and
available no matter how the leg 130 is extended.
[0058] Referring to FIGS. 3 and 6, the leg(s) 130 support the torso 140, which
may
have a shoulder 142 extending over and above the base 120. In the example
shown, the
torso 140 has a downward facing or bottom surface 144 (e.g., toward the base)
forming at
least part of the shoulder 142 and an opposite upward facing or top surface
146, with a
side surface 148 extending therebetween. The torso 140 may define various
shapes or
geometries, such as a circular or an elliptical shape having a central portion
141
supported by the leg(s) 130 and a peripheral free portion 143 that extends
laterally
beyond a lateral extent of the leg(s) 130, thus providing an overhanging
portion that
defines the downward facing surface 144. In some examples, the torso 140
defines a
polygonal or other complex shape that defines a shoulder, which provides an
overhanging
portion that extends beyond the leg(s) 130 over the base 120.
11
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[0059] The robot 100 may include one or more accessory ports 170 (e.g.,
mechanical
and/or electrical interconnect points) for receiving payloads. The accessory
ports 170 can
be located so that received payloads do not occlude or obstruct sensors of the
sensor
system 400 (e.g., on the bottom and/or top surfaces 144, 146 of the torso 140,
etc.). In
some implementations, as shown in FIG. 6, the torso 140 includes one or more
accessory
ports 170 on a rearward portion 149 of the torso 140 for receiving a payload
in the basket
340, for example, and so as not to obstruct sensors on a forward portion 147
of the torso
140 or other portions of the robot body 110.
[0060] An external surface of the torso 140 may be sensitive to contact or
touching
by a user, so as to receive touch commands from the user. For example, when
the user
touches the top surface 146 of the torso 140, the robot 100 responds by
lowering a height
HT of the torso with respect to the floor (e.g., by decreasing the height HL
of the leg(s)
130 supporting the torso 140). Similarly, when the user touches the bottom
surface 144
of the torso 140, the robot 100 responds by raising the torso 140 with respect
to the floor
(e.g., by increasing the height HL of the leg(s) 130 supporting the torso
140). Moreover,
upon receiving a user touch on forward, rearward, right or left portions of
side surface
148 of the torso 140, the robot 100 responds by moving in a corresponding
direction of
the received touch command (e.g., rearward, forward, left, and right,
respectively). The
external surface(s) of the torso 140 may include a capacitive sensor in
communication
with the controller 500 that detects user contact.
[0061] Referring again to FIGS. 1-3 and 7, the torso 140 supports the neck
150,
which provides panning and tilting of the head 160 with respect to the torso
140. In the
examples shown, the neck 150 includes a rotator 152 and a tilter 154. The
rotator 152
may provide a range of angular movement OR (e.g., about the Z axis) of between
about
90 and about 360 . Other ranges are possible as well. Moreover, in some
examples, the
rotator 152 includes electrical connectors or contacts that allow continuous
360 rotation
of the head 150 with respect to the torso 140 in an unlimited number of
rotations while
maintaining electrical communication between the head 150 and the remainder of
the
robot 100. The tilter 154 may include the same or similar electrical
connectors or
contacts allow rotation of the head 150 with respect to the torso 140 while
maintaining
electrical communication between the head 150 and the remainder of the robot
100. The
12
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
rotator 152 may include a rotator motor 151 coupled to or engaging a ring 153
(e.g., a
toothed ring rack). The tilter 154 may move the head at an angle OT (e.g.,
about the Y
axis) with respect to the torso 140 independently of the rotator 152. In some
examples
that tilter 154 includes a tilter motor 155, which moves the head 150 between
an angle OT
of 90 with respect to Z-axis. Other ranges are possible as well, such as
45 , etc. The
robot 100 may be configured so that the leg(s) 130, the torso 140, the neck
150, and the
head 160 stay within a perimeter of the base 120 for maintaining stable
mobility of the
robot 100. In the exemplary circuit schematic shown in FIG. I OF, the neck 150
includes
a pan-tilt assembly 151 that includes the rotator 152 and a tilter 154 along
with
corresponding motor drivers 156a, 156b and encoders 158a, 158b.
[0062] The head 160 may be sensitive to contact or touching by a user, so as
to
receive touch commands from the user. For example, when the user pulls the
head 160
forward, the head 160 tilts forward with passive resistance and then holds the
position.
More over, if the user pushes/pulls the head 160 vertically downward, the
torso 140 may
lower (via a reduction in length of the leg 130) to lower the head 160. The
head 160
and/or neck 150 may include strain gauges and/or contact sensors 165 (FIG. 7)
that sense
user contact or manipulation.
[0063] FIGS. 8A-8G provide exemplary schematics of circuitry for the robot
100.
FIGS. 8A-8C provide exemplary schematics of circuitry for the base 120, which
may
house the proximity sensors, such as the sonar proximity sensors 410 and the
cliff
proximity sensors 420, contact sensors 430, the laser scanner 440, the sonar
scanner 460,
and the drive system 200. The base 120 may also house the controller 500, the
power
source 105, and the leg actuator assembly 136. The torso 140 may house a
microcontroller 145, the microphone(s) 330, the speaker(s) 340, the scanning 3-
D image
sensor 450a, and a torso touch sensor system 480, which allows the controller
500 to
receive and respond to user contact or touches (e.g., as by moving the torso
140 with
respect to the base 120, panning and/or tilting the neck 150, and/or issuing
commands to
the drive system 200 in response thereto). The neck 150 may house a pan-tilt
assembly
151 that may include a pan motor 152 having a corresponding motor driver 156a
and
encoder 138a, and a tilt motor 154 152 having a corresponding motor driver
156b and
encoder 138b. The head 160 may house one or more web pads 310 and a camera
320.
13
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[0064] With reference to FIGS. 1-3 and 9, in some implementations, the head
160
supports one or more portions of the interfacing module 300. The head 160 may
include
a dock 302 for releasably receiving one or more computing tablets 310, also
referred to as
a web pad or a tablet PC, each of which may have a touch screen 312. The web
pad 310
may be oriented forward, rearward or upward. In some implementations, web pad
310
includes a touch screen, optional I/O (e.g., buttons and/or connectors, such
as micro-
USB, etc.) a processor, and memory in communication with the processor. An
exemplary web pad 310 includes the Apple iPad is by Apple, Inc. In some
examples, the
web pad and 10 functions as the controller 500 or assist the controller 500
and controlling
the robot 100. In some examples, the dock 302 includes a first computing
tablet 310a
fixedly attached thereto (e.g., a wired interface for data transfer at a
relatively higher
bandwidth, such as a gigabit rate) and a second computing tablet 31 Ob
removably
connected thereto. The second web pad 310b may be received over the first web
pad
31 Oa as shown in FIG. 9, or the second web pad 31 Ob may be received on an
opposite
facing side or other side of the head 160 with respect to the first web pad 31
Oa. In
additional examples, the head 160 supports a single web pad 310, which may be
either
fixed or removably attached thereto. The touch screen 312 may detected,
monitor, and/or
reproduce points of user touching thereon for receiving user inputs and
providing a
graphical user interface that is touch interactive. In some examples, the web
pad 310
includes a touch screen caller that allows the user to find it when it has
been removed
from the robot 100.
[0065] In some implementations, the robot 100 includes multiple web pad docks
302
on one or more portions of the robot body 110. In the example shown in FIG. 9,
the
robot 100 includes a web pad dock 302 optionally disposed on the leg 130
and/or the
torso 140. This allows the user to dock a web pad 310 at different heights on
the robot
100, for example, to accommodate users of different height, capture video
using a camera
of the web pad 310 in different vantage points, and/or to receive multiple web
pads 310
on the robot 100.
[0066] The interfacing module 300 may include a camera 320 disposed on the
head
160 (see e.g., FIGS. 2), which can be used to capture video from elevated
vantage point
of the head 160 (e.g., for videoconferencing). In the example shown in FIG. 3,
the
14
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
camera 320 is disposed on the neck 150. In some examples, the camera 320 is
operated
only when the web pad 310, 310a is detached or undocked from the head 160.
When the
web pad 310, 310a is attached or docked on the head 160 in the dock 302 (and
optionally
covering the camera 320), the robot 100 may use a camera of the web pad 31 Oa
for
capturing video. In such instances, the camera 320 may be disposed behind the
docked
web pad 310 and enters an active state when the web pad 310 is detached or
undocked
from the head 160 and an inactive state when the web pad 310 is attached or
docked on
the head 160.
[0067] The robot 100 can provide videoconferencing (e.g., at 24 fps) through
the
interface module 300 (e.g., using a web pad 310, the camera 320, the
microphones 320,
and/or the speakers 340). The videoconferencing can be multiparty. The robot
100 can
provide eye contact between both parties of the videoconferencing by
maneuvering the
head 160 to face the user. Moreover, the robot 100 can have a gaze angle of <
5 degrees
(e.g., an angle away from an axis normal to the forward face of the head 160).
At least
one 3-D image sensor 450 and/or the camera 320 on the robot 100 can capture
life size
images including body language. The controller 500 can synchronize audio and
video
(e.g., with the difference of <50 ms). In the example shown in FIGS. l0A-10E,
robot
100 can provide videoconferencing for people standing or sitting by adjusting
the height
of the web pad 310 on the head 160 and/or the camera 320 (by raising or
lowering the
torso 140) and/or panning and/or tilting the head 160. The camera 320 may be
movable
within at least one degree of freedom separately from the web pad 310. In some
examples, the camera 320 has an objective lens positioned more than 3 feet
from the
ground, but no more than 10 percent of the web pad height from a top edge of a
display
area of the web pad 310. Moreover, the robot 100 can zoom the camera 320 to
obtain
close-up pictures or video about the robot 100. The head 160 may include one
or more
speakers 340 so as to have sound emanate from the head 160 near the web pad
310
displaying the videoconferencing.
[0068] In some examples, the robot 100 can receive user inputs into the web
pad 310
(e.g., via a touch screen), as shown in FIG. 10E. In some implementations, the
web pad
310 is a display or monitor, while in other implementations the web pad 310 is
a tablet
computer. The web pad 310 can have easy and intuitive controls, such as a
touch screen,
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
providing high interactivity. The web pad 310 may have a monitor display 312
(e.g.,
touch screen) having a display area of 150 square inches or greater movable
with at least
one degree of freedom.
[0069] The robot 100 can provide EMR integration, in some examples, by
providing
video conferencing between a doctor and patient and/or other doctors or
nurses. The
robot 100 may include pass-through consultation instruments. For example, the
robot
100 may include a stethoscope configured to pass listening to the
videoconferencing user
(e.g., a doctor). In other examples, the robot includes connectors 170 that
allow direct
connection to Class II medical devices, such as electronic stethoscopes,
otoscopes and
ultrasound, to transmit medical data to a remote user (physician).
[0070] In the example shown in FIG. I OB, a user may remove the web pad 310
from
the web pad dock 302 on the head 160 for remote operation of the robot 100,
videoconferencing (e.g., using a camera and microphone of the web pad 310),
and/or
usage of software applications on the web pad 310. The robot 100 may include
first and
second cameras 320a, 320b on the head 160 to obtain different vantage points
for
videoconferencing, navigation, etc., while the web pad 310 is detached from
the web pad
dock 302.
[0071] Interactive applications executable on the controller 500 and/or in
communication with the controller 500 may require more than one display on the
robot
100. Multiple web pads 310 associated with the robot 100 can provide different
combinations of "FaceTime", Telestration, HD look at this-cam (e.g., for web
pads 310
having built in cameras), can act as a remote operator control unit (OCU) for
controlling
the robot 100 remotely, and/or provide a local user interface pad.
[0072] Referring again to FIG. 6, the interfacing module 300 may include a
microphone 330 (e.g., or micro-phone array) for receiving sound inputs and one
or more
speakers 330 disposed on the robot body 110 for delivering sound outputs. The
microphone 330 and the speaker(s) 340 may each communicate with the controller
500.
In some examples, the interfacing module 300 includes a basket 360, which may
be
configured to hold brochures, emergency information, household items, and
other items.
[0073] Referring to FIGS. 1-4C, 11 and 12, to achieve reliable and robust
autonomous movement, the sensor system 400 may include several different types
of
16
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
sensors which can be used in conjunction with one another to create a
perception of the
robot's environment sufficient to allow the robot 100 to make intelligent
decisions about
actions to take in that environment. The sensor system 400 may include one or
more
types of sensors supported by the robot body 110, which may include obstacle
detection
obstacle avoidance (ODOA) sensors, communication sensors, navigation sensors,
etc.
For example, these sensors may include, but not limited to, proximity sensors,
contact
sensors, three-dimensional (3D) imaging / depth map sensors, a camera (e.g.,
visible light
and/or infrared camera), sonar, radar, LIDAR (Light Detection And Ranging,
which can
entail optical remote sensing that measures properties of scattered light to
find range
and/or other information of a distant target), LADAR (Laser Detection and
Ranging), etc.
In some implementations, the sensor system 400 includes ranging sonar sensors
410 (e.g.,
nine about a perimeter of the base 120), proximity cliff detectors 420,
contact sensors
430, a laser scanner 440, one or more 3-D imaging/depth sensors 450, and an
imaging
sonar 460.
[0074] There are several challenges involved in placing sensors on a robotic
platform.
First, the sensors need to be placed such that they have maximum coverage of
areas of
interest around the robot 100. Second, the sensors may need to be placed in
such a way
that the robot 100 itself causes an absolute minimum of occlusion to the
sensors; in
essence, the sensors cannot be placed such that they are "blinded" by the
robot itself.
Third, the placement and mounting of the sensors should not be intrusive to
the rest of the
industrial design of the platform. In terms of aesthetics, it can be assumed
that a robot
with sensors mounted inconspicuously is more "attractive" than otherwise. In
terms of
utility, sensors should be mounted in a manner so as not to interfere with
normal robot
operation (snagging on obstacles, etc.).
[0075] In some implementations, the sensor system 400 includes a set or an
array of
proximity sensors 410, 420 in communication with the controller 500 and
arranged in one
or more zones or portions of the robot 100 (e.g., disposed on or near the base
body
portion 124a, 124b, 124c of the robot body 110) for detecting any nearby or
intruding
obstacles. The proximity sensors 410, 420 may be converging infrared (IR)
emitter-
sensor elements, sonar sensors, ultrasonic sensors, and/or imaging sensors
(e.g., 3D depth
17
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
map image sensors) that provide a signal to the controller 500 when an object
is within a
given range of the robot 100.
[0076] In the example shown in FIGS. 4A-4C, the robot 100 includes an array of
sonar-type proximity sensors 410 disposed (e.g., substantially equidistant)
around the
base body 120 and arranged with an upward field of view. First, second, and
third sonar
proximity sensors 410a, 4l Ob, 410c are disposed on or near the first
(forward) base body
portion 124a, with at least one of the sonar proximity sensors near a radially
outer-most
edge 125a of the first base body 124a. Fourth, fifth, and sixth sonar
proximity sensors
410d, 410e, 410f are disposed on or near the second (right) base body portion
124b, with
at least one of the sonar proximity sensors near a radially outer-most edge
125b of the
second base body 124b. Seventh, eighth, and ninth sonar proximity sensors 4l
Og, 410h,
410i are disposed on or near the third (right) base body portion 124c, with at
least one of
the sonar proximity sensors near a radially outer-most edge 125c of the third
base body
124c. This configuration provides at least three zones of detection.
[0077] In some examples, the set of sonar proximity sensors 410 (e.g., 410a-
410i)
disposed around the base body 120 are arranged to point upward (e.g.,
substantially in the
Z direction) and optionally angled outward away from the Z axis, thus creating
a
detection curtain 412 around the robot 100. Each sonar proximity sensor 410a-
410i may
have a shroud or emission guide 414 that guides the sonar emission upward or
at least not
toward the other portions of the robot body 110 (e.g., so as not to detect
movement of the
robot body 110 with respect to itself). The emission guide 414 may define a
shell or half
shell shape. In the example shown, the base body 120 extends laterally beyond
the leg
130, and the sonar proximity sensors 410 (e.g., 410a-410i) are disposed on the
base body
120 (e.g., substantially along a perimeter of the base body 120) around the
leg 130.
Moreover, the upward pointing sonar proximity sensors 410 are spaced to create
a
continuous or substantially continuous sonar detection curtain 412 around the
leg 130.
The sonar detection curtain 412 can be used to detect obstacles having
elevated lateral
protruding portions, such as table tops, shelves, etc.
[0078] The upward looking sonar proximity sensors 410 provide the ability to
see
objects that are primarily in the horizontal plane, such as table tops. These
objects, due to
their aspect ratio, may be missed by other sensors of the sensor system, such
as the laser
18
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
scanner 440 or imaging sensors 450, and as such, can pose a problem to the
robot 100.
The upward viewing sonar proximity sensors 410 arranged around the perimeter
of the
base 120 provide a means for seeing or detecting those type of
objects/obstacles.
Moreover, the sonar proximity sensors 410 can be placed around the widest
points of the
base perimeter and angled slightly outwards, so as not to be occluded or
obstructed by the
torso 140 or head 160 of the robot 100, thus not resulting in false positives
for sensing
portions of the robot 100 itself. In some implementations, the sonar proximity
sensors
410 are arranged (upward and outward) to leave a volume about the torso 140
outside of
a field of view of the sonar proximity sensors 410 and thus free to receive
mounted
payloads or accessories, such as the basket 340. The sonar proximity sensors
410 can be
recessed into the base body 124 to provide visual concealment and no external
features to
snag on or hit obstacles.
[0079] The sensor system 400 may include or more sonar proximity sensors 410
(e.g., a rear proximity sensor 410j) directed rearward (e.g., opposite to the
forward drive
direction F) for detecting obstacles while backing up. The rear sonar
proximity sensor
410j may include an emission guide 414 to direct its sonar detection field
412.
Moreover, the rear sonar proximity sensor 410j can be used for ranging to
determine a
distance between the robot 100 and a detected object in the field of view of
the rear sonar
proximity sensor 410j (e.g., as "back-up alert"). In some examples, the rear
sonar
proximity sensor 410j is mounted recessed within the base body 120 so as to
not provide
any visual or functional irregularity in the housing form.
[0080] Referring to FIGS. 3 and 4B, in some implementations, the robot 100
includes
cliff proximity sensors 420 arranged near or about the drive wheels 210a, 2l
Ob, 210c, so
as to allow cliff detection before the drive wheels 210a, 210b, 210c encounter
a cliff (e.g.,
stairs). For example, a cliff proximity sensors 420 can be located at or near
each of the
radially outer-most edges 125a-c of the base bodies 124a-c and in locations
therebetween.
In some cases, cliff sensing is implemented using infrared (IR) proximity or
actual range
sensing, using an infrared emitter 422 and an infrared detector 424 angled
toward each
other so as to have an overlapping emission and detection fields, and hence a
detection
zone, at a location where a floor should be expected. IR proximity sensing can
have a
relatively narrow field of view, may depend on surface albedo for reliability,
and can
19
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
have varying range accuracy from surface to surface. As a result, multiple
discrete
sensors can be placed about the perimeter of the robot 100 to adequately
detect cliffs
from multiple points on the robot 100. Moreover, IR proximity based sensors
typically
cannot discriminate between a cliff and a safe event, such as just after the
robot 100
climbs a threshold.
[0081] The cliff proximity sensors 420 can detect when the robot 100 has
encountered a falling edge of the floor, such as when it encounters a set of
stairs. The
controller 500 (executing a control system) may execute behaviors that cause
the robot
100 to take an action, such as changing its direction of travel, when an edge
is detected.
In some implementations, the sensor system 400 includes one or more secondary
cliff
sensors (e.g., other sensors configured for cliff sensing and optionally other
types of
sensing). The cliff detecting proximity sensors 420 can be arranged to provide
early
detection of cliffs, provide data for discriminating between actual cliffs and
safe events
(such as climbing over thresholds), and be positioned down and out so that
their field of
view includes at least part of the robot body 110 and an area away from the
robot body
110. In some implementations, the controller 500 executes cliff detection
routine that
identifies and detects an edge of the supporting work surface (e.g., floor),
an increase in
distance past the edge of the work surface, and/or an increase in distance
between the
robot body 110 and the work surface. This implementation allows: 1) early
detection of
potential cliffs (which may allow faster mobility speeds in unknown
environments); 2)
increased reliability of autonomous mobility since the controller 500 receives
cliff
imaging information from the cliff detecting proximity sensors 420 to know if
a cliff
event is truly unsafe or if it can be safely traversed (e.g., such as climbing
up and over a
threshold); 3) a reduction in false positives of cliffs (e.g., due to the use
of edge detection
versus the multiple discrete IR proximity sensors with a narrow field of
view).
Additional sensors arranged as "wheel drop" sensors can be used for redundancy
and for
detecting situations where a range-sensing camera cannot reliably detect a
certain type of
cliff.
[0082] Threshold and step detection allows the robot 100 to effectively plan
for either
traversing a climb-able threshold or avoiding a step that is too tall. This
can be the same
for random objects on the work surface that the robot 100 may or may not be
able to
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
safely traverse. For those obstacles or thresholds that the robot 100
determines it can
climb, knowing their heights allows the robot 100 to slow down appropriately,
if deemed
needed, to allow for a smooth transition in order to maximize smoothness and
minimize
any instability due to sudden accelerations. In some implementations,
threshold and step
detection is based on object height above the work surface along with geometry
recognition (e.g., discerning between a threshold or an electrical cable
versus a blob, such
as a sock). Thresholds may be recognized by edge detection. The controller 500
may
receive imaging data from the cliff detecting proximity sensors 420 (or
another imaging
sensor on the robot 100), execute an edge detection routine, and issue a drive
command
based on results of the edge detection routine. The controller 500 may use
pattern
recognition to identify objects as well. Threshold detection allows the robot
100 to
change its orientation with respect to the threshold to maximize smooth step
climbing
ability.
[0083] The proximity sensors 410, 420 may function alone, or as an
alternative, may
function in combination with one or more contact sensors 430 (e.g., bump
switches) for
redundancy. For example, one or more contact or bump sensors 430 on the robot
body
110 can detect if the robot 100 physically encounters an obstacle. Such
sensors may use
a physical property such as capacitance or physical displacement within the
robot 100 to
determine when it has encountered an obstacle. In some implementations, each
base
body portion 124a, 124b, 124c of the base 120 has an associated contact sensor
430 (e.g.,
capacitive sensor, read switch, etc.) that detects movement of the
corresponding base
body portion 124a, 124b, 124c with respect to the base chassis 122 (see e.g.,
FIG. 4A).
For example, each base body 124a-c may move radially with respect to the Z
axis of the
base chassis 122, so as to provide 3-way bump detection.
[0084] Referring again to FIGS. 1-4C, 11 and 12, in some implementations, the
sensor system 400 includes a laser scanner 440 mounted on a forward portion of
the robot
body 110 and in communication with the controller 500. In the examples shown,
the
laser scanner 440 is mounted on the base body 120 facing forward (e.g., having
a field of
view along the forward drive direction F) on or above the first base body 124a
(e.g., to
have maximum imaging coverage along the drive direction F of the robot).
Moreover,
the placement of the laser scanner on or near the front tip of the triangular
base 120
21
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
means that the external angle of the robotic base (e.g., 300 degrees) is
greater than a field
of view 442 of the laser scanner 440 (e.g., -285 degrees), thus preventing the
base 120
from occluding or obstructing the detection field of view 442 of the laser
scanner 440.
The laser scanner 440 can be mounted recessed within the base body 124 as much
as
possible without occluding its fields of view, to minimize any portion of the
laser scanner
sticking out past the base body 124 (e.g., for aesthetics and to minimize
snagging on
obstacles).
[0085] The laser scanner 440 scans an area about the robot 100 and the
controller
500, using signals received from the laser scanner 440, creates an environment
map or
object map of the scanned area. The controller 500 may use the object map for
navigation, obstacle detection, and obstacle avoidance. Moreover, the
controller 500 may
use sensory inputs from other sensors of the sensor system 400 for creating
object map
and/or for navigation.
[0086] In some examples, the laser scanner 440 is a scanning LIDAR, which may
use
a laser that quickly scans an area in one dimension, as a "main" scan line,
and a time-of-
flight imaging element that uses a phase difference or similar technique to
assign a depth
to each pixel generated in the line (returning a two dimensional depth line in
the plane of
scanning). In order to generate a three dimensional map, the LIDAR can perform
an
"auxiliary" scan in a second direction (for example, by "nodding" the
scanner). This
mechanical scanning technique can be complemented, if not supplemented, by
technologies such as the "Flash" LIDAR/LADAR and "Swiss Ranger" type focal
plane
imaging element sensors, techniques which use semiconductor stacks to permit
time of
flight calculations for a full 2-D matrix of pixels to provide a depth at each
pixel, or even
a series of depths at each pixel (with an encoded illuminator or illuminating
laser).
[0087] The sensor system 400 may include one or more three-dimensional (3-D)
image sensors 450 in communication with the controller 500. If the 3-D image
sensor
450 has a limited field of view, the controller 500 or the sensor system 400
can actuate
the 3-D image sensor 450a in a side-to-side scanning manner to create a
relatively wider
field of view to perform robust ODOA. Referring to FIGS. 1-3 and 12, in some
implementations, the robot 100 includes a scanning 3-D image sensor 450a
mounted on a
forward portion of the robot body 110 with a field of view along the forward
drive
22
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
direction F (e.g., to have maximum imaging coverage along the drive direction
F of the
robot). The scanning 3-D image sensor 450a can be used primarily for obstacle
detection/obstacle avoidance (ODOA). In the example shown, the scanning 3-D
image
sensor 450a is mounted on the torso 140 underneath the shoulder 142 or on the
bottom
surface 144 and recessed within the torso 140 (e.g., flush or past the bottom
surface 144),
as shown in FIG. 3, for example, to prevent user contact with the scanning 3-D
image
sensor 450a. The scanning 3-D image sensor 450 can be arranged to aim
substantially
downward and away from the robot body 110, so as to have a downward field of
view
452 in front of the robot 100 for obstacle detection and obstacle avoidance
(ODOA) (e.g.,
with obstruction by the base 120 or other portions of the robot body 110).
Placement of
the scanning 3-D image sensor 450a on or near a forward edge of the torso 140
allows the
field of view of the 3-D image sensor 450 (e.g., -285 degrees) to be less than
an external
surface angle of the torso 140 (e.g., 300 degrees) with respect to the 3-D
image sensor
450, thus preventing the torso 140 from occluding or obstructing the detection
field of
view 452 of the scanning 3-D image sensor 450a. Moreover, the scanning 3-D
image
sensor 450a (and associated actuator) can be mounted recessed within the torso
140 as
much as possible without occluding its fields of view (e.g., also for
aesthetics and to
minimize snagging on obstacles). The distracting scanning motion of the
scanning 3-D
image sensor 450a is not visible to a user, creating a less distracting
interaction
experience. Unlike a protruding sensor or feature, the recessed scanning 3-D
image
sensor 450a will not tend to have unintended interactions with the environment
(snagging
on people, obstacles, etc.), especially when moving or scanning, as virtually
no moving
part extends beyond the envelope of the torso 140.
[0088] In some implementations, the sensor system 400 includes additional 3-D
image sensors 450 disposed on the base body 120, the leg 130, the neck 150,
and/or the
head 160. In the example shown in FIG. 1, the robot 100 includes 3-D image
sensors 450
on the base body 120, the torso 140, and the head 160. In the example shown in
FIG. 2,
the robot 100 includes 3-D image sensors 450 on the base body 120, the torso
140, and
the head 160. In the example shown in FIG. 11, the robot 100 includes 3-D
image
sensors 450 on the leg 130, the torso 140, and the neck 150. Other
configurations are
possible as well. One 3-D image sensor 450 (e.g., on the neck 150 and over the
head
23
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
160) can be used for people recognition, gesture recognition, and/or
videoconferencing,
while another 3-D image sensor 450 (e.g., on the base 120 and/or the leg 130)
can be
used for navigation and/or obstacle detection and obstacle avoidance.
[0089] A forward facing 3-D image sensor 450 disposed on the neck 150 and/or
the
head 160 can be used for person, face, and/or gesture recognition of people
about the
robot 100. For example, using signal inputs from the 3-D image sensor 450 on
the head
160, the controller 500 may recognize a user by creating a three-dimensional
map of the
viewed/captured user's face and comparing the created three-dimensional map
with
known 3-D images of people's faces and determining a match with one of the
known 3-D
facial images. Facial recognition may be used for validating users as
allowable users of
the robot 100. Moreover, one or more of the 3-D image sensors 450 can be used
for
determining gestures of person viewed by the robot 100, and optionally
reacting based on
the determined gesture(s) (e.g., hand pointing, waving, and or hand signals).
For
example, the controller 500 may issue a drive command in response to a
recognized hand
point in a particular direction.
[0090] The 3-D image sensors 450 may be capable of producing the following
types
of data: (i) a depth map, (ii) a reflectivity based intensity image, and/or
(iii) a regular
intensity image. The 3-D image sensors 450 may obtain such data by image
pattern
matching, measuring the flight time and/or phase delay shift for light emitted
from a
source and reflected off of a target.
[0091] In some implementations, reasoning or control software, executable on a
processor (e.g., of the robot controller 500), uses a combination of
algorithms executed
using various data types generated by the sensor system 400. The reasoning
software
processes the data collected from the sensor system 400 and outputs data for
making
navigational decisions on where the robot 100 can move without colliding with
an
obstacle, for example. By accumulating imaging data over time of the robot's
surroundings, the reasoning software can in turn apply effective methods to
selected
segments of the sensed image(s) to improve depth measurements of the 3-D image
sensors 450. This may include using appropriate temporal and spatial averaging
techniques.
24
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[0092] The reliability of executing robot collision free moves may be based
on: (i) a
confidence level built by high level reasoning over time and (ii) a depth-
perceptive sensor
that accumulates three major types of data for analysis - (a) a depth image,
(b) an active
illumination image and (c) an ambient illumination image. Algorithms cognizant
of the
different types of data can be executed on each of the images obtained by the
depth-
perceptive imaging sensor 450. The aggregate data may improve the confidence
level a
compared to a system using only one of the kinds of data.
[0093] The 3-D image sensors 450 may obtain images containing depth and
brightness data from a scene about the robot 100 (e.g., a sensor view portion
of a room or
work area ) that contains one or more objects. The controller 500 may be
configured to
determine occupancy data for the object based on the captured reflected light
from the
scene. Moreover, the controller 500, in some examples, issues a drive command
to the
drive system 200 based at least in part on the occupancy data to
circumnavigate obstacles
(i.e., the object in the scene). The 3-D image sensors 450 may repeatedly
capture scene
depth images for real-time decision making by the controller 500 to navigate
the robot
100 about the scene without colliding into any objects in the scene. For
example, the
speed or frequency in which the depth image data is obtained by the 3-D image
sensors
450 may be controlled by a shutter speed of the 3-D image sensors 450. In
addition, the
controller 500 may receive an event trigger (e.g., from another sensor
component of the
sensor system 400, such as proximity sensor 410, 420, notifying the controller
500 of a
nearby object or hazard. The controller 500, in response to the event trigger,
can cause
the 3-D image sensors 450 to increase a frequency at which depth images are
captured
and occupancy information is obtained.
[0094] In some implementations, the robot includes a sonar scanner 460 for
acoustic
imaging of an area surrounding the robot 100. In the examples shown in FIGS. 1
and 3,
the sonar scanner 460 is disposed on a forward portion of the base body 120.
[0095] Referring to FIGS. 1, 3B and 12, in some implementations, the robot 100
uses
the laser scanner or laser range finder 440 for redundant sensing, as well as
a rear-facing
sonar proximity sensor 41 Oj for safety, both of which are oriented parallel
to the ground
G. The robot 100 may include first and second 3-D image sensors 450a, 450b
(depth
cameras) to provide robust sensing of the environment around the robot 100.
The first 3-
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
D image sensor 450a is mounted on the torso 140 and pointed downward at a
fixed angle
to the ground G. By angling the first 3-D image sensor 450a downward, the
robot 100
receives dense sensor coverage in an area immediately forward or adjacent to
the robot
100, which is relevant for short-term travel of the robot 100 in the forward
direction. The
rear-facing sonar 410j provides object detection when the robot travels
backward. If
backward travel is typical for the robot 100, the robot 100 may include a
third 3D image
sensor 450 facing downward and backward to provide dense sensor coverage in an
area
immediately rearward or adjacent to the robot 100.
[0096] The second 3-D image sensor 450b is mounted on the head 160, which can
pan and tilt via the neck 150. The second 3-D image sensor 450b can be useful
for
remote driving since it allows a human operator to see where the robot 100 is
going. The
neck 150 enables the operator tilt and/or pan the second 3-D image sensor 450b
to see
both close and distant objects. Panning the second 3-D image sensor 450b
increases an
associated horizontal field of view. During fast travel, the robot 100 may
tilt the second
3-D image sensor 450b downward slightly to increase a total or combined field
of view
of both 3-D image sensors 450a, 450b, and to give sufficient time for the
robot 100 to
avoid an obstacle (since higher speeds generally mean less time to react to
obstacles). At
slower speeds, the robot 100 may tilt the second 3-D image sensor 450b upward
or
substantially parallel to the ground G to track a person that the robot 100 is
meant to
follow. Moreover, while driving at relatively low speeds, the robot 100 can
pan the
second 3-D image sensor 450b to increase its field of view around the robot
100. The
first 3-D image sensor 450a can stay fixed (e.g., not moved with respect to
the base 120)
when the robot is driving to expand the robot's perceptual range.
[0097] In some implementations, at least one of 3-D image sensors 450 can be a
volumetric point cloud imaging device (such as a speckle or time-of-flight
camera)
positioned on the robot 100 at a height of greater than 1 or 2 feet above the
ground (or at
a height of about 1 or 2 feet above the ground) and directed to be capable of
obtaining a
point cloud from a volume of space including a floor plane in a direction of
movement of
the robot (via the omni-directional drive system 200). In the examples shown
in FIGS. 1
and 3, the first 3-D image sensor 450a can be positioned on the base 120 at
height of
greater than 1 or 2 feet above the ground and aimed along the forward drive
direction F to
26
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
capture images (e.g., volumetric point cloud) of a volume including the floor
while
driving (e.g., for obstacle detection and obstacle avoidance). The second 3-D
image
sensor 450b is shown mounted on the head 160 (e.g., at a height greater than
about 3 or 4
feet above the ground), so as to be capable of obtaining skeletal recognition
and
definition point clouds from a volume of space adjacent the robot 100. The
controller
500 may execute skeletal/digital recognition software to analyze data of the
captured
volumetric point clouds.
[0098] Referring again to FIG. 2 and 4A-4C, the sensor system 400 may include
an
inertial measurement unit (IMU) 470 in communication with the controller 500
to
measure and monitor a moment of inertia of the robot 100 with respect to the
overall
center of gravity CGR of the robot 100.
[0099] The controller 500 may monitor any deviation in feedback from the IMU
470
from a threshold signal corresponding to normal unencumbered operation. For
example,
if the robot begins to pitch away from an upright position, it may be "clothes
lined" or
otherwise impeded, or someone may have suddenly added a heavy payload. In
these
instances, it may be necessary to take urgent action (including, but not
limited to, evasive
maneuvers, recalibration, and/or issuing an audio/visual warning) in order to
assure safe
operation of the robot 100.
[00100] Since robot 100 may operate in a human environment, it may interact
with
humans and operate in spaces designed for humans (and without regard for robot
constraints). The robot 100 can limit its drive speeds and accelerations when
in a
congested, constrained, or highly dynamic environment, such as at a cocktail
party or
busy hospital. However, the robot 100 may encounter situations where it is
safe to drive
relatively fast, as in a long empty corridor, but yet be able to decelerate
suddenly, as
when something crosses the robots' motion path.
[00101] When accelerating from a stop, the controller 500 may take into
account a
moment of inertia of the robot 100 from its overall center of gravity CGR to
prevent robot
tipping. The controller 500 may use a model of its pose, including its current
moment of
inertia. When payloads are supported, the controller 500 may measure a load
impact on
the overall center of gravity CGR and monitor movement of the robot moment of
inertia.
For example, the torso 140 and/or neck 150 may include strain gauges to
measure strain.
27
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
If this is not possible, the controller 500 may apply a test torque command to
the drive
wheels 210 and measure actual linear and angular acceleration of the robot
using the IMU
470, in order to experimentally determine safe limits.
[00102] During a sudden deceleration, a commanded load on the second and third
drive wheels 2l Ob, 210c (the rear wheels) is reduced, while the first drive
wheel 210a
(the front wheel) slips in the forward drive direction and supports the robot
100. If the
loading of the second and third drive wheels 2l Ob, 210c (the rear wheels) is
asymmetrical, the robot 100 may "yaw" which will reduce dynamic stability. The
IMU
470 (e.g., a gyro) can be used to detect this yaw and command the second and
third drive
wheels 2l Ob, 210c to reorient the robot 100.
[00103] Referring to FIGS. 3-4C and 6, in some implementations, the robot 100
includes multiple antennas. In the examples shown, the robot 100 includes a
first antenna
490a and a second antenna 490b both disposed on the base 120 (although the
antennas
may be disposed at any other part of the robot 100, such as the leg 130, the
torso 140, the
neck 150, and/or the head 160). The use of multiple antennas provide robust
signal
reception and transmission. The use of multiple antennas provides the robot
100 with
multiple-input and multiple-output, or MIMO, which is the use of multiple
antennas for a
transmitter and/or a receiver to improve communication performance. MIMO
offers
significant increases in data throughput and link range without additional
bandwidth or
transmit power. It achieves this by higher spectral efficiency (more bits per
second per
hertz of bandwidth) and link reliability or diversity (reduced fading).
Because of these
properties, MIMO is an important part of modem wireless communication
standards such
as IEEE 802.1 In (Wifi), 4G, 3GPP Long Term Evolution, WiMAX and HSPA+.
Moreover, the robot 100 can act as a Wi-Fi bridge, hub or hotspot for other
electronic
devices nearby. The mobility and use of MIMO of the robot 100 can allow the
robot to
come a relatively very reliable Wi-Fi bridge.
[00104] MIMO can be sub-divided into three main categories, pre-coding,
spatial
multiplexing or SM, and diversity coding. Pre-coding is a type of multi-stream
beam
forming and is considered to be all spatial processing that occurs at the
transmitter. In
(single-layer) beam forming, the same signal is emitted from each of the
transmit
antennas with appropriate phase (and sometimes gain) weighting such that the
signal
28
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
power is maximized at the receiver input. The benefits of beam forming are to
increase
the received signal gain, by making signals emitted from different antennas
add up
constructively, and to reduce the multipath fading effect. In the absence of
scattering,
beam forming can result in a well defined directional pattern. When the
receiver has
multiple antennas, the transmit beam forming cannot simultaneously maximize
the signal
level at all of the receive antennas, and pre-coding with multiple streams can
be used.
Pre-coding may require knowledge of channel state information (CSI) at the
transmitter.
[00105] Spatial multiplexing requires a MIMO antenna configuration. In spatial
multiplexing, a high rate signal is split into multiple lower rate streams and
each stream is
transmitted from a different transmit antenna in the same frequency channel.
If these
signals arrive at the receiver antenna array with sufficiently different
spatial signatures,
the receiver can separate these streams into (almost) parallel channels.
Spatial
multiplexing is a very powerful technique for increasing channel capacity at
higher
signal-to-noise ratios (SNR). The maximum number of spatial streams is limited
by the
lesser in the number of antennas at the transmitter or receiver. Spatial
multiplexing can
be used with or without transmit channel knowledge. Spatial multiplexing can
also be
used for simultaneous transmission to multiple receivers, known as space-
division
multiple access. By scheduling receivers with different spatial signatures,
good
separability can be assured.
[00106] Diversity Coding techniques can be used when there is no channel
knowledge
at the transmitter. In diversity methods, a single stream (unlike multiple
streams in
spatial multiplexing) is transmitted, but the signal is coded using techniques
called space-
time coding. The signal is emitted from each of the transmit antennas with
full or near
orthogonal coding. Diversity coding exploits the independent fading in the
multiple
antenna links to enhance signal diversity. Because there is no channel
knowledge, there is
no beam forming or array gain from diversity coding. Spatial multiplexing can
also be
combined with pre-coding when the channel is known at the transmitter or
combined
with diversity coding when decoding reliability is in trade-off.
[00107] In some implementations, the robot 100 includes a third antenna 490c
and/or a
fourth antenna 490d and the torso 140 and/or the head 160, respectively (see
e.g., FIG. 3).
In such instances, the controller 500 can determine an antenna arrangement
(e.g., by
29
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
moving the antennas 490a-d, as by raising or lowering the torso 140 and/or
rotating
and/or tilting the head 160) that achieves a threshold signal level for robust
communication. For example, the controller 500 can issue a command to elevate
the
third and fourth antennas 490c, 490d by raising a height of the torso 140.
Moreover, the
controller 500 can issue a command to rotate and/or the head 160 to further
orient the
fourth antenna 490d with respect to the other antennas 490a-c.
[00108] In some implementations, the robot 100 includes a mediating security
device
350 (FIG. 9), also referred to as a bridge, for allowing communication between
a web pad
310 and the controller 500 (and/or other components of the robot 100). For
example, the
bridge 350 may convert communications of the web pad 310 from a web pad
communication protocol to a robot communication protocol (e.g., Ethernet
having a
gigabit capacity). The bridge 350 may authenticate the web pad 310 and
provided
communication conversion between the web pad 310 and the controller 500. In
some
examples, the bridge 350 includes an authorization chip which
authorizes/validates any
communication traffic between the web pad 310 and the robot 100. The bridge
350 may
notify the controller 500 when it has checked an authorized a web pad 310
trying to
communicate with the robot 100. Moreover, after authorization, the bridge 350
notify the
web pad 310 of the communication authorization. The bridge 350 maybe disposed
on
the neck 150 or head (as shown in FIGS. 2 and 3) or elsewhere on the robot
100.
The Session Initiation Protocol (SIP) is an IETF-defined signaling protocol,
widely used for controlling multimedia communication sessions such as voice
and video
calls over Internet Protocol (IP). The protocol can be used for creating,
modifying and
terminating two-party (unicast) or multiparty (multicast) sessions including
one or several
media streams. The modification can involve changing addresses or ports,
inviting more
participants, and adding or deleting media streams. Other feasible application
examples
include video conferencing, streaming multimedia distribution, instant
messaging,
presence information, file transfer, etc. Voice over Internet Protocol (Voice
over IP,
VoIP) is part of a family of methodologies, communication protocols, and
transmission
technologies for delivery of voice communications and multimedia sessions over
Internet
Protocol (IP) networks, such as the Internet. Other terms frequently
encountered and
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
often used synonymously with VoIP are IP telephony, Internet telephony, voice
over
broadband (VoBB), broadband telephony, and broadband phone.
[00109] FIG. 13 provides a telephony example that includes interaction with
the bridge
350 for initiating and conducting communication through the robot 100. An SIP
of
Phone A places a call with the SIP application server. The SIP invokes a dial
function of
the VoIP, which causes a HTTP post request to be sent to a VoIP web server.
The HTTP
Post request may behave like a callback function. The SIP application server
sends a
ringing to phone A, indicating that the call has been initiated. A VoIP server
initiates a
call via a PSTN to a callback number contained in the HTTP post request. The
callback
number terminates on a SIP DID provider which is configured to route calls
back to the
SIP application server. The SIP application server matches an incoming call
with the
original call of phone A and answers both calls with an OK response. A media
session is
established between phone A and the SIP DID provider. Phone A may hear an
artificial
ring generated by the VoIP. Once the VoIP has verified that the callback leg
has been
answered, it initiates the PSTN call to the destination, such as the robot 100
(via the
bridge 350). The robot 100 answers the call and the VoIP server bridges the
media from
the SIP DID provider with the media from the robot 100.
[00110] FIGS. 14A and 14B provide schematic views of exemplary robot system
architectures 1400a, 1400b, which may include the robot 100 (or a portion
thereof, such
as the controller 500 or drive system 200), a computing device 310 (detachable
or fixedly
attached to the head 160), a cloud 1420 (for cloud computing), and a portal
1430.
[00111] The robot 100 can provide various core robot features, which may
include:
mobility (e.g., the drive system 200); a reliable, safe, secure robot
intelligence system,
such as a control system executed on the controller 500, the power source 105,
the
sensing system 400, and optional manipulation with a manipulator in
communication
with the controller 500. The control system can provide heading and speed
control, body
pose control, navigation, and core robot applications. The sensing system 400
can
provide vision (e.g., via a camera 320), depth map imaging (e.g., via a 3-D
imaging
sensor 450), collision detection, obstacle detection and obstacle avoidance,
and/or inertial
measurement (e.g., via an inertial measurement unit 470).
31
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[00112] The computing device 310 may be a tablet computer, portable electronic
device, such as phone or personal digital assistant, or a dumb tablet or
display (e.g., a
tablet that acts as a monitor for an atom-scale PC in the robot body 110). In
some
examples, the tablet computer can have a touch screen for displaying a user
interface and
receiving user inputs. The computing device 310 may execute one or more robot
applications 1410, which may include software applications (e.g., stored in
memory and
executable on a processor) for security, medicine compliance, telepresence,
behavioral
coaching, social networking, active alarm, home management, etc. The computing
device 310 may provide communication capabilities (e.g., secure wireless
connectivity
and/or cellular communication), refined application development tools, speech
recognition, and person or object recognition capabilities. The computing
device 310, in
some examples utilizes an interaction/COMS featured operating system, such as
Android
provided by Google, Inc., iPad OS provided by Apple, Inc., other smart phone
operating
systems, or government systems, such as RS S A2.
[00113] The cloud 1420 provides cloud computing and/or cloud storage
capabilities.
Cloud computing may provide Internet-based computing, whereby shared servers
provide
resources, software, and data to computers and other devices on demand. For
example,
the cloud 1420 may be a cloud computing service that includes at least one
server
computing device, which may include a service abstraction layer and a
hypertext transfer
protocol wrapper over a server virtual machine instantiated thereon. The
server
computing device may be configured to parse HTTP requests and send HTTP
responses.
Cloud computing may be a technology that uses the Internet and central remote
servers to
maintain data and applications. Cloud computing can allow users to access and
use
applications 1410 without installation and access personal files at any
computer with
internet access. Cloud computing allows for relatively more efficient
computing by
centralizing storage, memory, processing and bandwidth. The cloud 1420 can
provide
scalable, on-demand computing power, storage, and bandwidth, while reducing
robot
hardware requirements (e.g., by freeing up CPU and memory usage). Robot
connectivity
to the cloud 1420 allows automatic data gathering of robot operation and usage
histories
without requiring the robot 100 to return to a base station. Moreover,
continuous data
32
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
collection over time can yields a wealth of data that can be mined for
marketing, product
development, and support.
[00114] Cloud storage 1422 can be a model of networked computer data storage
where
data is stored on multiple virtual servers, generally hosted by third parties.
By providing
communication between the robot 100 and the cloudy 1420, information gathered
by the
robot 100 can be securely viewed by authorized users via a web based
information portal.
[00115] The portal 1430 may be a web-based user portal for gathering and/or
providing information, such as personal information, home status information,
anger
robot status information. Information can be integrated with third-party
information to
provide additional functionality and resources to the user and/or the robot
100. The robot
system architecture 1400 can facilitate proactive data collection. For
example,
applications 1410 executed on the computing device 310 may collect data and
report on
actions performed by the robot 100 and/or a person or environment viewed by
the robot
100 (using the sensing system 400). This data can be a unique property of the
robot 100.
[00116] In some examples, the portal 1430 is a personal portal web site on the
World
Wide Web. The portal 1430 may provide personalized capabilities and a pathway
to
other content. The portal 1430 may use distributed applications, different
numbers and
types of middleware and hardware, to provide services from a number of
different
sources. In addition, business portals 1430 may share collaboration in
workplaces and
provide content usable on multiple platforms such as personal computers,
personal digital
assistants (PDAs), and cell phones/mobile phones. Information, news, and
updates are
examples of content that may be delivered through the portal 1430. Personal
portals 1430
can be related to any specific topic such as providing friend information on a
social
network or providing links to outside content that may help others.
[00117] FIG. 14C is a schematic view of an exemplary mobile human interface
robot
system architecture 1400c. In the example shown, application developers 1402
can
access and use application development tools 1440 to produce applications 1410
executable on the web pad 310 or a computing device 1404 (e.g., desktop
computer,
tablet computer, mobile device, etc.) in communication with the cloud 1420.
Exemplary
application development tools 1440 may include, but are not limited to, an
integrated
development environment 1442, a software development kit (SDK) libraries 1444,
33
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
development or SDK tools 1446 (e.g., modules of software code, a simulator, a
cloud
usage monitor and service configurator, and a cloud services extension
uploader/deployer), and/or source code 1448. The SDK libraries 1444 may allow
enterprise developers 1402 to leverage mapping, navigation, scheduling and
conferencing
technologies of the robot 100 in the applications 1410. Exemplary applications
1410 may
include, but are not limited to, a map builder 1410a, a mapping and navigation
application 141 Ob, a video conferencing application 141 Oc, a scheduling
application
1410d, and a usage application 1410e. The applications 1410 may be stored on
one or
more applications servers 1450 (e.g., cloud storage 1422) in the cloud 1420
and can be
accessed through a cloud services application programming interface (API). The
cloud
1420 may include one or more databases 1460 and a simulator 1470. A web
services API
can allow communication between the robot 100 and the cloud 1420 (e.g., and
the
application server(s) 1450, database(s) 1460, and the simulator 1470).
External systems
1480 may interact with the cloud 1420 as well, for example, to access the
applications
1410.
[00118] In some examples, the map builder application 1410a can build a map of
an
environment around the robot 100 by linking together pictures or video
captured by the
camera 320 or 3-D imaging sensor 450 using reference coordinates, as provided
by
odometry, a global positioning system, and/or way-point navigation. The map
may
provide an indoor or outside street or path view of the environment. For malls
or
shopping centers, the map can provide a path tour through-out the mall with
each store
marked as a reference location with additional linked images or video and/or
promotional
information. The map and/or constituent images or video can be stored in the
database
1460.
[00119] The applications 1410 may seamlessly communicate with the cloud
services,
which may be customized and extended based on the needs of each user entity.
Enterprise developers 1402 may upload cloud-side extensions to the cloud 1420
that fetch
data from external proprietary systems for use by an application 1410. The
simulator
1470 allows the developers 1402 to build enterprise-scale applications without
the robot
100 or associated robot hardware. Users may use the SDK tools 1446 (e.g.,
usage
monitor and service configurator) to add or disable cloud services.
34
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[00120] In some examples, the mapping and navigation application 141 Ob (FIG.
14C)
provides teleoperation functionality. For example, the user can drive the
robot 100 using
video waypoint driving (e.g., using one or more of the cameras or imaging
sensors 320,
450). The user may alter a height HL of the leg 130 to raise/lower the height
HT of the
torso 140 (FIG. 2) to alter a view field of one of the imaging sensors 450
and/or pan
and/or tilt the robot head 160 to alter a view field of a supported camera 320
or imaging
sensor 450, 450b (see e.g., FIGS 11 and 12). Moreover, the user can rotate the
robot
about its Z-axis using the drive system 200 to gain other fields of view of
the cameras or
imaging sensors 320, 450. In some examples, the mapping and navigation
application
1410b allows the user to switch between multiple layout maps 1810 (e.g., for
different
environments or different robots 100) and/or manage multiple robots 100 on one
layout
map 1810. The mapping and navigation application 141 Ob may communicate with
the
cloud services API to enforce policies on proper robot usage set forth by
owners or
organizations of the robots 100.
[00121] Referring again to FIG. 14C, in some implementations, the video
conferencing application 1410c allows a user to initiate and/or participate in
a video
conferencing session with other users. In some examples, the video
conferencing
application 1410c allows a user to initiate and/or participate in a video
conferencing
session with a user of the robot 100, a remote user on a computing device
connected to
the cloud 1420 and/or another remote user connected to the Internet using a
mobile
handheld device. The video conferencing application 1410c may provide an
electronic
whiteboard for sharing information, an image viewer, and/or a PDF viewer.
[00122] The scheduling application 1410d allows users to schedule usage of one
or
more robots 100. When there are fewer robots 100 than the people who want to
use
them, the robots 100 become scarce resources and scheduling may be needed.
Scheduling resolves conflicts in resource allocations and enables higher
resource
utilization. The scheduling application 1410d can be robot-centric and may
integrate
with third party calendaring systems, such as Microsoft Outlook or Google
Calendar. In
some examples, the scheduling application 1410d communicates with the cloud
1420
through one or more cloud services to dispatch robots 100 at pre-scheduled
times. The
scheduling application 1410d may integrate time-related data (e.g.,
maintenance
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
schedule, etc.) with other robot data (e.g., robot locations, health status,
etc.) to allow
selection of a robot 100 by the cloud services for missions specified by the
user.
[00123] In one scenario, a doctor may access the scheduling application 1410d
on a
computing device (e.g., a portable tablet computer or hand held device) in
communication with the cloud 1420 for scheduling rounds at a remote hospital
later in
the week. The scheduling application 1410d can schedule robots 100 in a
similar manner
to allocating a conference room on a electronic calendar. The cloud services
manage the
schedules. If in the middle of the night, the doctor gets a call that a
critical patient at a
remote hospital needs to be seen, the doctor can request a robot 100 using the
scheduling
application 1410d and/or send a robot 100 to a patient room using the mapping
and
navigation application 141 Ob. The doctor may access medical records on his
computing
device (e.g., by accessing the cloud storage 1422) and video or imagery of the
patient
using the video conferencing application 141 Oc. The cloud services may
integrate with
robot management, an electronic health record systems and medical imaging
systems.
The doctor may control movement of the robot 100 remotely to interact with the
patient.
If the patent speaks only Portuguese, the video conferencing application 141
Oc may
automatically translate languages or a 3rd party translator may join the video
conference
using another computing device in communication with the cloud 1420 (e.g., via
the
Internet). The translation services can be requested, fulfilled, recorded, and
billed using
the cloud services.
[00124] The usage / statistics application 1410e can be a general-purpose
application
for users to monitor robot usage, produce robot usage reports, and/or manage a
fleet of
robots 100. This application 141 Oe may also provide general operating and
troubleshooting information for the robot 100. In some examples, the usage /
statistics
application 1410e allows the user to add/disable services associated with use
of the robot
100, register for use of one or more simulators 1470, modify usage policies on
the robot,
etc.
[00125] In another scenario, a business may have a fleet of robots 100 for at
least one
telepresence application. A location manager may monitor a status of one or
more robots
100 (e.g., location, usage and maintenance schedules, battery info, location
history, etc.)
using the usage / statistics application 1410e executing on a computing device
in
36
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
communication with the cloud 1420 (e.g., via the Internet). In some examples,
the
location manager can assist a user with a robot issue by sharing a user
session. The
location manager can issue commands to any of the robots 100 using an
application 1410
to navigate the corresponding robot 100, speak through the robot 100 (i.e.,
telepresence),
enter into a power-saving mode (e.g., reduce functionality), find a charger,
etc. The
location manager or a user can use applications 1410 to manage users,
security, layout
maps 1810, video view fields, add/remove robots to/from the fleet, and more.
Remote
operators of the robot 100 can schedule/reschedule/cancel a robot appointment
(e.g.,
using the scheduling application 141 Od) and attend a training course using a
simulated
robot that roams a simulated space (e.g., using the simulator 1470 executing
on a cloud
server).
[00126] The SDK libraries 1444 may include one or more source code libraries
for use
by developers 1402 of applications 1410. For example, a visual component
library can
provide graphical user interface or visual components having interfaces for
accessing
encapsulated functionality. Exemplary visual components include code classes
for
drawing layout map tiles and robots, video conferencing, viewing images and
documents,
and/or displaying calendars or schedules. A robot communication library (e.g.,
a web
services API) can provide a RESTful (Representational State Transfer), JSON
(JavaScript Object Notation)-based API for communicating directly with the
robot 100.
The robot communication library can offer Objective-C binding (e.g., for iOS
development) and Java binding (e.g., for Android development). These object-
oriented
APIs allow applications 1410 to communicate with the robot 100, while
encapsulating
from the developers 1402 underlying data transfer protocol(s) of the robot
100. A person
following routine of the robot communication library may return a video screen
coordinate corresponding to a person tracked by the robot 100. A facial
recognition
routine of the , robot communication library may return a coordinate of a face
on a
camera view of the camera 320 and optionally the name of the recognized
tracked person.
Table 1 provides an exemplary list of robot communication services.
Service Description
Database Service List all available map databases on the robot 100.
Create Database Service Create a robot map database.
37
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
Delete Database Service Delete a robot map database.
Map List Service Return a list of maps from a map database on the robot 100.
Map Service Return a specific robot map.
Create Map Service Create a robot map in a database.
Delete Map Service Delete a robot map from a database.
Create Tag Service Create a tag for a map in a database (e.g., by providing x,
y,
and z coordinates of the tag position an orientation angle of
the robot, in radians, and a brief description of the tag).
Delete Tag Service Delete a tag for a map in a database.
List Tags Service List tags for a specified map in a specified database.
Cameras Service List available cameras 320, 450 on the robot 100.
Camera Image Service Take a snapshot from a camera 320, 450 on the robot 100.
Robot Position Service Return a current position of the robot 100. The
position can
be returned as:
= x -- a distance along an x-axis from an origin (in meters).
= y -- a distance along a y-axis from an origin (in meters).
= theta -- an angle from the x-axis, measured
counterclockwise in radians.
Robot Destination Sets a destination location of the robot 100 and commands
the
Service robot 100 to begin moving to that location.
Drive-To-Tag Service Drives the robot 100 to a tagged destination in a map.
Stop Robot Service Commands the robot 100 to stop moving.
Robot Info Service Provide basic robot information (e.g., returns a dictionary
of
the robot information).
Table 1
[00127] A cloud services communication library may include APIs that allow
applications 1410 to communicate with the cloud 1420 (e.g., with cloud storage
1422,
applications servers 1450, databases 1460 and the simulator 1470) and/or
robots 100 in
s communication with the cloud 1420. The cloud services communication library
can be
provided in both Objective-C and Java bindings. Examples of cloud services
APIs
include a navigation API (e.g., to retrieve positions, set destinations,
etc.), a map storage
38
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
and retrieval PAI, a camera feed API, a teleoperation API, a usage statistics
API, and
others.
[00128] A cloud services extensibility interface may allow the cloud services
to
interact with web services from external sources. For example, the cloud
services may
define a set of extension interfaces that allow enterprise developers 1403 to
implement
interfaces for external proprietary systems. The extensions can be uploaded
and
deployed to the cloud infrastructure. In some examples, the cloud services can
adopt
standard extensibility interface defined by various industry consortiums.
[00129] The simulator 1470 may allow debugging and testing of applications
1410
without connectivity to the robot 100. The simulator 1470 can model or
simulate
operation of the robot 100 without actually communicating with the robot 100
(e.g., for
path planning and accessing map databases). For executing simulations, in some
implementations, the simulator 1470 produces a map database (e.g., from a
layout map
1810) without using the robot 100. This may involve image processing (e.g.,
edge
detection) so that features (like walls, corners, columns, etc) are
automatically identified.
The simulator 1470 can use the map database to simulate path planning in an
environment dictated by the layout map 1810.
[00130] A cloud services extension uploader/deployer may allow users upload
extensions to the cloud 1420, connect to external third party user
authentication systems,
access external databases or storage (e.g., patient info for pre-consult and
post-consult),
access images for illustration in video conferencing sessions, etc. The cloud
service
extension interface may allow integration of proprietary systems with the
cloud 1420.
[00131] Referring to FIG. 15, in some implementations, the controller 500
executes a
control system 510, which includes a control arbitration system 510a and a
behavior
system 5l Ob in communication with each other. The control arbitration system
510a
allows applications 520 to be dynamically added and removed from the control
system
510, and facilitates allowing applications 520 to each control the robot 100
without
needing to know about any other applications 520. In other words, the control
arbitration
system 510a provides a simple prioritized control mechanism between
applications 520
and resources 530 of the robot 100. The resources 530 may include the drive
system 200,
39
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
the sensor system 400, and/or any payloads or controllable devices in
communication
with the controller 500.
[00132] The applications 520 can be stored in memory of or communicated to the
robot 100, to run concurrently on (e.g., a processor) and simultaneously
control the robot
100. The applications 520 may access behaviors 600 of the behavior system
510b. The
independently deployed applications 520 are combined dynamically at runtime
and to
share robot resources 530 (e.g., drive system 200, arm(s), head(s), etc.) of
the robot 100.
A low-level policy is implemented for dynamically sharing the robot resources
530
among the applications 520 at run-time. The policy determines which
application 520
has control of the robot resources 530 required by that application 520 (e.g.
a priority
hierarchy among the applications 520). Applications 520 can start and stop
dynamically
and run completely independently of each other. The control system 510 also
allows for
complex behaviors 600 which can be combined together to assist each other.
[00133] The control arbitration system 510a includes one or more resource
controllers
540, a robot manager 550, and one or more control arbiters 560. These
components do
not need to be in a common process or computer, and do not need to be started
in any
particular order. The resource controller 540 component provides an interface
to the
control arbitration system 51 Oa for applications 520. There is an instance of
this
component for every application 520. The resource controller 540 abstracts and
encapsulates away the complexities of authentication, distributed resource
control
arbiters, command buffering, and the like. The robot manager 550 coordinates
the
prioritization of applications 520, by controlling which application 520 has
exclusive
control of any of the robot resources 530 at any particular time. Since this
is the central
coordinator of information, there is only one instance of the robot manager
550 per robot.
The robot manager 550 implements a priority policy, which has a linear
prioritized order
of the resource controllers 540, and keeps track of the resource control
arbiters 560 that
provide hardware control. The control arbiter 560 receives the commands from
every
application 520 and generates a single command based on the applications'
priorities and
publishes it for its associated resources 530. The control arbiter 560 also
receives state
feedback from its associated resources 530 and sends it back up to the
applications 520.
The robot resources 530 may be a network of functional modules (e.g.
actuators, drive
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
systems, and groups thereof) with one or more hardware controllers. The
commands of
the control arbiter 560 are specific to the resource 530 to carry out specific
actions.
[00134] A dynamics model 570 executable on the controller 500 can be
configured to
compute the center for gravity (CG), moments of inertia, and cross products of
inertia of
various portions of the robot 100 for the assessing a current robot state. The
dynamics
model 570 may also model the shapes, weight, and/or moments of inertia of
these
components. In some examples, the dynamics model 570 communicates with the
inertial
moment unit 470 (IMU) or portions of one (e.g., accelerometers and/or gyros)
disposed
on the robot 100 and in communication with the controller 500 for calculating
the various
center of gravities of the robot 100. The dynamics model 570 can be used by
the
controller 500, along with other programs 520 or behaviors 600 to determine
operating
envelopes of the robot 100 and its components.
[00135] Each application 520 has an action selection engine 580 and a resource
controller 540, one or more behaviors 600 connected to the action selection
engine 580,
and one or more action models 590 connected to action selection engine 580.
The
behavior system 5l Ob provides predictive modeling and allows the behaviors
600 to
collaboratively decide on the robot's actions by evaluating possible outcomes
of robot
actions. In some examples, a behavior 600 is a plug-in component that provides
a
hierarchical, state-full evaluation function that couples sensory feedback
from multiple
sources with a-priori limits and information into evaluation feedback on the
allowable
actions of the robot. Since the behaviors 600 are pluggable into the
application 520 (e.g.,
residing inside or outside of the application 520), they can be removed and
added without
having to modify the application 520 or any other part of the control system
510. Each
behavior 600 is a standalone policy. To make behaviors 600 more powerful, it
is possible
to attach the output of multiple behaviors 600 together into the input of
another so that
you can have complex combination functions. The behaviors 600 are intended to
implement manageable portions of the total cognizance of the robot 100.
[00136] The action selection engine 580 is the coordinating element of the
control
system 510 and runs a fast, optimized action selection cycle
(prediction/correction cycle)
searching for the best action given the inputs of all the behaviors 600. The
action
selection engine 580 has three phases: nomination, action selection search,
and
41
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
completion. In the nomination phase, each behavior 600 is notified that the
action
selection cycle has started and is provided with the cycle start time, the
current state, and
limits of the robot actuator space. Based on internal policy or external
input, each
behavior 600 decides whether or not it wants to participate in this action
selection cycle.
During this phase, a list of active behavior primitives is generated whose
input will affect
the selection of the commands to be executed on the robot 100.
[00137] In the action selection search phase, the action selection engine 580
generates
feasible outcomes from the space of available actions, also referred to as the
action space.
The action selection engine 580 uses the action models 590 to provide a pool
of feasible
commands (within limits) and corresponding outcomes as a result of simulating
the
action of each command at different time steps with a time horizon in the
future. The
action selection engine 580 calculates a preferred outcome, based on the
outcome
evaluations of the behaviors 600, and sends the corresponding command to the
control
arbitration system 51 Oa and notifies the action model 590 of the chosen
command as
feedback.
[00138] In the completion phase, the commands that correspond to a
collaborative best
scored outcome are combined together as an overall command, which is presented
to the
resource controller 540 for execution on the robot resources 530. The best
outcome is
provided as feedback to the active behaviors 600, to be used in future
evaluation cycles.
[00139] Received sensor signals from the sensor system 400 can cause
interactions
with one or more behaviors 600 to execute actions. For example, using the
control
system 510, the controller 500 selects an action (or move command) for each
robotic
component (e.g., motor or actuator) from a corresponding action space (e.g., a
collection
of possible actions or moves for that particular component) to effectuate a
coordinated
move of each robotic component in an efficient manner that avoids collisions
with itself
and any objects about the robot 100, which the robot 100 is aware of. The
controller 500
can issue a coordinated command over robot network, such as an EtherlO
network, as
described in U.S. Serial No. 61/305,069, filed February 16, 2010, the entire
contents of
which are hereby incorporated by reference.
42
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[00140] The control system 510 may provide adaptive speed/acceleration of the
drive
system 200 (e.g., via one or more behaviors 600) in order to maximize
stability of the
robot 100 in different configurations/positions as the robot 100 maneuvers
about an area.
[00141] In some implementations, the controller 500 issues commands to the
drive
system 200 that propels the robot 100 according to a heading setting and a
speed setting.
One or behaviors 600 may use signals received from the sensor system 400 to
evaluate
predicted outcomes of feasible commands, one of which may be elected for
execution
(alone or in combination with other commands as an overall robot command) to
deal with
obstacles. For example, signals from the proximity sensors 410 may cause the
control
system 510 to change the commanded speed or heading of the robot 100. For
instance, a
signal from a proximity sensor 410 due to a nearby wall may result in the
control system
510 issuing a command to slow down. In another instance, a collision signal
from the
contact sensor(s) due to an encounter with a chair may cause the control
system 510 to
issue a command to change heading. In other instances, the speed setting of
the robot
100 may not be reduced in response to the contact sensor; and/or the heading
setting of
the robot 100 may not be altered in response to the proximity sensor 410.
[00142] The behavior system 5l Ob may include a speed behavior 600a (e.g., a
behavioral routine executable on a processor) configured to adjust the speed
setting of the
robot 100 and a heading behavior 600b configured to alter the heading setting
of the
robot 100. The speed and heading behaviors 600a, 600b may be configured to
execute
concurrently and mutually independently. For example, the speed behavior 600a
may be
configured to poll one of the sensors (e.g., the set(s) of proximity sensors
410, 420), and
the heading behavior 600b may be configured to poll another sensor (e.g., the
kinetic
bump sensor).
[00143] Referring to FIGS. 15, 16A and 16B, the behavior system 5l Ob may
include
an augmented reality behavior 600c that becomes active during telepresence
operation of
the robot 100. For example, when the robot 100 engages in video conferencing
or other
forms of telepresence between a local user adjacent the robot 100 and one or
more remote
users in communication with the robot (e.g., via computing device 1404
communicating
with the robot 100 via the Internet or cloud 1420) the augmented reality
behavior 600c
may become enabled. In some examples, the augmented reality behavior 600c
becomes
43
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
active automatically once enabled or a user actives the enabled behavior 600c.
The
augmented reality behavior 600c can be implemented as an application 1410
executed on
a computing device 310, 1404 (local and/or remote). For example, the web pad
310 on
the robot 100 may execute an augmented reality application 1410 and a remote
computing device 1404 in communication with the robot 100 (e.g., via the
Internet or
cloud 1420) can execute the augmented reality application 1410 as well. The
augmented
reality behavior 600c can interact with the cloud 1420 (e.g., with cloud
storage 1422,
applications servers 1450, databases 1460) to store and/or retrieve
information for shape
detection/identification, image labeling, person analysis, etc.
[00144] In some implementations, the augmented reality behavior 600c detects
shapes
1610 within an image 1602 (e.g., people or object recognition) and overlays a
label 1620
on the image 1602 on top of or adjacent the recognized shape 1610. The labels
1620 may
float on the image 1602, such that a user can move or otherwise manipulate the
labels
1620 (e.g., using a pointing device or touch gesture on a touch screen).
[00145] In the example shown in FIG. 16A, the augmented reality behavior 600c
analyzes an image 1602 displayed electronically on the display 312 of the web
pad 310
and detects a doctor 1610a as one shape 1610 and a patient 161 Ob as another
shape 1610.
After identifying the two shapes 1610a, 161 Ob, the augmented reality behavior
600c can
discern shapes 1610 (e.g., by matching the identified shapes 1610a, 161Ob with
known
shapes 1610 in a database 1460 in the cloud 1420), and then apply respective
labels 1620
to the identified shapes 1610a, 161 Ob on the image 1602. The labels 1620 may
include
information specific to the identified shape 1610. For example, for a person,
the
information may include a name, title, occupation, address, business address,
email
address, web-page address, user notes, etc. For a non-person object, such as a
store, the
information may include an object name (e.g., store name), location, business
address,
email address, web-page address, user notes, etc. A user can select
information on the
label 1620 (e.g., as HTML links (hyper text markup language)) to navigate to
additional
information or actions. For example, selecting the web-page opens an Internet
browser
with an addresses of the selected web page or selection of an email address
opens an
email program for sending an email. The label information may be linked to
information
stored on the cloud 1420 (e.g., in a cloud database 1460).
44
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[00146] Referring to FIG. 17, the augmented reality behavior 600c may use
image data
captured from a camera 320 and/or 3-D image sensor 450 on the robot 100 for
displaying
the image 1602. The control system 510 can identify a person 1714 (e.g., via
pattern or
image recognition), so as to label and/or follow that person 1714 using a
person detection
routine. If the robot 100 encounters another person 1714, as the first person
1714 turns
around a corner, for example, the robot 100 can discern that the second person
1714 is
not the first person 1714, label the second person 1714 and optionally
continue following
the first person 1714. In some implementations, the 3-D image sensor 450
provides 3-D
image data 1702 (e.g., a 2-d array of pixels, each pixel containing depth
information (e.g.,
distance to the camera)) to a segmentor 1704 for segmentation into objects or
blobs 1706.
For example, the pixels are grouped into larger objects based on their
proximity to
neighboring pixels. Each of these objects (or blobs) is then received by a
size filter 1708
for further analysis. The size filter 1708 processes the objects or blobs 1706
into right
sized objects or blobs 1710, for example, by rejecting objects that are too
small (e.g., less
than about 3 feet in height) or too large to be a person (e.g., greater than
about 8 feet in
height). A shape filter 1712 receipts the right sized objects or blobs 1710
and eliminates
objects that do not satisfy a specific shape. The shape filter 1712 may look
at an
expected width of where a midpoint of a head is expected to be using the angle-
of-view
of the camera 320 or image sensor 450 and the known distance to the object.
The shape
filter 1712 processes or renders the right sized objects or blobs 1710 into
person data
1714 (e.g., images or data representative thereof). In some examples, the
robot 100 can
detect and track multiple persons 1714 by maintaining a unique identifier for
each person
1714 detected.
[00147] The augmented reality behavior 600c may use the person detection
routine to
execute biometric analysis of the detected person 1714. Moreover, the
augmented reality
behavior 600c may use voice signals received from the one or more microphones
330 on
the robot 100 (or on the web pad 310 or remote computing device 1404) for
analyzing,
altering, and/or performing voice recognition on the voice signals. For
example, the
augmented reality behavior 600c may increase or decrease the volume so as to
make a
particular person sound relatively louder than others or surrounding sound
(e.g., to
discern that person's speech over others or background noise). In additional
examples,
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
the augmented reality behavior 600c can analyze the voice signals to determine
if the
person is nervous or distressed. For example, a security robot may detect and
identify a
person 1610, 1714 and then determine if that person is nervous, distressed, or
agitated by
analyzing a posture of the person and/or a tone of that person's voice. In
some
implementations, the augmented reality behavior 600c can translate the voice
signals into
other languages, perform transcription, record the voice signals as an audio
file (and
optionally store on the cloud 1420), etc.
[00148] The augmented reality behavior 600c can provide features that assist
with or
enhance telepresence or video conferencing. For example, a user may add labels
or
mark-ups (pictures and/or sounds) over an identified shape 1610 or person
1714, such as
personal notes, shared notes, sketches, drawings, active or real-time
sketches, humor
items, etc. The augmented reality behavior 600c may provide audience attention
metering by tracking gestures, postures, responsiveness, etc. of local and/or
remote users.
For meetings, the augmented reality behavior 600c can provide auto prompt
attendee
callouts and/or automated gesturing.
[00149] Referring to FIGS. 18A and 18B, in some circumstances, the robot 100
receives an occupancy map 1800 of objects 12 in a scene 10 and/or work area 5,
or the
robot controller 500 produces (and may update) the occupancy map 1800 based on
image
data and/or image depth data received from an imaging sensor 450 (e.g., the
second 3-D
image sensor 450b) over time. In addition to localization of the robot 100 in
the scene 10
(e.g., the environment about the robot 100), the robot 100 may travel to other
points in a
connected space (e.g., the work area 5) using the sensor system 400. The robot
100 may
include a short range type of imaging sensor 450a (e.g., mounted on the
underside of the
torso 140, as shown in FIGS. 1 and 3) for mapping a nearby area about the
robot 110 and
discerning relatively close objects 12, and a long range type of imaging
sensor 450b (e.g.,
mounted on the head 160, as shown in FIGS. 1 and 3) for mapping a relatively
larger area
about the robot 100 and discerning relatively far away objects 12. The robot
100 can use
the occupancy map 1800 to identify known objects 12 in the scene 10 as well as
occlusions 16 (e.g., where an object 12 should or should not be, but cannot be
confirmed
from the current vantage point). The robot 100 can register an occlusion 16 or
new
object 12 in the scene 10 and attempt to circumnavigate the occlusion 16 or
new object
46
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
12 to verify the location of new object 12 or any objects 12 in the occlusion
16.
Moreover, using the occupancy map 1800, the robot 100 can determine and track
movement of an object 12 in the scene 10. For example, the imaging sensor 450,
450a,
450b may detect a new position 12' of the object 12 in the scene 10 while not
detecting a
mapped position of the object 12 in the scene 10. The robot 100 can register
the position
of the old object 12 as an occlusion 16 and try to circumnavigate the
occlusion 16 to
verify the location of the object 12. The robot 100 may compare new image
depth data
with previous image depth data (e.g., the map 1800) and assign a confidence
level of the
location of the object 12 in the scene 10. The location confidence level of
objects 12
within the scene 10 can time out after a threshold period of time. The sensor
system 400
can update location confidence levels of each object 12 after each imaging
cycle of the
sensor system 400. In some examples, a detected new occlusion 16 (e.g., a
missing
object 12 from the occupancy map 1800) within an occlusion detection period
(e.g., less
than ten seconds), may signify a "live" object 12 (e.g., a moving object 12)
in the scene
10.
[00150] In some implementations, a second object 12b of interest, located
behind a
detected first object 12a in the scene 10, may be initially undetected as an
occlusion 16 in
the scene 10. An occlusion 16 can be area in the scene 10 that is not readily
detectable or
viewable by the imaging sensor 450, 450a, 450b. In the example shown, the
sensor
system 400 (e.g., or a portion thereof, such as imaging sensor 450, 450a,
450b) of the
robot 100 has a field of view 452 with a viewing angle 6v (which can be any
angle
between 0 degrees and 360 degrees) to view the scene 10. In some examples, the
imaging sensor 170 includes omni-directional optics for a 360 degree viewing
angle 6v;
while in other examples, the imaging sensor 450, 450a, 450b has a viewing
angle 6v of
less than 360 degrees (e.g., between about 45 degrees and 180 degrees). In
examples,
where the viewing angle 6v is less than 360 degrees, the imaging sensor 450,
450a, 450b
(or components thereof) may rotate with respect to the robot body 110 to
achieve a
viewing angle 6v of 360 degrees. In some implementations, the imaging sensor
450,
450a, 450b or portions thereof, can move with respect to the robot body 110
and/or drive
system 120. Moreover, in order to detect the second object 12b, the robot 100
may move
the imaging sensor 450, 450a, 450b by driving about the scene 10 in one or
more
47
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
directions (e.g., by translating and/or rotating on the work surface 5) to
obtain a vantage
point that allows detection of the second object 10b. Robot movement or
independent
movement of the imaging sensor 450, 450a, 450b, or portions thereof, may
resolve
monocular difficulties as well.
[00151] A confidence level may be assigned to detected locations or tracked
movements of objects 12 in the working area 5. For example, upon producing or
updating the occupancy map 1800, the controller 500 may assign a confidence
level for
each object 12 on the map 1800. The confidence level can be directly
proportional to a
probability that the object 12 actually located in the working area 5 as
indicated on the
map 1800. The confidence level may be determined by a number of factors, such
as the
number and type of sensors used to detect the object 12. For example, the
contact sensor
430 may provide the highest level of confidence, as the contact sensor 430
senses actual
contact with the object 12 by the robot 100. The imaging sensor 450 may
provide a
different level of confidence, which may be higher than the proximity sensor
430. Data
received from more than one sensor of the sensor system 400 can be aggregated
or
accumulated for providing a relatively higher level of confidence over any
single sensor.
[00152] Odometry is the use of data from the movement of actuators to estimate
change in position over time (distance traveled). In some examples, an encoder
is
disposed on the drive system 200 for measuring wheel revolutions, therefore a
distance
traveled by the robot 100. The controller 500 may use odometry in assessing a
confidence level for an object location. In some implementations, the sensor
system 400
includes an odometer and/or an angular rate sensor (e.g., gyroscope or the IMU
470) for
sensing a distance traveled by the robot 100. A gyroscope is a device for
measuring or
maintaining orientation, based on the principles of conservation of angular
momentum.
The controller 500 may use odometry and/or gyro signals received from the
odometer
and/or angular rate sensor, respectively, to determine a location of the robot
100 in a
working area 5 and/or on an occupancy map 1800. In some examples, the
controller 500
uses dead reckoning. Dead reckoning is the process of estimating a current
position
based upon a previously determined position, and advancing that position based
upon
known or estimated speeds over elapsed time, and course. By knowing a robot
location
in the working area 5 (e.g., via odometry, gyroscope, etc.) as well as a
sensed location of
48
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
one or more objects 12 in the working area 5 (via the sensor system 400), the
controller
500 can assess a relatively higher confidence level of a location or movement
of an object
12 on the occupancy map 1800 and in the working area 5 (versus without the use
of
odometry or a gyroscope).
[00153] Odometry based on wheel motion can be electrically noisy. The
controller
500 may receive image data from the imaging sensor 450 of the environment or
scene 10
about the robot 100 for computing robot motion, independently of wheel based
odometry
of the drive system 200, through visual odometry. Visual odometry may entail
using
optical flow to determine the motion of the imaging sensor 450. The controller
500 can
use the calculated motion based on imaging data of the imaging sensor 450 for
correcting
any errors in the wheel based odometry, thus allowing for improved mapping and
motion
control. Visual odometry may have limitations with low-texture or low-light
scenes 10,
if the imaging sensor 450 cannot track features within the captured image(s).
[00154] Other details and features on odometry and imaging systems, which may
combinable with those described herein, can be found in U.S. Patent 7,158,317
(describing a "depth-of field" imaging system), and U.S. Patent 7,115,849
(describing
wavefront coding interference contrast imaging systems), the contents of which
are
hereby incorporated by reference in their entireties.
[00155] Various implementations of the systems and techniques described here
can be
realized in digital electronic circuitry, integrated circuitry, specially
designed ASICs
(application specific integrated circuits), computer hardware, firmware,
software, and/or
combinations thereof. These various implementations can include implementation
in one
or more computer programs that are executable and/or interpretable on a
programmable
system including at least one programmable processor, which may be special or
general
purpose, coupled to receive data and instructions from, and to transmit data
and
instructions to, a storage system, at least one input device, and at least one
output device.
[00156] These computer programs (also known as programs, software, software
applications or code) include machine instructions for a programmable
processor, and can
be implemented in a high-level procedural and/or object-oriented programming
language,
and/or in assembly/machine language. As used herein, the terms "machine-
readable
medium" and "computer-readable medium" refer to any computer program product,
49
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
apparatus and/or device (e.g., magnetic discs, optical disks, memory,
Programmable
Logic Devices (PLD5)) used to provide machine instructions and/or data to a
programmable processor, including a machine-readable medium that receives
machine
instructions as a machine-readable signal. The term "machine-readable signal"
refers to
any signal used to provide machine instructions and/or data to a programmable
processor.
[00157] Implementations of the subject matter and the functional operations
described
in this specification can be implemented in digital electronic circuitry, or
in computer
software, firmware, or hardware, including the structures disclosed in this
specification
and their structural equivalents, or in combinations of one or more of them.
Embodiments of the subject matter described in this specification can be
implemented as
one or more computer program products, i.e., one or more modules of computer
program
instructions encoded on a computer readable medium for execution by, or to
control the
operation of, data processing apparatus. The computer readable medium can be a
machine-readable storage device, a machine-readable storage substrate, a
memory device,
a composition of matter effecting a machine-readable propagated signal, or a
combination
of one or more of them. The term "data processing apparatus" encompasses all
apparatus, devices, and machines for processing data, including by way of
example a
programmable processor, a computer, or multiple processors or computers. The
apparatus can include, in addition to hardware, code that creates an execution
environment for the computer program in question, e.g., code that constitutes
processor
firmware, a protocol stack, a database management system, an operating system,
or a
combination of one or more of them. A propagated signal is an artificially
generated
signal, e.g., a machine-generated electrical, optical, or electromagnetic
signal, that is
generated to encode information for transmission to suitable receiver
apparatus.
[00158] A computer program (also known as a program, software, software
application, script, or code) can be written in any form of programming
language,
including compiled or interpreted languages, and it can be deployed in any
form,
including as a stand alone program or as a module, component, subroutine, or
other unit
suitable for use in a computing environment. A computer program does not
necessarily
correspond to a file in a file system. A program can be stored in a portion of
a file that
holds other programs or data (e.g., one or more scripts stored in a markup
language
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
document), in a single file dedicated to the program in question, or in
multiple
coordinated files (e.g., files that store one or more modules, sub programs,
or portions of
code). A computer program can be deployed to be executed on one computer or on
multiple computers that are located at one site or distributed across multiple
sites and
interconnected by a communication network.
[00159] The processes and logic flows described in this specification can be
performed
by one or more programmable processors executing one or more computer programs
to
perform functions by operating on input data and generating output. The
processes and
logic flows can also be performed by, and apparatus can also be implemented
as, special
purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an
ASIC
(application specific integrated circuit).
[00160] Processors suitable for the execution of a computer program include,
by way
of example, both general and special purpose microprocessors, and any one or
more
processors of any kind of digital computer. Generally, a processor will
receive
instructions and data from a read only memory or a random access memory or
both. The
essential elements of a computer are a processor for performing instructions
and one or
more memory devices for storing instructions and data. Generally, a computer
will also
include, or be operatively coupled to receive data from or transfer data to,
or both, one or
more mass storage devices for storing data, e.g., magnetic, magneto optical
disks, or
optical disks. However, a computer need not have such devices. Moreover, a
computer
can be embedded in another device, e.g., a mobile telephone, a personal
digital assistant
(PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to
name just
a few. Computer readable media suitable for storing computer program
instructions and
data include all forms of non volatile memory, media and memory devices,
including by
way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash
memory devices; magnetic disks, e.g., internal hard disks or removable disks;
magneto
optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can
be supplemented by, or incorporated in, special purpose logic circuitry.
[00161] Implementations of the subject matter described in this specification
can be
implemented in a computing system that includes a back end component, e.g., as
a data
server, or that includes a middleware component, e.g., an application server,
or that
51
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
includes a front end component, e.g., a client computer having a graphical
user interface
or a web browser through which a user can interact with an implementation of
the subject
matter described is this specification, or any combination of one or more such
back end,
middleware, or front end components. The components of the system can be
interconnected by any form or medium of digital data communication, e.g., a
communication network. Examples of communication networks include a local area
network ("LAN") and a wide area network ("WAN"), e.g., the Internet.
[00162] The computing system can include clients and servers. A client and
server are
generally remote from each other and typically interact through a
communication
network. The relationship of client and server arises by virtue of computer
programs
running on the respective computers and having a client-server relationship to
each other.
[00163] While this specification contains many specifics, these should not be
construed as limitations on the scope of the invention or of what may be
claimed, but
rather as descriptions of features specific to particular implementations of
the invention.
Certain features that are described in this specification in the context of
separate
implementations can also be implemented in combination in a single
implementation.
Conversely, various features that are described in the context of a single
implementation
can also be implemented in multiple implementations separately or in any
suitable sub-
combination. Moreover, although features may be described above as acting in
certain
combinations and even initially claimed as such, one or more features from a
claimed
combination can in some cases be excised from the combination, and the claimed
combination may be directed to a sub-combination or variation of a sub-
combination.
[00164] Similarly, while operations are depicted in the drawings in a
particular order,
this should not be understood as requiring that such operations be performed
in the
particular order shown or in sequential order, or that all illustrated
operations be
performed, to achieve desirable results. In certain circumstances, multi-
tasking and
parallel processing may be advantageous. Moreover, the separation of various
system
components in the embodiments described above should not be understood as
requiring
such separation in all embodiments, and it should be understood that the
described
program components and systems can generally be integrated together in a
single
software product or packaged into multiple software products.
52
CA 02800372 2012-11-19
WO 2011/146254 PCT/US2011/035465
[00165] A number of implementations have been described. Nevertheless, it will
be
understood that various modifications may be made without departing from the
spirit and
scope of the disclosure. Accordingly, other implementations are within the
scope of the
following claims. For example, the actions recited in the claims can be
performed in a
different order and still achieve desirable results.
53
