Note: Descriptions are shown in the official language in which they were submitted.
CA 02896357 2015-07-03
- 1 -
A Method and Laser Pointing Machine System for Indicating Items in a Display
Case
SUMMARY OF INVENTION
A method and technological system is provided for the indication of
purchasable items situated
within a display-case with a controllable laser pointing apparatus. Multiple
embodiments of the
invention are described wherein the claimed method of indication can be
achieved.
A system for the control and manipulation of a laser pointing apparatus is
described in one
embodiment.
In another embodiment a plane equation of the display-case's product-placement
surface is
obtained through a technique of topography and is represented in world
coordinates relative to a
reference frame. A capture device is used to detect objects that have moved to
within a
predefined proximity of the display-case's transparent window. The detected
objects are
matched to probabilities that determine the object's eligibility as a detected
pointing-object. In
one embodiment detected pointing-objects are analyzed to compute a straight
line that collinearly
traverses the central moment of the most linear portion of the object that is
most proximal to the
display-case's transparent window surface. A 3-dimensional coordinate of the
point of
intersection of the straight line and the plane is computed in world-
coordinates. These
coordinates are outputted to a controllable robot with an end effector adapted
to fit and orient a
controllable laser emitting diode to illuminate the point of intersection. The
pointing object's
position is successively followed and the straight line, collinearly
traversing the pointing-object's
most linear portion, is continually calculated. The point of intersection is
recalculated and the
robotic mechanism continually adjusts the laser emitting diode to illuminate
the point of
intersection.
TECHNICAL FIELD
The present invention provides an improved method of indicating product items
within a display-
case and laser pointing systems that can be controlled by customers for the
indication of these
products. Laser pointing systems are comprised of displacement and rotational
sensors, such as
accelerometers and gyroscopes respectively, which are configured to track and
limit the angular
CA 02896357 2015-07-03
- 2 -
and spatial regions within which the laser pointing systems can operate. Other
laser pointing
systems are mechanically configured for lateral motion and pivotable motion
about a display-
case's transparent such that the direction of the lasers' emitted laser beams
are only toward points
within a display-case.
A robotic system is comprised of an RGB camera and an end-effector that
manipulates a laser
diode. The robotic system may also be configured for topographical analysis of
a display-case
surface using the orientation information of the end-effector and processed
image data acquired
from RGB camera. The robotic system is also communicably linked to a gesture
recognition
system comprising a central computing system and capture device or range
imaging camera.
The gesture recognition system may use several object tracking techniques and
depth data
processing to detect pointing gestures and pointing members and also track the
pointing
members. A vector generated from pointing members within the capture device's
field of vision
is used to find points of intersection on a topographically analyzed surface
within a display-case.
The central computing system computes the spatial coordinates of the point of
intersection and
manipulates the robotic system to illuminate the point of intersection with a
laser beam emitted
by the laser diode.
BACKGROUND AND PRIOR ART
In many retail stalls such as in groceries, bakeries and jewelry stores
display cases generally hold
a variety of products for customer viewing, selection, and choice. The
customer service
representative (CSR) in attendance will retrieve the specified selection for
the customer.
These displays can hold a multitude of products allocated to some apportioned
space. They can
be organized in rows and columns, and sometimes randomly, but often in close
proximity to each
other so that a customer's ability to communicate specific items of interest
to the CSR from
among the many products can become a rather difficult prospect. Usually, there
are tags that are
appended to the end of the display case, or the products' apportioned spaces
that distinguishes the
corresponding products by name and price. Each displayed product may consist
of a multitude
of pieces or units. These pieces or units are often differentiated by
variations in quality that are
substantiated by different physical characteristics such as dimensions, weight
and visual
CA 02896357 2015-07-03
- 3 -
appearance. A customer will assess these units or pieces and decide on an item
or items with the
most desirable quality.
Display shelves are often equipped with transparent windows that separate the
product on
display from the customer for hygienic or security reasons, but they can
impede the customer's
ability to readily indicate to the CSR what specific item they want.
Currently, the process of a customer indicating their choice of a specific
item or product within a
display case involves two different methods or combinations thereof. The
customer can verbally
describe the location of their choice of a particular product to the CSR. This
process begins with
the customer using some arbitrary reference point that is mutually discernible
to both the
customer service representative and the customer, such as the first item
closest to the customer
on their left or right. With the customer identifying and relaying the
reference point to the
representative, they then direct the representative with a series of
navigating instructions to their
choice of items. This process also uses trial and error as the representative
can wrongly interpret
the customer's instructions and thus select the wrong item at which point the
process begins
again. This item is then used as a new reference point from which the customer
can re-direct the
representative to their right choice.
A customer may also attempt to physically point out the product from the
transparent window
barrier that separates the customer from the product. This involves the
customer service
representative attempting to determine the specific product being pointed to
by approximating a
straight line from the pointing finger tip to the intended item. This process
often results in
erroneous selections due to the CSR's inability to correctly estimate the
projected line between
the customer's pointing finger and the item in question.
These efforts can be tedious and time wasting, and result in customer
frustration, longer waiting
times for service, and occasional loss of sales.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 illustrates a customer indicating a purchasable item within a display-
case to a Customer
Service Representative using a laser pointing apparatus
FIG. 2 illustrates an example embodiment of a laser pointing apparatus that
maybe used to
indicate items within a display-case
FIG. 3 illustrates the components that maybe used to create a laser pointing
system of FIG. 2
CA 02896357 2015-07-03
- 4 -
FIG. 4 illustrates a flow method of controlling the on-off powering of the
laser pointing system
of FIG. 2
FIG. 5 illustrates a flow method to set up a spatial region and range of
angular rotation in which
a laser pointing apparatus may be operable.
FIG. 6 illustrates a flow method by which the laser pointing apparatus maybe
be used in
accordance with the steps of FIG.5
FIG. 7 illustrates another example embodiment of a laser pointing apparatus
that maybe used to
indicate item within a display-case
FIG.8 illustrates the components of a laser pointing apparatus that may be
used to create a laser
pointing apparatus of FIG. 7
FIG. 9 illustrates another example embodiment wherein a customer is indicating
a purchasable
item within a display-case to a Customer Service Representative with a laser
pointing system
FIG. 10 illustrates the components that maybe used to create a laser pointing
system of FIG.9
FIG. 11 is a close up view of a pointing-object and ifs geometry
FIG. 12 is a flowchart illustrating a method of establishing a coordinate
system and the relative
spatial distributions of components of a laser pointing system of FIG. 10
FIG. 13 is a flowchart illustrating a method of acquiring a plane equation and
bounding region
for use with the laser pointing system of FIG. 10
FIG. 14 is a flowchart illustrating a method of detecting a pointing object
and illuminating a
point being pointed to by the pointing object using the laser pointing system
of FIG. 10
FIG. 15 illustrates the laser pointing system of FIG.10, an approximated plane
and a bounding
region that is collinear with the plane
FIGS. 16a, 16b and 16c illustrate a sequence whereby a robotic end-effector
with a laser diode
maybe manipulated to illuminate a specific point on a plane
FIG. 17 illustrates an alternate embodiment of the present invention adapted
for use with a multi-
shelved display case.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
Turning now to the drawings in which like reference characters indicate
corresponding elements
throughout the several views, attention is first directed to FIG. 1 in which
is seen a Customer 14
CA 02896357 2015-07-03
- 5 -
indicating a specific purchasable item 30 disposed within a display-case 10 to
a Customer
Service Assistant, (CSA), 12. A customer service assistant maybe referred to
by many different
names including "retail clerk", "retail assistant", "customer service
representative" and the like.
The term "customer service assistant" will henceforth be referred to as CSA,
12, in this Detailed
Description of Preferred Embodiments. The customer 14 indicates a specific
purchasable item
30 to the CSA, 12, by manipulating a laser pointing apparatus 26 to direct a
coherent beam of
light 24 across a transparent material 16 to illuminate at least one point 22
on the surface of the
specific purchasable item 32. The transparent material 16 may also be referred
to as a "display-
case window" or "transparent window" and may be comprised of glass, plastic,
acrylic or the
like. The illuminated point 22 is visible to the customer 14 and the CSA 12.
Also disposed
within the display-case 10 is a plurality of purchasable items 32 which the
customer can view
through the transparent material 16. The side of the display-case 10 on which
the transparent
material 16 is attached may be referred to as the "customer side". The side of
the display-case 10
on which the CSA 12 has access to the purchasable items 32 and 30 may be
referred to as the
"clerk side". The purchasable items 32 and 30 within the display-case 10 are
made accessible to
the CSA through a door 12 attached to the display-case 10 and located on the
"clerk side". The
customer may wish to acquire this indicated item 30 and communicate this wish
to the CSA 12
who in turn retrieves the indicated item 32 through the access provided by the
door 12. The
customer 14 may wish to indicate the item 30 to the CSA for a number or
combination of reasons
such as but not limited to: acquiring the item for purchase, inquiring the
price of the item,
inquiring the weight of the item, acquiring the item for closer inspection of
the item and the like.
As illustrated in FIGS. 1 and 2 the laser pointing apparatus 26 may be
connected to a connecting
component 28 such as a chord, wire, cable and the like, which may be further
attached to an
anchoring component 20 which may be further secured to the display-case 10.
The connecting
component 28 may serve to limit the distances through which the laser pointing
apparatus 26 can
be moved and manipulated. The connecting component 28 may also be comprised of
a
gooseneck or flexible neck or flexible tube. Other examples of connecting
components that may
be adapted to fit a laser pointing apparatus 26 include flexible arms, movable
joints, telescopic or
extendable tubes such as found in US Patent Number 1,010,335; 1,735,212;
3,381,122;
4,895,708, US 6803525 B1 for example.
CA 02896357 2015-07-03
- 6 -
As shown in FIG. 1 the anchoring component 20 may also be comprised of a
holstering
component 21. In accordance with the current embodiment the holstering
component 21 may
serve to confine the laser pointing apparatus 26 when not in use and prevent
undesirable
movement as that caused by gravity and arbitrary vibrations. The holstering
component 21 may
comprise a rubberized band that provides sufficient friction to hold and
prevent the laser
apparatus 26 from unwanted movement.
In an alternate embodiment not here shown the anchoring unit 20 may be
comprised of a
magnetic surface to which a laser pointing apparatus 26 that may be comprised
or partly
comprised of a ferromagnetic material attracted to the magnetic surface with
sufficient force to
avoid unwanted movement.
In yet another embodiment not here shown the anchoring component 20 may be
comprised of a
holder adapted to hold the laser pointing apparatus 26 when not in use.
Suitable examples of
holders that may be adapted to hold the laser pointing apparatus 26 when not
in use include: US
patent 5405024A entitled "Pen Holder", patented on 11 April, 1996 and hereby
fully
incorporated herein by reference; US patent 6202862 B1 entitled "Tubular
yielding holder for
various size pens", patented on 20 March, 2001 and hereby fully incorporated
herein by
reference; US patent 5232103A entitled "Holder for Elongate Elements",
patented on 3 August,
1993.
In an alternated embodiment of the invention, not here shown, the anchoring
component 20 may
be comprised of a retractable reel mechanism to which the connecting component
20 may be
connected. In such an embodiment the connecting component 20, such as a chord,
may be
retracted into the reel mechanism when the laser pointing apparatus 26 is not
in use. Suitable
examples of retractable reel mechanisms include: US patent US 8387763 B2
entitled
"Retractable cord reel" published 5 March 2013, and hereby fully incorporated
herein by
reference; US patent US 7661855 B2 entitled "Retractable reel assembly"
published 16
February 2010, and hereby fully incorporated herein by reference
CA 02896357 2015-07-03
- 7 -
In an alternate embodiment of the invention, not here shown, the connecting
component 28 may
be directly secured to the display-case 10 via a fastener such as a screw,
glue or the like. The
connecting component 28 may also be connected to a part of which the display-
case is
comprised. Such a part may be a supporting member or a fastener such as a
screw, bolt or the
like.
Display-cases are commonly found in retail stores such as grocery stores,
markets, warehouses,
boutiques and the like. In such an environment the connecting component 28 may
be connected
to a surface that is a part of the retail store or retail environment. Such a
surface may be in
proximity to the display-case such that the laser pointing apparatus 26 may be
used as described
in the present embodiment of the invention. Examples of such surfaces in a
retail environment to
which a connecting component 28 may be fixed include walls, adjacent display
cases, shelving
unit, refrigerated coolers, doors and the like.
A retailer, in the interest of safety, may wish to constrain the operation of
the laser pointing
apparatus 26 such that the emitted laser beam 24 may only illuminate points
within the display
case 10. One example embodiment of a system by which to achieve such said
constrained
operation of a laser pointing apparatus 26 is illustrated in FIGS. 2 and 3.
FIGS. 2 and 3 illustrate
a laser pointing apparatus 26 that maybe comprised of a pressure activated
switch 57
communicably or electrically coupled to a controller 54. As shown in FIG. 2
the pressure
activated switch 57 maybe fitted to the laser pointing apparatus 26 such that
the laser beam 24
emitted from the laser diode 44 may traverse, unobtrusively, to the external
environment. As
shown in Fig. 3, the pressure sensor 57 maybe communicably linked to a
controller 54 to control
the on-off powering of the laser emitting diode 44. The pressure activated
switch 57 may be
fitted with the laser pointing apparatus 26 such that when the laser pointing
apparatus 26 is
manipulated to be in a sufficient force generating contact with the customer-
side surface of the
transparent medium 16, the resultant opposite force is transferred to the
pressure activated sensor
57 which may output a signal to the controller 54 which switches the laser
emitting diode 44 to a
power-on state. The beam of laser 24 emitted by the laser emitting diode 44
may traverse the
direction of the vector of the applied contact generating force where this
said vector intersects
with the transparent medium 16 at an angle of incidence that is less than the
critical angle of
CA 02896357 2015-07-03
- 8 -
reflection of the transparent medium 16 and the wavelength of light of the
emitted laser beam 24.
"Critical angle" is defined as the angle of incidence above which total
internal reflection of light
Occurs.
An example embodiment of such a pressure activated laser pointing apparatus is
shown in FIG. 3
which illustrates a pressure sensor 57 electrically couple to a controller 54.
Suitable examples of
such pressure sensors 57 include a MEMS pressure sensor, silicon pressure
sensors and
piezoelectric pressure sensors. A suitable example of a signal generating
pressure sensor that
can be used in accordance with the present embodiment is US patent US
7290453B2, entitled
"Composite MEMS pressure sensor configuration", patented on 6 November, 2007
and hereby
fully incorporated herein by reference.
Another example embodiment of a system by which to achieve such constrained
spatial
operation of a laser pointing apparatus 26 is illustrated in FIGS. 1-3. As
illustrated in FIG. 1 an
external photo-optic component 18 may be affixed to the display-case 10 such
that a beam of
coherent light emitted by the photo-optic sensory unit 18 may traverse
tangentially and vertically
superior to the surface of the transparent material 16. As shown in FIG. 3,
the external photo-
optic component 18 may be comprised of a photoemitting component 50 which may
further be
comprised of a coherent infrared or visible light emitting device 51 such as
an infrared line-laser
or visible light line-laser. As shown in FIG. 2 a laser pointing apparatus 26
adapted for use with
the external photo-optic component 18 may include a transparent covering 46
through which said
emitted beam of coherent light may traverse. The laser pointing apparatus 26
may further
include a photodetector 40 disposed within the transparent covering 46. The
photodetector 40
may be sensitive to the wavelength of the light emitted by the infrared or
visible light emitting
device 51. Such an arrangement of components may be referred to, by one
skilled in the art, as a
photoelectric sensor. In accordance with the prior art of photoelectric
sensors, the photodetector
40 may be used with a controller 57 to control the on-off powering of the
laser emitting diode 44
in response to light signals received from a light emitting device 51.
FIG. 4 illustrates a method by which the control of the on-off powering of a
laser emitting diode
44 may be achieved. At a first exposure of the photodetector 40 to a light
signal emitted by a
CA 02896357 2015-07-03
- 9 -
photoemitting device 51 the laser diode 44 is switched to a power-on state 62.
At a consecutive
exposure to a light signal from a photoemitting device 51 the laser diode 44
is switched to a
oower-off state 64.
In accordance with the present embodiment of the invention, FIG. 1 illustrates
a customer 14
manipulating the laser pointing apparatus 26 to a first exposure of the
photodetector 40 to a light
signal emitted by a photoemitting device 51. This first exposure causes the
laser emitting diode
44 to switch to a power-on state. The customer 14 may further manipulate the
laser pointing
apparatus 26 to direct a coherent beam of light 24 to illuminate at least one
point 22 on product
surface that is visible to the customer 14 and to the CSA 12. The illuminated
point 22 may be
used as a reference point by which the customer 14 can communicate the
relative location of a
indicated product item 30 to the CSA.
The customer 14 may wish to acquire additional purchasable items 32 and choose
to further
indicate additional purchasable items to the CSA according to the method
described in the above
paragraph. Upon completing the indication of the additional purchasable items
to the CSA, 12,
the customer 14 may chose to return the laser pointing apparatus 26 to the
holstering unit, 20,
whereby there is a second exposure of the photodetector 40 to light signal
emitted by the
photoemitting device 51. The second exposure may then cause the laser diode 44
to switch to a
power-off state.
As illustrated in FIG. 3 the photoemitting component 50 may also be comprised
of a two-
photoemtitter device 52 such as two visible or infrared lighting components
that are at a fixed
and known distance relative to each other. The electromagnetic signal emitted
34 by these
components may be received by the photodector, 40. The photodetector 40 may be
at a known
distance and orientation within the laser pointing apparatus 26. Upon
reception of these
electromagnetic signals 34, the photodetector, 40 may output a digital signal
to a controller 54
which may use the signal for triangulation calculations to obtain 3-
Dimensional distance and
angular rotation measurements of the laser pointing apparatus 26 with respect
to some reference
point. Such a system may be known to those skilled in the art as a 'distance
detection system' or
'distance detection circuit'.
CA 02896357 2015-07-03
- 10 -
As illustrated in FIG. 3 that laser pointing apparatus 26 may also be
comprised of a 2-
photoemitter component 53 at a fixed and known distance relative to each
other. The
electromagnetic signal emitted 34 by these components 53 may be received by
the photodetector,
41 which may output a digital signal to a controller 54. The controller may
use the outputted
signal for triangulation calculations to obtain a 3-dimensional distance and
angular orientation
measurement of the laser pointing apparatus 26 with respect to some reference
point. Suitable
examples of such distance detection systems include: US patent US 8873069,
entitled "Motion
sensing method for determining whether to perform motion sensing according to
distance
detection result and related apparatus thereof', published 28 October, 2014
and hereby fully
incorporated herein by reference; US patent US 7193731 B2, entitled "Optical
displacement
sensor", published 20 March 2007 and hereby fully incorporated herein by
reference.
As illustrated in FIG. 3, the laser pointing apparatus 26 may also be
comprised of a 3-
Dimensional gyroscope 56 used to obtain the angular orientation of the laser
pointing apparatus
26 with respect to a fixed reference frame and a 3-Dimensional accelerometer
55 that may be
used to obtain angular rotation values and displacement values in three
dimensions with respect
to some reference point. The gyroscope 56 and accelerometer 55 may output
signals to a
controller 54 that may be comprised of a clock and cpu that may process these
signals to
calculate and determine real time displacement, 3-Dimensional coordinates and
orientation
values.
FIG. 5 illustrates a method in accordance with the present embodiment by which
the components
illustrated in FIG. 3 may be used to set up a spatial region within which the
laser pointing
apparatus 26 may be constrained to operation and usability. According to one
embodiment, at
step 422, the laser pointing apparatus 26 is at a known orientation and
distance relative to some
reference point. An example of such a reference point may be a point disposed
on the holstering
unit 21. An example of such a known angular orientation could be a zenith of
direction defined
by the direction of the pull of gravity. According to step 424 the laser
pointing apparatus 26 may
be manipulated such that there is a first exposure of the photodetector 40 to
the electromagnetic
radiation emitted by the photoemitter 51 which switches the laser emitting
diode 44 to a power-
on state, 62. The point at which such a first exposure occurs may be the
maximum perpendicular
CA 02896357 2015-07-03
- 1 1 -
distance that is vertically superior to the transparent medium 16 and at which
the laser emitting
diode 44 may be switched to a power-on state or power-off state. According to
steps 426 - 432,
the laser pointing apparatus 26 is further manipulated to illuminate a vertex
of a bounded region
within the display case 10. As shown in step 426 four repetitions may be
necessary to set up and
store the maximum and minimum distance and angular rotation ranges with
respect to the
maximum and minimum distance within which the laser emitting diode 44 can be
in a power-on
state. As shown in step 430 the laser pointing apparatus 26 may be manipulated
to illuminate a
vertex point of a virtual quadrilateral or virtual bounded region within the
display-case 10. At
step 432 the 3-Dimensional distance coordinates and respective angular
orientation values of the
laser pointing apparatus 26 are calculated and stored in a memory unit such as
which the
controller 54 and, or, 58, may be comprised.
FIG.6 illustrates a method in accordance with the current embodiment by which
the laser
pointing apparatus 26 can be used, with accordance to the steps of FIG. 5, to
illuminate points
within the display-case. As shown in step 440 the laser emitting diode 44 is
initially in a power-
off state. At step 442 the laser pointing apparatus may be manipulated to a
first exposure of the
photodetectors to the photoemitters wherein such a first exposure may be at a
maximum
displacement vertically superior and perpendicular to the customer-side of the
transparent
material 16. At step 444 the distance coordinates in 3-dimensions and angular
orientation of the
laser pointing apparatus 26 is calculated. At step 446 the maximum and minimum
angular
orientation values are interpolated with respect to the current distance
coordinates. The angular
orientation values can be interpolated given the current distance coordinates
and the maximum
and minimum angular orientation values and distance coordinates obtained in
steps 426, 430 and
432. Examples of such interpolation techniques include linear interpolation,
polynomial
interpolation, spline interpolation and the like. At step 448 the current
distance coordinates and
current angular orientation values of the laser pointing apparatus 26 are
compared with the
maximum and minimum angular orientation and distance coordinates interpolated
in step 446.
As shown in step 452, the laser emitting diode 44 remains in the power-off
state until the laser
pointing apparatus is within the interpolated distance ranges and respective
angular orientation
ranges. As shown in step 450 if the current distance coordinates and current
angular orientation
CA 02896357 2015-07-03
- 12 -
values are within the interpolated bounds the laser emitting diode 44 is
switched to a power-on
state.
As shown by the steps in FIG. 6, a user may manipulate the laser pointing
apparatus 26 to within
the angular orientation bounds and distance coordinates to illuminate a point
22 that is visible to
both the customer 14 and CSA 12. The illuminated point 22 may then be used as
a reference
point by the customer 14 to communicate the relative location of a indicated
product item 30 to
the CSA 12. The customer 14 may wish to acquire additional purchasable items
32 and choose
to further indicate additional purchasable items to the CSA 12 according to
the method described
in FIG. 6.
Attention is now direction to FIGS.7 and 8 in which is seen an alternate
embodiment of a laser
pointing apparatus that maybe used in accordance with the current invention.
FIGS. 7 and 8
show a laser pointing machine 404 that exhibits pivotable motion along 2-axes
420, 408, 410 and
linear motion 416 along a rail component 406. The rail component 406 can be
adapted for use
with the laser pointing machine 404 such that linear motion 416 can be
achieved with the laser
pointing machine 404. Suitable examples of such a rail component 406 that can
be adapted for
linear motion include US patent 7207432 B2 entitled "Linear motion drive
system and rail
holder" published 24 April, 2007 and hereby fully incorporated herein by
reference; US patent
6499588 B1 entitled "Conveyor System" published 31 December, 2002 2007 and
hereby fully
incorporated herein by reference; US patent 6435719 B1 entitled "Linear guide
device"
published 20 August, 2002 and hereby fully incorporated herein by reference.
As shown in
FIGS. 7 and 8 an elongated component 402 maybe be adapted to fit and retain a
laser emitting
diode 44. The elongated component 402 may be fitted to the geometrical center
of and
orthogonally to the axes of rotation of two rotatable, slotted shafts 412,
414. The two rotatable,
slotted shafts 412 and 414 are placed perpendicularly to each other such that
the elongated
component 402 may be pivoted along a y-axis 408 and x-axis 410 of the two
rotatable slotted
shafts 412, 414. The laser pointing machine 404 and railing 406 may be fixed
to the 'customer-
side' of the transparent material 26 where a customer 14 may manipulate the
elongated
component 402 with a manipulating member such as, but not limited to, a hand
418. As shown
in FIGS. 8 and 9 the arrangement of components 44, 402, 406, 412 and 414
allows the emitted
CA 02896357 2015-07-03
- 13 -
laser beam 24 to traverse unobtrusively and collinearly along the length of
the elongated
component 402 and orthogonally to the rotary axes of slotted shafts 412 and
414 to intersect and
traverse through the transparent material 16 to illuminate points 22 within
the display-case 10.
As one skilled in the art of machining would know, the laser pointing machine
404 can be so
adapted to constrain the direction of the emitted laser beam 24 to within the
display case by
limiting the range of angular rotation of the rotatable slotted shafts 412,414
and the length of the
rail component 406 to achieve such constraints.
Other suitable examples of multi-axial rotatable mechanisms for which may be
adapted for use
with a laser pointing apparatus in accordance with the present invention
include: US Patent US
8770768 entitled "Spherical mechanical linkage and multi-axis trackers"
published 8 July 2014
and hereby fully incorporated herein by reference; US patent US 7905463
entitled "Multiple
axis gimbal employing nested spherical shells" published 15 March 2011 and
hereby fully
incorporated herein by reference; US patent US 4628765 entitled 'Spherical
robotic wrist joint"
published 16 December 1986 and hereby fully incorporated herein by reference.
In accordance with the present embodiment a customer 14 may manipulate the
laser pointing
machine 404 to illuminate a point 22 within the display case 10. The customer
14 may then use
this point as a reference point by which to communicate the relative location
of a specific
product item 30 to the CSA 12. The customer 14 may wish to acquire additional
purchasable
items 32 and choose to further indicate additional purchasable items to the
CSA 12 by further
manipulating the laser pointing machine 404 to illuminate a point within the
display-case 10
which may be used as a reference point by which to communicate the relative
location of a
specific product item 30.
Other suitable examples of laser pointing systems in which the emission of a
laser beam is
confined to illuminate points within a defined area are found in the following
patents, all of
which are hereby fully incorporated by reference: US patent 6,761,456 B2
entitled "Laser
Presentation System Using A Laser Pointer", patented on 13 July, 2004 and
hereby fully
incorporated herein by reference; US patent 6,910,778 B2 entitled
"Presentation System Using
Laser Pointer", patented on 28 June, 2005 and hereby fully incorporated herein
by reference; US
CA 02896357 2015-07-03
- 14 -
Patent 20020011987 entitled "Detection of pointed position using image
processing", published
on 20 July, 2004 and hereby fully incorporated herein by reference.
Attention is now directed to FIG. 9 which illustrates an alternate embodiment
of the present
invention and in which is seen a CSA 12 in attendance of a display-case 10.
Disposed within the
display-case 10 is a plurality of purchasable items 32. A transparent material
or 'display-case'
window 16 separates the customer 14 from physically interacting with the
purchasable items 32
and makes the purchasable items 32,30 visible to the customer 14.
A capture device 70 is affixed to the display-case 10 such that objects moving
to within a pre-
defined spatial region of the capture device and transparent window 16 may be
detected and
tracked. The capture device 70 is used for range imaging and extracting
volumetric data, depth
information and image features, such as textures, colors and the like, of
objects moving within
the said predefined spatial region.
According to an example embodiment, the capture device 70 may be a range
camera that collects
range imaging data via any suitable range imaging technique including stereo
camera
triangulation, sheet of light triangulation, structured light, time-of-flight,
or the like. The capture
device may organize the collected depth information into a depth image. Such a
depth image
may be comprised of pixel whose values correspond to distances along a z-axis
that extends from
the depth camera along its line of sight.
As shown in FIG. 9 customer 14 is shown making a pointing gesture within the
capture device's
70 field of vision. The pointing gesture is made with the customer's 14 hand
and the index finger
may be referred to as a pointing member 76. This pointing member 76 may be
used to indicate a
specific purchasable item 30 to the customer service assistant 12.
In this document a pointing gesture may also consist of any motion toward an
item with a
pointing member. A pointing member shall refer to any object of sufficient
visibility to the naked
eye and with such a spatial property that when displaced toward an item to be
indicated may only
be in contact with said item if brought into point contact with said item
along the direction of a
CA 02896357 2015-07-03
- 15 -
vector formed between points of the item and points of said pointing object
which are in closest,
line-of-sight proximity.
The capture device 70 may be used for object recognition, object tracking,
detecting gestures,
such as pointing gestures, detecting pointing members and extracting
volumetric and spatial data
of objects like, for instance, pointing members. The capture device 70 may
also be used to
estimate a virtual line 78.
A laser-pointing machine 72, communicably linked to the capture device,70, is
affixed to the
display-case 10 such that it is vertically superior to the purchasable items
32. The laser pointing
machine 72 is shown directing a coherent beam of light 24 to impinge upon a
point 22 on the
surface of a specific purchasable item 30 which intersects with the virtual
line 78 extending from
a linear portion of the customer's pointing member 76.
Customer 14 may manipulate a pointing member 76 within the capture device's 70
field of vision
so to direct the laser pointing machine 72 to illuminate points within the
display case 10, such as
a point 22 on the surface of a specific purchasable item 30, to indicate this
item 30 to the
customer service representative 12. In an alternate embodiment a customer 14
may manipulate a
pointing member 76 within the capture device's 70 field of vision to direct
the laser pointing
machine 72 to illuminate a point such as 22 wherein such point 22 is used as a
point of reference
by which a customer 14 may describe the relative location of a purchasable
item 30 to a CSA 12.
FIG. 10 illustrates an example machine system that may be used in accordance
with the present
embodiment. FIG. 10 illustrates a system that comprises a capture device 70, a
laser pointing
machine 72, a computer module 170, a calibration switch 185 and an 110
controller interface
180.
As shown in FIG. 10 the Laser pointing machine 72 may be comprised of a
robotic component
129 and a digital camera 82 which may be communicably linked with the robotic
component
129.
CA 02896357 2015-07-03
-16-
As illustrated, the robotic component 129 comprises a spherically shaped
robotic end effector 80
to which a cylindrically shaped laser emitting diode 154 maybe concentrically
fitted. It is to be
understood however, that in other embodiments, other robotic end effectors of
a different design
and constitution may be used and laser emitting diodes of a different design
and construction
maybe also be used. A switch 152 may be used to control the on-off powering of
the laser diode
154. Such a switch 152 may be communicably linked to a controller or
processor. As illustrated
two drive shaft mechanisms are arranged perpendicularly to each other and on
the same plane.
Each drive shaft mechanism is comprised of an electric motor 120 that provides
torque and
rotational movement 190 along a shaft 122, a rotatable contact mechanism 124,
a coupling 126
and an incremental encoder 128
As illustrated in FIG. 10 the 2 motors 120 provide motive power or drive to
the robotic
component 129. The motor 120 may be commonly referred to, by one skilled in
robotics as an
"actuator" or "drive" component, and maybe powered by air, water pressure or
electricity. The
shaft 122 transmits the rotational movement of the motor to the rotating
contact 124. The
rotating contact 124 is in point contact with the spherically shaped robotic
end effector 80 and
causes counter-rotational movement to the end effector 80 about the same axis
190, 192. The
rotatable contact mechanisms 124 maybe referred to, by one skilled in
robotics, as a
"manipulator".
The coupling device 126 is used to join the motor shaft to the incremental
rotary encoder 138.
The incremental rotary encoder 138 may be calibrated to measure the total
angular rotation
provided by the motor 120. The rotary encoder may be referred to, by one
skilled in the art of
robotics, as a "robotic sensor".
A microcontroller 130 may process the angular rotation data from the
incremental rotary encoder
138, control the rate and direction of rotation of the motor 130 and calculate
the values of the
angular rotation of the spherically shaped robotic end effector 80. The
controller 130 may also
be communicably linked 160 to the computing module 170 and may receive signals
process by
the computing module 170 or transmit processed signals to it. The communicable
link 160
maybe by means of a wireless or wired transceiver.
CA 02896357 2015-07-03
- 17 -
As illustrated in FIG. 10, the laser pointing machine 72 also comprises a
camera 82 that maybe
used to acquire visual data and calibrated for use in topographical analysis.
The camera 82 is at
a fixed and known position from the rotational center of the spherically
shaped robotic end
effector 80. Camera 82 may be, for example, a fixed-focus digital camera that
collects and
encodes image data in RGB color mode, a range-imaging camera and the like.
The laser pointing machine 72 may also include a 3-dimensional accelerometer
110 that
determines the magnitude and direction of acceleration of the laser pointing
machine 72 relative
to a point or frame of reference. The laser pointing machine 72 may also
include 3-dimensional
gyroscope 112 that determines the magnitude of angular rotation and direction
such as the pitch,
heading and bank of the laser pointing machine 72 relative to a frame of
reference.
FIG. 10 also illustrates an example embodiment of the capture device 70. As
shown in FIG. 10
the capture device 70 may include an image camera component 106. The image
camera
component may be a range camera or depth camera that captures the depth image
of a scene
according to a number of different techniques. The depth image may include a
two-dimensional
(2-D) image whose pixel values correspond to the length of or distance
extending along the
image camera's line of sight to points in the scene. The depth value may be a
length or distance
measured in centimeters, meters, or the like of points of objects in the scene
captured.
For example in structured light 3-D scanning, the illuminating unit 104 may
project a striped or
checkered pattern unto a scene using, for example, infrared or visible
electromagnetic radiation.
The emitted striped and patterned electromagnetic radiation is reflected off
the surfaces of
objects in the scene and appears distorted or displaced from the perspective
of a 3-D camera 105
or RGB camera 107 separated by a known distance from the illuminating unit
104. The
distorted patterns and displaced lines are recorded by the 3-D camera 105 or
RGB camera 107
and this data may be analyzed by a processing unit, such as a micro-controller
108 or
microprocessor, and converted into 3-Dimensional coordinate data.
In another embodiment, a stereo camera system, not here shown, consisting of
at least 2 cameras
separated and at fixed and known distances from each other may be configured
to determine the
CA 02896357 2015-07-03
- 18 -
3-Dimensional coordinates of points in a scene using stereo triangulation.
Examples of stereo-
triangulation methods include mid-point method and direct linear
transformation.
In yet another embodiment the standard time-of-flight range imaging technique
may be used to
obtain depth information from a captured scene. In such an example an
illuminating component
104 may emit a short electromagnetic pulse to illuminate a scene and the
camera component
such as the 3-D camera 105 or RGB camera 106 may record the intensity of the
reflected
electromagnetic radiation which is further processed by a processing unit such
as a
microcontroller 108 to determine the physical distances from the capture
device 70 to objects in
the scene.
The capture device 70 may also include a 3-Dimensional (3-D) accelerometer 112
and 3-D
gyroscope 110. These components can be used to determine orientation and
change is
displacement relative to some rest frame. Changes in acceleration and
orientation of the capture
device 70 and the time at which these changes have taken place can be
processed by the
microcontroller 108 to determine the displacement and orientation of the
capture device 70
relative to some rest frame.
As illustrated in FIG. 10 the capture device 70 may also include a
microcontroller unit 108 that
may store and execute instructions and store image data captured by the image
components 106.
The microcontroller 108 may be comprised of a processor and memory component.
The
microcontroller 108 may also be configured for object and target tracking,
feature extraction,
such as texture and blob analysis, and target recognition.
As illustrated in FIG.10 the capture device 70 and laser pointing machine 72
maybe
communicably linked 160, 162 to a computing centre 170. The communicable link
160, 162
maybe by means of a wireless or wired transceiver such as an ethernet cable,
wifi, bluetooth or
the like. The computing center 170 may include a central processing unit
(cpu), 176, a memory
component, 172, a clock 174 and an I/O or "in-out" interface 177. The
computing centre 170
may receive and exchange data and processed information with the capture
device 70 and the
laser pointing machine 72 via communicable links 160,162. Examples of such
received and
CA 02896357 2015-07-03
- 19 -
exchanged data include angular orientation data, displacement data, image data
and depth image
data. The computing centre 170 may also be used for more complex image
processing
operations such as advanced gesture recognition, multiple object tracking and
advanced feature
extraction. The computer center 170 may also be used to compute the angular
orientation and
and 3-Dimensional coordinates of the capture device 70 and laser pointing
machine 72 from data
retrieved from the accelerometer 112 and gyroscope 110.
The computing center may also be comprised of an 110 interface 177 to which a
controller 180
may be communicably linked via, for instance, a USB input or the like.
Controller 180 may also be communicably linked 164 to the computing center
170. The
communicable link 164 may be by means of a wireless or wired transceiver such
as an ethernet
cable, wifi, bluetooth or the like.
In one example embodiment the control device 180 may interface with the
computing center 170
and be configured to transmit and receive data. The control device may
transmit data to control
the laser pointing machine 72 and, for example, manipulate the laser pointing
robot 169 and
control the on-off powering of the laser emitting diode 154. An example of
such a controller 180
may be a pointing device such as a computer mouse or touch screen device that
detects two-
dimensional motion across a surface. This motion may be translated by the
computing center
170 or laser pointing robot 169 into, for example, rotational movement of the
end effector 80.
In yet another example embodiment, not here shown, the controlling device 180
may be a radio-
control transmitter control column with multi-axis movement, such as a
joystick. Each control
column movement along an axis of movement may be translated by the computing
center 170 or
laser pointing robot, into, for example, rotational movement of the effector
80. Such a controller
may also be configured to control the on-off powering of the laser emitting
diode 154.
Yet another example embodiment of a controlling device 180 is a touch-screen
device. Suitable
examples of touch screen devices include, but are not limited to, smartphones,
tablet computers
and the like. Touch gestures detected by the touch screen device may be
translated by
computing center 170 or laser pointing robot 169, into, for example,
rotational movement of the
CA 02896357 2015-07-03
- 20 -
end effector 80 or like robotic devices. Furthermore, image data collected by
the camera device
82 may be transmitted to said touch screen device via, for example, a
communicable link 164 or
I/O interface 177. This image data may be displayed on the touch-screen device
and may be, for
example, RGB images of a scene captured by the camera device 82.
In accordance with the example embodiment, FIG. 10 also illustrates a
calibration switch 185
which may also be included to start the set up of a frame of reference for
which the relative
locations in 3-Dimensions and angular orientations of the capture device 70
and laser pointing
machine 72 are measured. The calibration switch 185 may be a coded signal
transmitted to
computing center 170 via a communication link 166. Another example of the
calibration switch
may be, for example a hardwired electronic switch communicably linked to the
computing center
170. An example of such an electronic switch may be a button-switch, pressure
switch or the
like which may be activated upon compression or decompression. hi such an
example
embodiment the laser pointing machine 72 and capture device 70 may be in a
close proximity or
contact with the switch to accomplish compression or decompression upon
separation or
increased displacement between the two devices.
FIG. 11 illustrates a sectional side-view of the current embodiment that
illustrates a pointing
object 76 and the spatial region 90 of the pointing object 76 that may be used
to calculate a
virtual line 78. The Spatial region 90 may be a group of pixelized depth
values acquired by the
capture device 70, from which a linear line 78 is approximated from the
central moment of these
depth values.
The virtual line 78 may also be calculated as the line of best fit through a
series of centroids per
unit length of the pointing object 76. Centroids of the pointing object 76
that are not within a
certain threshold value may be discarded from the calculation of the line of
best fit. Line fitting
or 'linear line approximations" may be known to one skilled in the art as
Linear Regression.
Techniques for the approximation of line of best fit include the "Least-Square
Method",
"Maximum Likelihood estimation", "Bayesian Linear Regression" and the like.
CA 02896357 2015-07-03
-21 -
The system of the current embodiment, as illustrated in FIG. 9 -11 and
discussed in the preceding
paragraphs, may require a calibration process. Such a calibration process may
determine the
angular orientations and locations given in 3-Dimensional coordinates, of the
individual
components such as the capture device 70 and laser pointing machine 72,
relative to some
reference point. An example of such a calibration process or method is
illustrated in FIG.12.
FIG. 12 illustrates an example flow method by which the coordinates and
orientations of the
capture device 70 and laser pointing machine 72 relative to one or more frames
of reference or
points of reference may be determined. The frame of reference or point of
reference may be, for
example, the laser pointing machine 72 or some point therein, the capture
device 70 or some
point therein, the computing center 170 or some point therein. However, it is
known to one
skilled in the art of Physics or mathematics, that spatial coordinates and
angular orientation
values relative to one frame of reference can be calculated or transformed
with respect to another
frame of reference or point of reference.
As illustrated in FIG. 12, at step 201, the capture device 70 and laser
pointing machine 72 are at
a known orientation and distance from each other. At step 203, the calibration
switch 185 is
triggered, for example, upon displacement of the capture device 70 and, or,
laser pointing
machine 72 from the point of known orientation and distance described in step
201. At step 205
data acquired from the accelerometer 112 and gyroscope 110 and the time the
data was acquired
is recorded. This data may be transmitted to and recorded by, for example, the
computing center
170. At step 207 and 209 the coordinates and orientation of the laser pointing
machine 72 and
capture device 70 relative to the frame of reference is calculated using the
data recorded in step
205. As illustrated in step 216 and 222, any change, or, further change, in
the angular orientation
and, or, position of the capture device and, or, laser pointing machine
relative to the frame of
reference or point of reference produces additional output data from the
accelerometers and
gyroscopes as shown in steps 213, 215, 218 and 220. This data and the time of
the output of this
data 205 is recorded and processed, for instance, by the computing center,
170, to calculate the
new, modified coordinates and, or, orientation values of the capture device
and, or, laser pointing
machine as shown in steps 207 and 209. At step 211, the calculated coordinates
and orientation
CA 02896357 2015-07-03
- 22 -
values of the laser pointing machine 72 and capture device 70 is stored. This
data may be stored,
for example, within a read/write memory device such as 172.
FIG. 15 illustrates a display -case 10 wherein a virtual bounded region 94
lies on a 3-
Dimensional surface 92, such a plane. As illustrated the 3-Dimensional surface
92 may be a
plane that is tangential and, or, parallel to the surface within the display-
case 10 on which
purchasable items 32 may be placed for viewing. As shown in FIG. 15 the laser
pointing
machine 72 is configured to illuminate points within the virtual bounded
region 94. The virtual
bounded region may be of, for example, quadrilateral dimensions. The system of
the current
embodiment , as illustrated in FIG. 9 -12 and discussed in the preceding
paragraphs, may also be
configured such that the emitted laser beam 24 may only illuminate points
within a specific
region, such as, for example, the display case 10. Furthermore the laser
pointing machine 72
may also be configured such that the laser emitting diode may illuminate
points where the virtual
line 78 intersects with a 3-Dimensional surface such as, for example, the 3-
Dimensonal surface
92. Furthermore, the laser pointing machine 72 may also be configured such
that the laser
emitting diode may illuminate points only where the virtual line 78 intersects
with the virtual
bounded region 94. FIG. 13 illustrates an example flow method in accordance
with the current
embodiment, as illustrated in FIGS. 10 -12 and FIG. 15 wherein the laser
pointing machine 72
may be configured to only illuminate points within a virtual bounded region
94.
Turning now to FIG. 13 we see that at step 230 the spherical end effector 80
is at a known
angular orientation. This angular orientation may be, for example, some known
angular
displacement from the fixed zenith direction about one or more axes. As shown
in step 232 a
calibration mode may be started. This calibration mode may be started by, for
example, the
computing center 170 executing programmed code stored in the memory device
170. At step
234 there is a 'For Loop' consisting of at least four repetitions of steps
254. Steps 254 consists of
3 steps 236,238 and 240. A step 236 the laser pointing machine 72 is
manipulated to illuminate
a point within the display-case 70 that corresponds to a vertex of the virtual
bounding region 94
or bounding quadrilateral. This illuminated point within the display case may
be, for example, a
point on the product placement surface. The product placement surface is the
surface or surfaces
of a display-case 70 wherein purchasable items are placed for customer
viewing. The laser
CA 02896357 2015-07-03
- 23 -
pointing machine 72 may be manipulated according to step 236 with an external
controller such
as 180. At step 238 the coordinates of the illuminated point of step 236 is
calculated in 3-
Dimensional Coordinates. One example, in accordance with step 238, by which
the coordinates
of the illuminated point may be calculated in 3-Dimensional coordinates is as
follows:
1. Capture a grayscale image or an average of images before the
illumination of the point of
step 236, with, for example, camera 82. For the purposes of this example
camera 82 may
be a camera calibrated according to a camera resectioning process.
2. Capture a grayscale image at time of the illumination of point of step
236, with, for
example, camera 82
3. Calculate the current angular orientation of spherical end effector 80 with
data obtained
from the incremental encoder 128 at time of illumination of point of step 236
4. Obtain an absolute difference image of the images of steps 1 and 2
above.
5. Binarize the absolute differenced image of step 4 above to isolate the
pixels that
correspond to the illuminated point of step 236 within the binarized image.
The
technique of binarizing may also be referred to as 'gray image thresholding'.
6. Calculate the pixel coordinates of centroid of illuminated point from
step 5, above
7. Convert centroid pixel coordinates into angular values. The angular values
may be, for
example, the visual angles or visual degrees between the illuminated point of
step 236
and the focal point of a resectioned camera such as 82.
8. Calculate 3-Dimensional coordinates of the illuminated point of step 236
using the
angular values from step 7, above, and angular orientation values of step 3,
above.
At step 240 the 3-Dimensional coordinates of the point illuminated in step 236
is stored as
bounding vertex point in a memory unit such as 172. At step 242 the process of
indicating the
vertices of the virtual bounding region 94 stops. At step 244 two vertices of
the virtual bounding
region are paired by proximate vertical displacements. In the current example,
the laser pointing
machine 72 is vertically superior to the surface in the display-case 10 on
which the purchasable
products are placed for customer viewing. In this example the term "points
with proximate
vertical displacements" is to be understood as points that share the closest
vertical displacement
from the laser pointing machine 72. At step 246 the midpoint of the paired
points of step 244 is
calculated in 3-Dimensional coordinates. At step 248 an equation of a plane 92
is calculated
CA 02896357 2015-07-03
- 24 -
using the midpoint of step 246 and two unpaired bounding points of steps 234-
242. At step 250
the equation of the plane 92 of step 249 is stored in a memory unit such as
172. At step 252 a
virtual quadrilateral or the virtual bounding region 94 is approximated from
the four points of
steps 234-242. The virtual quadrilateral or bounding region 94 may be
collinear with the surface
of 92. At step 254 the equation representing the bounding quadrilateral or
bounding region 94 is
stored in a memory unit such as 172. Although not illustrated in the drawings,
it can be inferred
by one skilled in the art of topography or topology analysis, that a 3-
Dimensional equation
modeling the surface within the bounding region 94 may be calculated by
acquiring a population
of points within the bounding region 94 according the steps 236-240. A 3-
Dimensional equation
of a surface may then be fitted to this population of points using curve
fitting techniques.
Furthermore, the coordinates of this population of points may be stored in a
memory unit such as
172. Other examples of systems and methods by which such a surface such as 92
may be
measure and mapped include: US patent US 6915243 entitled "Surface topology
and geometry
reconstruction from wire-frame models, published 25 August, 2000 and fully
incorporated herein
by reference; US patent US 6809803 entitled "Surface topology inspection",
published 26
October, 2004 and fully incorporated herein by reference; US patent US 5311286
entitled
"Apparatus and method for optically measuring a surface", published 10 May,
1994 and fully
incorporated herein by reference.
Turning now to FIG. 14 we see an example flow method in accordance with the
current
embodiment by which a pointing object, such as 76, may be used to manipulate a
laser pointing
machine such as 72 to indicate a point within a display-case 10 by
illumination with an emitted
laser beam 24. At step 300 an object moves into the field of vision of the
capture device 70 and
within certain range parameters of the capture device 70. In one embodiment
such range
parameters may be a distance along the line of sight of the capture device 70
greater or shorter
than certain distances. In another embodiment, the capture device 70 may be
placed tangentially
to the transparent material 16 and a range parameter may exclude points along
the axis
perpendicular to the transparent material 16 and vertically inferior to the
transparent material 16
or capture device 70. Additionally, the range parameter may exclude points
along the axis
perpendicular to the transparent material 16 and which exceed a certain
vertically superior
distance from the transparent material 16 or capture device 70.
CA 02896357 2015-07-03
- 25 -
Step 300 may also include image segmentation of the depth image or depth
images captured by
the capture device 70 at step 300. Suitable examples of methods of image
segmentation include:
US patent US 80009871 B2, entitled "Method and system to segment depth images
and to detect
shapes in three-dimensionally acquired data", published 30 August, 2011 and
fully incorporated
herein by reference; US patent US 8401225 B2, entitled "Moving object
segmentation using
depth images", published 19 March 2013 and fully incorporated herein by
reference.
At step 302 the spatial measurements of the detected object or objects of step
300 are analyzed
and calculated. Examples of such spatial measurements include, but are not
limited to, the
volumetric data of the objects, the volume of the object per unit distance
along an axis or
multiple axes, the surface gradient of object, and the like.
At step 304 the spatial measurements of the detected object or objects of step
300 are compared
against a database of spatial parameter constraints. The spatial parameter
constraints may
contain data such as, but not limited to, a minimum and, or, maximum volume
and a minimum
and, or, maximum rate of change of volume per unit distance along an axis or
multiple axes. Step
304 may also include an object classification of the detected object or
objects of step 300 which
may be compared against a database of objects. Suitable examples of such
object classification
include: US patent US 7646922 B2 entitled "Object classification in video
images", published 12
January 2010 and fully incorporated herein by reference; US patent US 8934709
B2 entitled
"Dynamic object classification", published 12 January 2015 and fully
incorporated herein by
reference.
Detected objects with spatial features that do not fall within the constraints
of step 304 and
remain with the range parameters of step 300 are passed onto an object
tracking step 328 and are
continually analyzed by the capture device according to steps 302 and 304.
Suitable examples of
object tracking systems and methods include: US patent US 7907750 B2, entitled
"System and
method for autonomous object tracking", published 15 March 2011, and hereby
fully
incorporated herein by reference; US patent US 7590262 B2, entitled "Visual
tracking using
depth data", published 15 September 2009, and hereby fully incorporated herein
by reference;
CA 02896357 2015-07-03
- 26 -
US patent US 20120327125, entitled "System and method for close-range movement
tracking"
published 27 December 2012, and hereby fully incorporated herein by reference;
US patent
201200069143, entitled "Object tracking and highlighting in stereoscopic
images", published 22
March 2012 and hereby fully incorporated herein by reference; US patent US
20120309532,
entitled "System for finger recognition and tracking", published 6 December
2012, and hereby
fully incorporated herein by reference.
Detected objects with spatial features that do fall within the constraints of
step 304 are classified
as pointing objects and passed unto step 306. At step 306, 3-Dimensional line
equations, curved
line equations or vectors are approximated from the pointing objects. For
example, a curve may
be fitted or interpolated from a group of centroids calculated per unit length
of the pointing
object 76.
At step 308 a derivative of the curved line equation of step 306 is
calculated.
At step 310 the maxima and minima of the equation of step 308 is extracted and
analyzed in step
312. The maxima and minima of the equation of step 308 may, for example,
indicate the spatial
points of the pointing object at which the greatest change in volume per unit
length of the
pointing object occurs, or the point at which significant bending or curvature
of the detected
pointing object occurs. At step 312 the maxima and minima values acquired at
step 310 are
filtered according to their proximity to the transparent medium.
At step 314 a line of constant slope or a vector is approximated from regions
within the maxima
and minima of step 312 wherein the rate of change of according to step 308 is
closest to zero.
This approximated line equation or vector is the virtual line 78 of the
detected pointing object.
At step 316 the line or vector or virtual line 78 of step 314 is used to
calculate a point of
intersection with a plane or curved surface such as 92.
At step 318 the calculated point of intersection of step 316 may be analyzed
to determine if it is a
point within the virtual bounding region or bounding quadrilateral, 94. If the
point of
CA 02896357 2015-07-03
- 27 -
intersection of step 316 is not a point within the bounding region 94, the
detected object is
tracked according to step 328. A point of intersection lying within the
bounding region 94 or
bounding quadrilateral is used in step 320.
Step 320 consists of a sequence of three steps 322, 324 and 326. At step 322
the current
orientation values of the spherically shaped end effect 82 is acquired from
the laser pointing
machine 72. Orientation values of step 322 may correspond to a direction
vector and line that is
coincident and collinear with a line or vector along which a beam of light
emitted from the laser
emitting diode 44 may traverse. This line or vector intersects with the plane
92 or surface at a
point that may be displaced from the point of intersection of step 318.
At step 324 new rotation values or angular orientation values of the robotic
end effector 80 are
computed such that the laser pointing machine 72 may orient the robotic end
effector 80 and the
laser emitting diode 154 to illuminate the point of intersection of step 318.
At step 326 the point of intersection of step 318 is illuminated by the laser
emitting diode 154.
After step 326 we return to step 328 where the detected pointing object of
step 320 is further
tracked.
We now turn to FIGS. 16a-c, which illustrate a method, in accordance with the
current
embodiment, by which a spherically shaped robotic end effector 80 configured
to fit and retain a
laser emitting diode 154, may be manipulated to illuminate a point 342 on a
plane or surface
such as 92. The point 342 may, for example, be the point of intersection of
the virtual line 78
with a plane of surface such as 92.
FIG. 16a-c illustrates a spherically shaped robotic end effector 80 adapted to
fit and retain a laser
emitting diode 154. The origin of the coordinate system may be the rotational
center of the
spherically shaped robotic end effector 80. As illustrated in FIGS. 16a-c a
plane or curved
surface such as 92 and the point 342 maybe vertically inferior to the
rotational center of the
spherically shaped robotic end effector 80. The robotic end effector 80 may
also be comprised
of a an aperture 340 through which a laser beam emitted from the laser
emitting diode 154 may
CA 02896357 2015-07-03
- 28 -
traverse unobtrusively to the external environment. FIGS. 16a-c also
illustrate rotational
contacts 124 which may be in point contact with the robotic end effector 80
and which rotate
about a z-axis and x-axis. In accordance with the current embodiment the
rotational contacts 124
may be cylindrically shaped with a radius of Rrotator= The spherically shaped
robotic end effector
may have a radius of Rsphere=
As illustrated in FIG.16a, the spherically shaped robotic end effector may be
oriented to some
angular orientation, such as polar angles ac and 13e. a, may be the angular
displacement about the
x-axis from the fixed zenith direction and f3, may be the angular displacement
about the z-axis
from the fixed zenith direction such as in a spherical coordinate system. The
point to be
illuminated 342 may have 3-Dimensional coordinates with respect to the origin:
Xpoinb, Ypoint, Zpoint
and angular displacements ap from the fixed zenith direction about the x-axis
and 13p from the
fixed zenith direction about the z-axis, where:
f
ap ¨ Lau k(Zpoint Ypoint), and, Pp ¨ sin-i kXpoint Ypotnt)
As illustrated in FIG.16b, a rotational contact 124 rotates about the x-axis
346 which in turn
generates a counter-rotational movement of the spherical end effector. For
example, the contact
rotator 124 rotating about the x-axis may complete an angular rotation of a,
about the x-axis
where:
R sphere
¨ ____________________________________________ x (ap-ac)
R rotator
As illustrated in FIG.16c, a rotational contact 124 rotates about the z-axis
348 which in turn
generates a counter-rotational movement of the spherical end effector. For
example, the contact
rotator 124 rotating about the z-axis may complete an angular rotation of 13,
about the z-axis
where:
R sphere
Or R rotator X (Op - 13)
As FIG.16c illustrates the spherically shaped robotic end effector and laser
emitting diode have
been oriented to illuminate the point 342 which lies on the plane or surface
92. The laser
emitting diode 154 illuminates the point 342 by emitting a laser beam 344
which traverses
CA 02896357 2015-07-03
- 29 -
through the aperture 340 along a displacement vector from the rotation center
of the spherically
shaped robotic end effector to the point 342.
The system and methods as described above and as illustrated in FIGS. 10 -16
is an example
embodiment and, as understood, is not meant to limit the scope of the
invention. For example,
the system and methods as illustrated in FIGS. 10-16 may be adapted for use
with a multi-
shelved display-case 500. FIG. 17 is an example embodiment wherein the system
as described
and illustrated in FIGS. 10-16 may be modified and configured for used with a
multi-shelved
display-case.
FIG. 17 illustrates a display-case 500 with multiple shelves 504 disposed
within the display-case
500. Each shelf 504 may hold a plurality of purchasable items 32 which are
separated from the
environment external to the display-case 500 and made visible to customers by
a transparent
medium 16. FIG.17 also illustrates multiple capture devices 70 which may be
affixed, for
example, tangentially to the transparent medium 16. As shown in FIG.17 the
laser pointing
machine 72 may be modified and configured for linear displacement along a rail
component 504.
The capture devices 70 and laser pointing machine 72 are calibrated such that
a pointing object
such as 76 may manipulate the laser pointing machine 72 to direct a coherent
beam of light 24 to
illuminate a point 22 on a purchasable item 30. For example, the flow method
according to FIG.
13 maybe repeated for every shelf 504 within the display-case 500. The flow
method according
to FIG. 14 may be modified such that step 320 includes an additional step of
acquiring the
current linear displacement of the laser pointing machine relative to a
reference point, calculating
the necessary amount of displacement along the railing 504 required by the
laser pointing
machine 72 and displacing the laser pointing machine 72 along the rail
component 504 such that
it may illuminate the point being pointed to by the pointing object.
The methods and system described with reference to details illustrated by the
drawings
represented example embodiments are not intended to limit the scope of the
invention and many
alternate embodiments may become obvious to those skilled in the art. For
example, the laser
pointing machine 72 as described herein is a robotic device which has been
adapted to fit, retain
and manipulate a laser emitting device to illuminate points within a pre-
defined area. The range
CA 02896357 2015-07-03
- 30 -
and type of motion exhibited by a robot is commonly referred to by those
skilled in the art as
"degrees of motion" where each type of motion along an axis or about of axis
is a degree of
freedom. A suitable example of a robotic laser pointing machine which may be
adapted for use
with the current invention is: US patent US 6374158 entitled "Robotic Laser
Pointer" published
16 April, 2002 and fully incorporated herein by reference. The system may also
be configured
such that multiple objects may be tracked by the capture device 70 and
multiple points within a
display-case may be simultaneously indicated and illuminated by a laser
pointing mechanism.
For example, it may become apparent to one skilled in that art that a laser
pointing machine may
be configured to illuminate a series points within the display case such as to
appear as a line
whose length is proportional to the distance of separation between the tips of
2 pointing objects
detected by a capture device. Additionally the system may also be configured
such that
movement gestures, such as swiping with a hand or finger, that are detected by
the capture
device are translated into movement vectors whose displacement magnitude and
direction are
translated into a proportional displacement in the direction of the vector of
the emitted laser
beam. Furthermore, a display-case as illustrated and described in the
description is not meant to
limit the interpretation of a display-case. A display-case includes any
display arrangement of
items whereby said items are viewable to a customer through a transparent
material and said
transparent material separates said customer from said items and wherein such
an arrangement
prevents said customer from physically interacting with said items. Even
further, the various
embodiments of the present invention as described refer to the use of a laser
and laser pointing
mechanism to illuminate a region of or a specific item, however, it will
become apparent to one
skilled in the art that a beam of non-coherent light may be focused, using
optical techniques, to
obtain a similar illuminating effect as the lasers and laser pointing
mechanisms described. The
scope of the present invention is defined by the claims in the appended
claims.
