Note: Descriptions are shown in the official language in which they were submitted.
CA 02283382 1999-10-04
F:\I\ICL.536\Pat-App.013\Patent.Doc
AN APPAR,~TUS AND METHOD FOR CAPTURING AND
PF~OJECT'ING AN IMAGE OF A CART
Field of the Invention
This invention relates to electronic surveillance systems for loss-prevention
and security, and in
particular, to an inspection apparatus for capturing and processing an image
of a portion of a cart
at a checkout location of a retail store.
Background to the Inventiam
Shopping carts, as used in supermarkets, for example, often include racks
below the main storage
basket for carrying additional items. The lower rack is a convenience for the
customer, and
increases the carrying capacity of the carts. However, due to the obscuring
presence of the
check-out desk, goods carried on the lower rack tend not to enter the
attendant's lines of sight for
long, if at all, and consequently may leave the store unnoticed and unpaid
for.
A variety of devices exist ~to assist cashiers in detecting the presence of
goods in the lower
portion of the shopping cart. The most common are mirrors positioned so as to
give the
attendant an improved view of the lower cart region for at least a short
period of time. However,
this approach requires the attendant to look at the proper moment when the
cart is in the viewing
area, which is not always practical.
A number of devices that automatically detect the presence of objects on the
lower rack of a cart
as it passes a fixed location have been used. United States Patent Nos.
4327819 and 4723118
describe detection means responsive to the weight of objects placed on the
lower rack. United
States Patent Nos. 3725894.. 5485006, and 4338594 describe a detection
approach based on the
interruption of optical beams by obje-cts placed on the lower rack. Note that
while the properties
of object weight or optical ~~pacity rnay be exploited to detect their
presence on the lower rack,
neither technique can determine the type or number of objects on the lower
rack. Consequently,
the detection signal must initiate an alarm (audible or visible) which prompts
the attendant to
perform a visual inspection of the lower rack. These devices, therefore, are
an aid to the cashier,
as they do not perform inspection automatically.
CA 02283382 2000-12-05
-2-
United States Patent Nos. 5883968 describes a device that employs imaging and
automated
image analysis to detect the presence and type of goods in the lower rack. In
particular, it
describes the use of a digital image analysis technique whereby a reference
image of an empty
cart is compared to a database of images. It also describes the use of colour-
discriminating and
Identification Code discriminating techniques. Note that, in general,
automated image-based
techniques for object discrimination and identification are subject to a large
number of variables
(e.g. lighting colour, lighting direction, objection position, object
orientation, shadowing,
occlusion) that affect the reliability and accuracy of the identification
process. It is often difficult
in practice to control these variables. Such systems, therefore, can produce
both false-positive
and false-negative type errors during detection.
Accordingly, it is an object of one aspect of the present invention to solve
many, if not all, of the
drawbacks of the prior art systems by providing an improved inspection aid to
the cashier
through the use of images as opposed to alarms. Moreover, the present
invention uses human
visual processing as opposed to automated image processing, and is therefore
less likely to
produce identification errors.
Summary of the Invention
In one aspect of the present invention , there is provided a system for
monitoring and evaluating
the performance of a checkout clerk at a retail store, the system comprising:
a. a detector to detect the presence at a predetermined location of a shopping
cart
containing items to be purchased;
b. a camera to capture at least one image of a portion of the shopping cart
containing
the items to be purchased;
c. means to electronically stamp the at least one captured image with a
current date
and time, and a unique location identifier;
d. storage and retrieval media for storing the at least one captured image,
e. means to compare the items to be purchased in the captured image with data
from
corresponding sales receipts so as to monitor and evaluate the performance of
the
checkout clerk; and
f. means to connect the system with other similar systems over a local area
network.
CA 02283382 2000-12-05
-3-
In a second aspect of the present invention, there is provided a method of
monitoring and
evaluating the performance of a checkout clerk at a retail store, the method
comprising:
a. detecting the presence at a predetermined location of a shopping cart
containing
items to be purchased;
b. capturing at least one image of a portion of the shopping cart containing
the items
to be purchased;
c. projecting the at least one captured image from a display such that the
image can
be viewed by the checkout clerk to determine if there are items on the portion
of
the cart.
d. electronically stamping the at least one captured image with a current date
and
time, and a unique location identifier; and
e. storing the at least one captured image in storage and retrieval media,
f. retrieving the stored image and comparing the items to be purchased in the
image
with data from corresponding sales receipts to monitor and evaluate the
performance of the checkout clerk.
More specifically, in a preferred embodiment, the present invention is
directed to an apparatus
for use in stores such as supermarkets and other retail outlets for aiding
cashiers in the inspection
of goods contained in the lower portion of shopping carts. In addition, the
preferred embodiment
provides a sequential record that may be inspected by store managers to audit
the performance of
cashier's effectiveness with respect to said inspection task.
Brief Description of the Drawings
FIG. 1 is a pictorial perspective of one embodiment of the apparatus of the
present invention
showing the physical arrangement of the sensing, imaging and display elements
when used as an
aid to supermarket cashiers in examining goods in the lower tray of a shopping
cart;
FIG. 2 is a functional system block diagram of one embodiment of the present
invention; and
FIG. 3 is a state-transition diagram of one embodiment of the present
invention showing the state
machine that controls the image display.
Description of the Preferred Embodiment
One embodiment of the present invention is shown in FIG. l, which shows
detection and
imaging unit 1 arranged so as to provide an inspection aid to a cashier at a
supermarket checkout
CA 02283382 1999-10-04
-4-
desk. Included in the detection and imaging unit housing are windows la and lb
which provide
an optically transparent protection for the optical sensing means and imaging
means that are
described in more detail later. Also shown is a retro-reflective optical
target 3b, which is
typically a self adhesive patch of conformable, retro-reflective material that
is bonded to a
convenient surface of the shopping cart. The spacing of the detection and
imaging elements is
arranged by design such that an image of the lower portion of each passing
shopping cart is
captured at approximately the time that the cart passes through the field of
view ld. The image
is then displayed to the cashier at an appropriate time on the display unit 2.
In FIG. 2 a functional block diagraan of the preferred embodiment is shown.
The description
begins following initial power-on and before the detection of an object. For
the purposes of this
description, the short-term l;~uffer 5 has two slots, corresponding to two
images, one of which is
for immediate display and t:he second retained for later display. Note that in
general, the short-
term buffer can be designed to accommodate as many images as are necessary for
a specific
application.
Imaging element 4 provides a continuous stream of digitized image data to the
short-term buffer
5. Typically, this image data stream is produced by means of a standard
interlaced format video
camera 4a, providing an analog video signal through anti-abasing filter 4b to
synchronization
signal detection circuitry and analog-to-digital conversion circuitry 4e. The
sampling frequency
of the analog-to-digital converter must be sufficiently high to capture the
chrominance and
luminance portions of the siignal as well as the line and frame
synchronization pulses contained
in the signal. The digitized video signal data is written to the short-term
buffer on a continuous
basis at a rate matching the standard video frame rate of 60 frames per second
(30 even frames
plus 30 odd frames). Note however, that only data corresponding to either the
even frames or the
odd frames, but not both, is retained, for reasons that will now be explained.
A video frame is composed of a series of parallel horizontal lines. In the
standard interlaced
format, the so-called even and odd frames are acquired 1/60 of a second apart.
Consequently,
movement of an object seen by the camera produces an apparent displacement in
the image of
that object between even and odd fr~unes. This apparent displacement goes
unnoticed by human
CA 02283382 1999-10-04
- S -
vision when the video disp lay is continuously updated at the video frame rate
of thirty fresh
even/odd frames per second, as happens in a normal video display mode.
However, if a single even/odd frame pair is replayed repetitively with the aim
of creating a so-
called "freeze-frame" or still image mode, the aforementioned displacement
artifact is highly
S noticeable. It has been found that for this special case, eliminating either
the even or the odd
frame from the pair can produce a more satisfactory displayed image.
When a first object with an attached target 3b passes by sensor 3a, a binary
output signal is
produced. This signal is received by a logic-processing element 8, which
determines that the
video frame buffered at the current instant should be retained (that is, not
erased by the arnval of
subsequent video data) and sent to the display element 7 for viewing by an
attendant. The
display of the image is typically effected by reading the digitized frame from
the buffer to a
digital-to-analog converter '1a, then conditioning the signal through a
reconstruction filter 7b,
and finally connecting to the input of a standard video display monitor 7c.
For reasons
previously explained, only the even frame or the odd frame, but not both, is
retained. Therefore,
to preserve the standardized video signal timing (essential for proper display
on a standard
monitor) the frame is read tvice every 1/30 second, thereby approximating the
form and timing
of a normal even/odd pair.
The displayed image remains visible; on the monitor until receipt of an event
signal 8c, which
causes the logic-processing element to terminate the displayed image. The
event signal 8c is
typically derived from the detection of an event that coincides with the end
of the inspection
process. If detection of ~~uch an event is impractical, or cannot be achieved
with high
consistency, a second display termination signal can be created on the basis
of a pre-determined
elapsed time using a timer 8a.
If a second object with an attached target 3b now passes by sensor 3a, an
output signal is again
produced. Again, this signal is recc;ived by the logic-processing element 8.
However, if this
signal is received before the current displayed image is terminated by the
action of either signal
8c or 8a, then the logic-processing element determines that the video frame
buffered at the
current instant should be retained, but not displayed at this point.
CA 02283382 1999-10-04
The receipt of a display-terminating signal, either 8c or 8a, will cause the
logic-processing
element to replace the currently displayed image by the frame retained in the
buffer. If yet
another display-terminating signal 8c or 8a is received prior to the detection
of another object
target 3b, the display clears to a "blank" state.
The data for every image that is displayed is also routed to a medium-term
storage unit 6. This
buffer has a multiplicity of memory slots 6a accommodating a larger number of
images, as
would be required to account for say., the activity over a period of a day or
a week. Included is a
date/time generator 6b for adding date/time information to each image and a
general processing
element 6c for memory mmlagement and data transfer tasks. The medium-term
memory is
arranged as a first-in, first-out buffer. Therefore, the finite number of
slots always holds the most
recent images, older images being automatically discarded.
A communications interface 9 provides the ability to transfer images from the
medium-term
storage unit to a computer ~t0 where. they can be saved as data files, viewed,
and merged with
other data. The communications interface 9 can be a direct point-to-point
connection, such as an
asynchronous serial link or ~:elephonc: modem, or a mufti-point local area
network, which would
support access to a multiplicity of irr~aging stations via distributed
communications ports 9a, 9b,
etc.
In the typical arrangement for the preferred embodiment, detection of the
presence and position
of a shopping cart is by means of an infra-red transmitting optical sensor 3a,
responsive only to
special retro-reflective target 3b material attached at an appropriate
location on every shopping
cart. The sensor 3a and th.e target 3b material may be of state-of the-art
design, embodying
features such as polarizing filter, frequency modulation and background
suppression to ensure
high detection accuracy.
A typical form of imaging means 4 is the combination of electronic video
camera 4a and analog-
to-digital converter 7d. Cameras with standardized analog video output signal
formats such as
PAL or NTSC are readily available. Typically, said camera 4a is position in
the check-out aisle
so as to achieve a wide-angle, side-on view of the lower region of the
shopping cart, though
alternate views are possible, An analog-to-digital converter 7a operating at
minimum 12 MHz
CA 02283382 2000-04-13
sampling rate is used to digitize the NTSC video frame without sacrificing the
quality of the
image. An anti-aliasing filter 4b is used to ensure that the analog signal
contains only frequency
components commensurate with the sampling frequency.
While being digitized, the video signal is written to short-term memory 6a
typically static
random access memory (SRAM). The timing of the conversion and writing process
is arranged
such that a single complete video frame is captured. Said timing is
facilitated by standardized
synchronization signals that are contained within the video signal. The amount
of short-term
memory capacity required by the design relates to the practical imperative of
processing a queue.
For example, a cashier will frequently encounter situations whereby a second
shopping cart
passes the camera location before the first cart has completed inspection.
This situation requires
that both the displayed image and the most recent image are buffered,
otherwise, one or the other
would be lost.
The digitized video frame is rapidly retrieved for display by first scanning
the corresponding set
of SRAM memory cell locations at the appropriate rate (i.e. 12 MHz) and
streaming the data to a
digital-to-analog converter 7a. The converter 7a output is then passed through
a reconstruction
filter 7b and directed to a compatible monitor 7c (e.g. NTSC). The digitized
video frame is re-
scanned repeatedly at the standard video rate for the duration that the image
remains displayed.
The preferred form of video monitor 7c uses a compact, flat-screen technology
such as liquid
crystal display (LCD). The display is mounted at a convenient height where the
cashier can
easily see it. A swivel mount may be employed so that the monitor may be
easily adjusted to suit
cashiers of different heights.
To display the correct images at the correct times requires logic processing
and event-signalling,
typically implemented in the form of a state-machine. The term "state machine"
refers to a
defined set of logical variables (i.e. variables that can have the value 1 or
0). A state machine is
the name conventionally used for a process that produces a specific logical
output state
depending on the combination of the current internal state and external
events. State machines
can be implemented in either hardware logic circuits or in software programs
executed by
microprocessor.
CA 02283382 1999-10-04
In this preferred embodiment, the state machine manages the various possible
display states in
response to event signals. The set of display states could include, for
example, "display blank",
"display buffered image 1" and "display buffered image 2". The set of event
signals could
include "cart detected", "transaction completed" and "timeout exceeded". Each
event signal is
derived from a measurable physical event: for example, the detection of an
optical target when
the cart passes a predefined position; the detection of a voltage change on
the cash drawer
unlatch solenoid when the cashier totals the sale; and the passing of a pre-
defined period if time
since a given image appeared on the display. The following examples provide an
explanation of
the action of the state machine.
Receipt of the "cart detected" event causes the video frame captured during
that sampling instant
to be displayed if the display state at the time of the vent is "display
blank". Otherwise the video
frame is buffered, pending clearing of the currently displayed image by either
a "transaction
completed" event or a "timeout exceeded" event.
Receipt of the "transaction completed" or "timeout exceeded" event causes the
display to clear to
the "display blank" state if there is no frame waiting in the buffer at the
time of the vent.
Otherwise, the display changes to show the buffered image.
Note that the event signal "transaction completed" is the most usual cause of
a change in the
output display state. However, the "timeout exceeded" event is provided as an
alternate signal
with a similar effect. The "timeout" function ensures that an image never
remains displayed
longer than a pre-determined length of time, for example four minutes. An
example of when this
is used is when a single can moves forward, then briefly backward, across the
sensor thereby
creating two "cart detected" event signals, and (if the buffer is clear)
acquiring two images of the
same cart. Since the cashier will only create a single "transaction completed"
event
(corresponding to the single cart), the display will be cleared of the first
image but not the
second. When this occurs, the display state is out-of step with the sequence
of passing carts, and
remains so indefinitely. To remedy to this condition, an artificial
"transaction completed" event
is created after a short time-limit. With this provision, the system self
corrects as soon as there is
a gap between incoming carts that exceeds the time-out limit.
CA 02283382 1999-10-04
-9-
The images that are acquired) and displayed in the manner described in the
preceding paragraphs
may also be stored along with a corresponding date and time code to create a
retrievable record.
This storage function is fa<;ilitated by a microprocessor 6c and time-code
generator 6b and
medium-term memory 6a. Medium germ memory 6a is preferably of a non-volatile
type such as
"flash" (electrically prograrrunable) RAM or battery-backed RAM. The memory
management
may be arranged in the manner of a circular buffer, whereby the oldest data is
overwritten by new
data once the memory capacity is exceeded. Typically, the memory capacity is
designed to
accommodate enough images to cover at least one workshift.
Retrieval of images stored in medium-term memory 6a is facilitated by the use
of a
communications interface f to a computer system 10. Once transferred from the
long-term
memory to the computer system 10 the images may be viewed, correlated with
other data such as
sales receipts, and archived to a mass storage system. The communications
interface 9 may be
any of a number of standaJ-dized types, from simple point-to-point serial-data
or parallel-data
ports to a mufti-point local area network.
1 S The present invention has been described with reference to its preferred
embodiment, however, it
will be understood that th.e scope of the present invention is not limited to
the preferred
embodiment, but its scope is defined by the claims. Further, it will be
understaood modifications
may be made within the scope of the present invention without departing from
the spirit thereof,
and the present invention includes all such modifications.
