Note: Descriptions are shown in the official language in which they were submitted.
CA 02684020 2014-05-09
AN APPARATUS SYSTEM AND METHOD FOR HUMAN-MACHINE-INTERFACE
[001] (This paragraph intentionally left blank.)
FIELD OF THE INVENTION
[002] The present invention generally relates to user interfaces and more
particularly to
methods and systems of 3D Human-Machine-Interface.
BACKGROUND OF THE INVENTION
[003] One of the largest patterns in the history of software is the shift from
computation-
intensive design to presentation-intensive design. As machines have become
more and more
powerful, inventors have spent a steadily increasing fraction of that power on
presentation. The
history of that progression can be conveniently broken into three eras: batch
(1945-1968),
command-line (1969-1983) and graphical (1984 and after). The story begins, of
course, with the
invention of the digital computer. The opening dates on the latter two eras
are the years when
vital new interface technologies broke out of the laboratory and began to
transform users'
expectations about interfaces in a serious way. Those technologies were
interactive timesharing
and the graphical user interface.
1
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[004] In the batch era, computing power was extremely scarce and expensive.
The largest computers of that time commanded fewer logic cycles per second
than a typical toaster or microwave oven does today, and quite a bit fewer
than
today's cars, digital watches, or cellphones. User interfaces were,
accordingly,
rudimentary. Users had to accommodate computers rather than the other way
around; user interfaces were considered overhead, and software was designed
to keep the processor at maximum utilization with as little overhead as
possible.
[005] The input side of the user interfaces for batch machines were mainly
punched cards or equivalent media like paper tape. The output side added line
printers to these media. With the limited exception of the system operator's
console, human beings did not interact with batch machines in real time at
all.
[006] Submitting a job to a batch machine involved, first, preparing a deck of
punched cards describing a program and a dataset. Punching the program cards
wasn't done on the computer itself, but on specialized typewriter-like
machines
that were notoriously balky, unforgiving, and prone to mechanical failure. The
software interface was similarly unforgiving, with very strict syntaxes meant
to be
parsed by the smallest possible compilers and interpreters.
[007] Once the cards were punched, one would drop them in a job queue and
wait. Eventually, operators would feed the deck to the computer, perhaps
mounting magnetic tapes to supply another dataset or helper software. The job
would generate a printout, containing final results or (all too often) an
abort
notice with an attached error log. Successful runs might also write a result
on
magnetic tape or generate some data cards to be used in later computation.
2
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[008] The turnaround time for a single job often spanned entire days. If one
were
very lucky, it might be hours; real-time response was unheard of. But there
were
worse fates than the card queue; some computers actually required an even
more tedious and error-prone process of toggling in programs in binary code
using console switches. The very earliest machines actually had to be partly
rewired to incorporated program logic into themselves, using devices known as
plugboards.
[009] Early batch systems gave the currently running job the entire computer;
program decks and tapes had to include what we 'would now think of as
operating-system code to talk to I/O devices and do whatever other
housekeeping was needed. Midway through the batch period, after 1957, various
groups began to experiment with so-called "load-and-go" systems. These used a
monitor program which was always resident on the computer. Programs could
call the monitor for services. Another function of the monitor was to do
better
error checking on submitted jobs, catching errors earlier and more
intelligently
and generating more useful feedback to the users. Thus, monitors represented a
first step towards both operating systems and explicitly designed user
interfaces.
[0010] Command-line interfaces (CLIs) evolved from batch monitors connected
to the system console. Their interaction model was a series of request-
response
transactions, with requests expressed as textual commmands in a specialized
vocabulary. Latency was far lower than for batch systems, dropping from days
or
hours to seconds. Accordingly, command-line systems allowed the user to
change his or her mind about later stages of the transaction in response to
real-
time or near-real-time feedback on earlier results. Software could be
exploratory
3
CA 02684020 2014-05-09
and interactive in ways not possible before. But these interfaces still placed
a relatively heavy
mnemonic load on the user, requiring a serious investment of effort and
learning time to master.
[0011] Command-line interfaces were closely associated with the rise of
timesharing
computers. The concept of timesharing dates back to the 1950s; the most
influential early
experiment was the MULTICS Tm operating system after 1965; and by far the most
influential of
present-day command-line interfaces is that of Unix TM itself, which dates
from 1969 and has
exerted a shaping influence on most of what came after it.
[0012] The earliest command-line systems combined teletypes with computers,
adapting a
mature technology that had proven effective for mediating the transfer of
information over wires
between human beings. Teletypes had originally been invented as devices for
automatic
telegraph transmission and reception; they had a history going back to 1902
and had already
become well-established in newsrooms and elsewhere by 1920. In reusing them,
economy was
certainly a consideration, but psychology and the Rule of Least Surprise
mattered as well;
teletypes provided a point of interface with the system that was familiar to
many engineers and
users.
[0013] The widespread adoption of video-display terminals (VDTs) in the mid-
1970s ushered in
the second phase of command-line systems. These cut latency further, because
characters
could be thrown on the phosphor dots of a screen more quickly than a printer
head or carriage,
can move. They helped quell conservative resistance to interactive programming
by cutting ink
and paper consumables out of the cost picture, and were to the first TV
generation of the
4
CA 02684020 2014-05-09
late 1950s and 60s even more iconic and comfortable than teletypes had been to
the
computer pioneers of the 1940s.
[0014] Just as importantly, the existance of an accessible screen, a two-
dimensional display of
text that could be rapidly and reversibly modified made it economical for
software designers to
deploy interfaces that could be described as visual rather than textual. The
pioneering
applications of this kind were computer games and text editors; close
descendants of some of
the earliest specimens, such as rogue(6), and vi(1), are still a live part of
UnixTM tradition.
[0015] Screen video displays were not entirely novel, having appeared on
minicomputers as
early as the PDP-1 TM back in 1961. But until the move to VDTs attached via
serial cables,
each exceedingly expensive computer could support only one addressable
display, on its
console. Under those conditions it was difficult for any tradition of visual
Ul to develop; such
interfaces were one-offs built only in the rare circumstances where entire
computers could
be at least temporarily devoted to serving a single user.
[0016] There were sporadic experiments with what we would now call a graphical
user interface
as far back as 1962 and the pioneering SPACEWARTM game on the PDP-1 TM. The
display on
that machine was not just a character terminal, but a modified oscilloscope
that could be made to
support vector graphics. The SPACEWAR interface, though mainly using toggle
switches, also
featured the first crude trackballs, custom-built by the players themselves.
Ten years later, in the
early 1970s these experiments spawned the video-game industry, which actually
began with an
attempt to produce an arcade version of SPACEWAR.
CA 02684020 2014-05-09
[0017] The PDP-1TM console display had been descended from the radar
display
tubes of World War II, twenty years earlier, reflecting the fact that some key
pioneers of
minicomputing at MIT's Lincoln Labs were former radar technicians. Across the
continent in
that same year of 1962, another former radar technician was beginning to blaze
a different
trail at Stanford Research Institute. His name was Doug Engelbart. He had been
inspired by
both his personal experiences with these very early graphical displays and by
Vannevar
Bush's seminal essay As We May Think, which had presented in 1945 a vision of
what we
would today call hypertext.
[0018] In December 1968, Engelbart and his team from SRI gave a 90-minute
public
demonstration of the first hypertext system, NLS/AugmentTM. The demonstration
included
the debut of the three-button mouse (Engelbart's invention), graphical
displays with a
multiple-window interface, hyperlinks, and on-screen video conferencing. This
demo was a
sensation with consequences that would reverberate through computer science
for a
quarter century, up to and including the invention of the World Wide Web in
1991.
[0019] So, as early as the 1960s it was already well understood that
graphical
presentation could make for a compelling user experience. Pointing devices
equivalent to
the mouse had already been invented, and many mainframes of the later 1960s
had display
capabilities comparable to those of the PDP1TM. One of your authors retains
vivid
memories of playing another very early video game in 1968, on the console of a
Univac."'
1108 mainframe that would cost nearly forty-five million dollars if you could
buy it today in
2004. But at $45M a throw, there were very few actual customers for
interactive graphics
The custom
6
CA 02684020 2014-05-09
hardware of the NLS/AugmentTM system, while less expensive, was still
prohibitive for
general use. Even the PDP1, costing a hundred thousand dollars, was too
expensive a
machine on which to found a tradition of graphical programming.
[0020] Video games became mass-market devices earlier than computers
because
they ran hardwired programs on extremely cheap and simple processors. But on
general-
purpose computers, oscilloscope displays became an evolutionary dead end. The
concept
of using graphical, visual interfaces for normal interaction with a computer
had to wait a few
years and was actually ushered in by advanced graphics-capable versions of the
serial-line
character VDT in the late 1970s.
[0021] Since the earliest PARCTM systems in the 1970s, the design of GUIs
has been
almost completely dominated by what has come to be called the WIMP (Windows,
Icons,
Mice, Pointer) model pioneered by the AIt0TM. Considering the immense changes
is in
computing and display hardware over the ensuing decades, it has proven
surprisingly
difficult to think beyond the WIMP.
[0022] A few attempts have been made. Perhaps the boldest is in VR (virtual
reality)
interfaces, in which users move around and gesture within immersive graphical
3-D
environments. VR has attracted a large research community since the mid-1980s.
While the
computing power to support these is no longer expensive, the physical display
devices still
price VR out of general use in 2004. A more fundamental problem, familiar for
many years
to designers of flight simulators, is the way VR can confuse the human
proprioceptive
system; VR motion at even moderate speeds can induce dizziness and nausea as
the brain
7
CA 02684020 2014-05-09
tries to reconcile the visual simulation of motion with the inner ear's report
of the body's real-
-
world motions.
[0023] Jef Raskin's THE TM project (The Humane Environment) is exploring
the zoom
world model of GUIs, described in that spatializes them without going 3D. In
THE the
screen becomes a window on a 2-D virtual world where data and programs are
organized
by spatial locality. Objects in the world can be presented at several levels
of detail
depending on one's height above the reference plane, and the most besic
selection'
operation is to zoom in and land on them.
[0024] The LifestreamsTM project at Yale University goes in a completely
opposite
direction, actually de-spatializing the GUI. The user's documents are
presented as a kind of
world-line or temporal stream which is organized by modification date and can
be filtered in
various ways.
[0025] All three of these approaches discard conventional filesystems in
favor of a
context that tries to avoid naming things and using names as the main form of
reference
This makes them difficult to match with the filesystems and hierarchical
namespaces of
Unix'sTM architecture, which seems to be one of its most enduring and
effective features
Nevertheless, it is possible that one of these early experiments may yet prove
as seminal as
Engelbart's 1968 demo of NLS/AugmentTM.
[0026] There is a need in the field of user interfaces for an improved
system and
method of a Human-Machine-Interface.
8
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
SUMMARY OF THE INVENTION
[0027] According to some embodiments of the present invention, there is
provided a 3D human machine interface ("3D HMI"), which 3D HMI may include
(1) an image acquisition assembly, (2) an initializing module, (3) an image
segmentation module, (4) a segmented data processing module, (5) a scoring
module, (6) a projection module, (7) a fitting module,(8) a scoring and error
detection module, (9) a recovery module, (10) a three dimensional correlation
module, (11) a three dimensional skeleton prediction module, and (12) an
output
module.
[0028] According to some embodiments of the present invention, the image
acquisition assembly may be adapted to acquire a set of images, wherein
, substantially each image is associated with a different point in time.
According to
some further embodiments of the present invention, the images may be of a
single user or multiple users.
[0029] According to some embodiments of the present invention, the
initialization
module may be adapted to detect and define the user's (1) colors, (2) organ's
parameters, surrounding, and other parameters which are associated with the
user.
[0030] According to some embodiments of the present invention, the user may
be any person and/or animal and/or moving object which enters the frame.
[0031] According to some embodiments of the present invention, the image
segmentation module may be adapted to extract segmented data from the
image. According to yet further embodiments of the present invention, the
segmented data may also comprise:
9
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
= Color
= Movement
= Edge detection
= Texture.
[0032] According to some embodiments of the present invention, the segmented
data processing module may be adapted to process the segmented data.
According to yet further embodiments of the present invention, the segmented
data may be processed in the following way:
= Color ¨ using known color parameters to detect elements and/or light
changes, for example, use skin color to detect palms and face.
= Movement ¨ detecting moving elements in the frame.
= Background removal.
= Edge detection ¨ detect the edges of the image.
= Texture- using known texture parameters to detect elements.
[0033] According to some embodiments of the present invention, the segmented
data processing module may be adapted to detect deviation in the distance of
an
organ from the image acquisition assembly, in accordance with the deviation of
the organs relative size.
[0034] According to some embodiments of the present invention, the scoring
module may be adapted to (1) examine the processed segmented data, (2)
estimate the quality of the processed segmented data, and according to the
quality (3) decide which portions of the segmented data are reliable enough to
be
used by the HMI system.
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0035] According to some embodiments of the present invention, the three
dimensional skeleton prediction module may be adapted to predict the position
of
the three dimensional skeleton which will have the best match or correlation
with
the processed image.
[0036] According to further embodiments of the present invention, the three
dimensional prediction module may use constraints which derive from the type
of
skeleton used, for example, if the skeleton is of a human figure, the head of
the
skeleton can't rotate 360 degrees.
[0037] According to yet further embodiments of the present invention, the
three
dimensional prediction module may also use a set of dynamic and motion
process to predict the position of the three dimensional skeleton.
[0038] According to some embodiments of the present invention, the projection
module may be adapted to project the skeleton onto the image. According to
some further embodiments of the present invention, the projection may be
applied in the two-dimensional plane.
[0039] According to some embodiments of the present invention, the fitting
module may be adapted to fit segmented data to the projected skeleton.
According to some further embodiments of the present invention, the fitting
module may be adapted to associate portions of the segmented data with
portions of the projected skeleton.
[0040] According to some embodiments of the present invention, the scoring and
error detection module may be adapted (1) to examine the processed skeleton
after it was associated with segmented data, (2) to evaluate the fitting
quality of
11
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
said skeleton and (3) determine whether an error has occurred during the
skeleton prediction process or the association of segmented data.
[0041] According to some embodiments of the present invention, the recovery
module may be adapted to recover from a detected error. According to some
further embodiments of the present invention, the recovery may be a process of
multiple processing layers, re segmenting the image, using the 3D skeleton
motion history to re predict correct position, re projecting and re fitting
the 3D
skeleton. The recovery module may also decide to skip a frame if the image
information is corrupt.
[0042] According to some embodiments of the present invention, the three
dimensional correlation module may be adapted to update the position of
the three dimensional skeleton in accordance with the position of the fitted
skeleton.
[0043] According to some further embodiments of the present invention,
said updating process associates the 3D skeleton on the fitted skeleton,
fits between the 3D skeleton and the fitted skeleton, and updates the 3D
skeleton to the correct position.
BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The subject matter regarded as the invention is particularly pointed
out
and distinctly claimed in the concluding portion of the specification. The
invention, however, both as to organization and method of operation, together
with objects, features, and advantages thereof, may best be understood by
12
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
reference to the following detailed description when read with the
accompanying
drawings in which:
[0045] Fig. 1, there is shown a block diagram depicting a system in accordance
with some embodiments of the present invention.
[0046] FIG. 2, there is shown a flow-chart depicting the steps of an HMI
system
in accordance with some embodiments of the present invention.
[0047] FIG. 3, there is shown a block diagram depicting a system in accordance
with some embodiments of the present invention.
[0048] FIG. 4, there is shown a flow-chart depicting the steps of an HMI
system ,
in accordance with some embodiments of the present invention.
[0049] FIG. 5, there is shown a block diagram depicting a system in accordance
with some embodiments of the present invention.
[0050] FIG. 6, there is shown a flow-chart depicting the steps of an HMI
system
in accordance with some embodiments of the present invention.
[0051] It will be appreciated that for simplicity and clarity of illustration,
elements
shown in the figures have not necessarily been drawn to scale. For example,
the
dimensions of some of the elements may be exaggerated relative to other
elements for clarity. Further, where considered appropriate, reference
numerals
may be repeated among the figures to indicate corresponding or analogous
elements.
13
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
DETAILED DESCRIPTION
[0052] In the following detailed description, numerous specific details are
set
forth in order to provide a thorough understanding of the invention. However,
it
will be understood by those skilled in the art that the present invention may
be
practiced without these specific details. In other instances, well-known
methods,
procedures, components and circuits have not been described in detail so as
not
to obscure the present invention.
[0053] Unless specifically stated otherwise, as apparent from the following
discussions, it is appreciated that throughout the specification discussions
utilizing terms such as "processing", "computing", "calculating",
"determining", or
the like, refer to the action and/or processes of a computer or computing
system,
or similar electronic computing device, that manipulate and/or transform data
represented as physical, such as electronic, quantities within the computing
system's registers and/or memories into other data similarly represented as
physical quantities within the computing system's memories, registers or other
such information storage, transmission or display devices.
[0054] Embodiments of the present invention may include apparatuses for
performing the operations herein. This apparatus may be specially constructed
for the desired purposes, or it may comprise a general purpose computer
selectively activated or reconfigured by a computer program stored in the
computer. Such a computer program may be stored in a computer readable
storage medium, such as, but is not limited to, any type of disk including
floppy
disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories
(ROMs), random access memories (RAMs) electrically programmable read-only
14
CA 02684020 2009-10-14
WO 2008/126069
PCT/1L2007/000475
memories (EPROMs), electrically erasable and programmable read only
memories (EEPROMs), magnetic or optical cards, or any other type of media
suitable for storing electronic instructions, and capable of being coupled to
a
computer system bus.
[0055] According to some embodiments of the present invention, there is
provided a 3D human machine interface ("3D HMI"), which 3D HMI may include
(1) an image acquisition assembly, (2) an initializing module, (3) an image
segmentation module, (4) a segmented data processing module, (5) a scoring
module, (6) a projection module, (7) a fitting module,(8) a scoring and error
detection module, (9) a recovery module, (10) a three dimensional correlation
module, (11) a three dimensional skeleton prediction module, and (12) an
output
module.
[0056] According to some embodiments of the present invention, the image
acquisition assembly may be adapted to acquire a set of images, wherein
= substantially each image is associated with a different point in time.
According to
some further embodiments of the present invention, the images may be of a
single user or multiple users.
[0057] According to some embodiments of the present invention, the
initialization
module may be adapted to detect and define the user's (1) colors, (2) organ's
parameters, surrounding, and other parameters which are associated with the
user and decide on the best way for image segmentation in the next steps
(thresholds, score for every image segmentation etc.)
[0058] According to some embodiments of the present invention, the image
segmentation module may be adapted to extract segmented data from the
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
image. According to yet further embodiments of the present invention, the
segmented data may also comprise:
= Color
= Movement
= Edge detection
= Texture.
[0059] According to some embodiments of the present invention, the segmented
data processing module may be adapted to process the segmented data.
According to yet further embodiments of the present invention, the segmented
data may be processed in the following way:
= Color ¨ using known color parameters to detect elements and/or light
changes, for example, use skin color to detect palms and face.
= Movement ¨ detecting moving elements in the frame.
= Background removal.
= Edge detection ¨ detect the edges of the image.
= Texture- using known texture parameters to detect elements.
[0060] According to some embodiments of the present invention, the scoring
module may be adapted to (1) examine the processed segmented data, (2)
estimate the quality of the processed segmented data, and according to the
quality (3) decide which portions of the segmented data are reliable enough to
be
used by the HMI system.
[0061] According to some embodiments of the present invention, the three
dimensional skeleton prediction module may be adapted to predict the position
of
16
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
the three dimensional skeleton which will have the best match or correlation
with
the processed image.
[0062] According to further embodiments of the present invention, the three
dimensional prediction module may use constraints which derive from the type
of
skeleton used, for example, if the skeleton is of a human figure, the head of
the
skeleton can't rotate 360 degrees.
[0063] According to yet further embodiments of the present invention, the
three
dimensional prediction module may use a set of dynamic and motion process to
predict the position of the three dimensional skeleton.
[0064] According to some embodiments of the present invention, the projection
module may be adapted to project the skeleton onto the image. According to
some further embodiments of the present invention, the projection may be
applied in the two-dimensional plane.
[0065] According to some embodiments of the present invention, the fitting
module may be adapted to fit segmented data to the projected skeleton.
According to some further embodiments of the present invention, the fitting
module may be adapted to associate portions of the segmented data with
portions of the projected skeleton.
[0066] According to some embodiments of the present invention, the scoring and
error detection module may be adapted (1) to examine the processed skeleton
after it was associated with segmented data, (2) to evaluate the fitting
quality of
said skeleton and (3) determine whether an error has occurred during the
skeleton prediction process or the association of segmented data.
17
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0067] According to some embodiments of the present invention, the recovery
module may be adapted to recover from a detected error. According to some
further embodiments of the present invention, the recovery may be a process of
multiple processing layers, re segmenting the image, using the 3D skeleton
motion history to re predict correct position, re projecting and re fitting
the 3D
skeleton. The recovery module may also decide to skip a frame if the image
information is corrupt.
[0068] According to some embodiments of the present invention, the three
dimensional correlation module may be adapted to update the position of
the three dimensional skeleton in accordance with the position of the fitted
skeleton.
[0069] According to some further embodiments of the present invention,
said updating process associates the 3D skeleton on the fitted skeleton,
fits between the 3D skeleton and the fitted skeleton, and updates the 3D
skeleton the correct position.
[0070] Turning now to Figure 1, there is shown an exemplary HMI system in
accordance with some embodiments of the present invention, which system may
be best described in conjunction with Figure 2, there is shown a flow chart
depicting the steps of such an HMI system.
[0071] According to some embodiments of the present invention, Figure 1 shows
a 3D human machine interface ("3D HMI"), which 3D HMI may include (1) an
image acquisition assembly 1000, (2) an initializing module 1100, (3) an image
segmentation module 1200, (4) a segmented data processing module 1300, (5)
a scoring module 1400, (6) a projection module 1450, (7) a fitting module
18
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
1500,(8) a scoring and error detection module 1550, (9) a recovery module
1600,
(10) a three dimensional correlation module 1700, (11) a three dimensional
skeleton prediction module 1800, and (12) an output module 1900.
[0072] According to some embodiments of the present invention, the image
acquisition assembly may be adapted to acquire a set of images, as seen in
step
2000, wherein substantially each image is associated with a different point in
time. According to some further embodiments of the present invention, the
images may be of a single user or multiple users.
[0073] According to yet further embodiments of the present invention the image
acquisition assembly may comprise of a digital camera, a web camera, a film'
camera, a video camera, a web camera, a digital video camera, an analogue
video camera, a stereo-camera and/or any other camera known today or to be
derived in the future.
[0074] According to some embodiments of the present invention, after the
system
has acquired one or more images, the system may enter an initialization phase,
step 2100, which is performed by the initialization module 1100. Which
initialization module may be adapted to detect and define the user's (1)
colors,
(2) organ's parameters, (3) surrounding, and other parameters which are
associated with the user.
[0075] According to some embodiments of the present invention, the system may
be adapted to extract segmentation data, as shown in step 2200, which
segmented data may comprise:
= Color
= Movement
= Edge detection
= Texture.
19
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0076] According to yet further embodiments of the present invention, the
image
segmentation module 1200 may be adapted to extract the segmented data from
the image.
[0077] According to some embodiments of the present invention, the system may
be adapted to process the segmented data, as shown in step 2300. According to
yet further embodiments of the present invention, the segmented data may be
processed in the following way:
= Color ¨ using known color parameters to detect elements and/or light
changes, for example, use skin color to detect palms and face.
= Movement ¨ detecting moving elements in the frame.
= Background removal.
= Edge detection ¨ detect the edges in the image.
= Texture- using known texture parameters to detect elements.
[0078] According to yet further embodiments of the present invention, the
segmented data processing module, 1300, may be adapted to process the
segmented data.
[0079] According to some embodiments of the present invention, the system may
be adapted to evaluate the quality of the segmented data, as shown in step
2400, the evaluation is performed by (1) examining the processed segmented
data, (2) estimating the quality of the processed segmented data, and
according
to the estimated quality (3) decide which portions of the segmented data are
reliable enough to be used by the HMI system.
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0080] According to yet further embodiments of the present invention, the
scoring
module 1400, may be adapted evaluate the quality of the segmented
information.
[0081] According to further embodiments of the present invention, the system
may be adapted to predict the position of the three dimensional skeleton, as
shown in step 2800, which position will have the best match or correlation
with
the processed image. According to some further embodiments of the present
invention the prediction may be more accurate with the use of constraints
which
derive from the type of skeleton used, for example, if the skeleton is of a
human
figure, the head of the skeleton can't rotate 360 degrees without a motion of
the
shoulders.
[0082] According to some embodiments of the present invention, the prediction
sequence may also use a set of dynamic and motion process and so on.
[0083] According to some embodiments of the present invention, the three
dimensional skeleton prediction module 1800 may be adapted to predict the
position of the three dimensional skeleton.
[0084] According to some embodiments of the present invention, the system may
be further adapted to project the skeleton onto the image, as shown in step
2450. According to some further embodiments of the present invention, the
projection may be applied in the two-dimensional plane.
[0085] According to some embodiments of the present invention, the projection
module, 2450, may be adapted to project the skeleton onto the image.
[0086] According to some embodiments of the present invention, the system may
be further adapted to fit the segmented data with the projected skeleton, as
21
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
shown in step 2500. According to some further embodiments of the present
invention, the fitting process may comprise the association of portions of the
segmented data with portions of the projected skeleton.
[0087] According to some embodiments of the present invention, fitting the
segmented data may comprise associating portions of the extracted segmented
data with current skeleton parameters, which current skeleton parameters may
support the associated portions of extracted segmented data.
[0088] According to some further embodiments of the present invention, the
outcome of this process is a "fitted skeleton".
[0089] According to some further embodiments of the present invention, the
fitting module, 2500, may be adapted to associate the segmented data with the
projected skeleton.
[0090] According to some embodiments of the present invention, the system may
be further adapted to give score to the fitted skeleton and detect errors, as
shown in step 2550. According to some embodiments of the present invention,
giving score and detecting errors may comprise of (1) examining the fitted
skeleton, (2) evaluating the fitting quality of said skeleton and (3)
determining
whether an error has occurred during the skeleton prediction process or the
association of segmented data.
[0091] According to some embodiments of the present invention, the scoring and
error detection module 1550, may be adapted to give score and detect errors.
[0092] According to some embodiments of the present invention, if an error was
detected during step 2550, the system may enter a recovery phase, as shown in
step 2600. The recover process may be a process of multiple processing layers.
22
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0093] According to some embodiments of the present invention, the recovery
phase may comprise re-segmenting the image, re-predicting the 3D skeleton
position, re-projecting and re-fitting the skeleton using extended effort.
According
to yet further embodiments of the present invention, the recovery module may
also decide to skip a frame or more if the image information is corrupt.
[0094] According to some embodiments of the present invention, during recover
the system may be adapted to detect that the object its tracking is not in the
frame. According to yet further embodiments of the present invention, the
system
may be adapted to skip one or more frames until the object is back in the
frame.
[0095] According to yet further embodiments of the present invention, the
recovery phase may direct the system back to the initialization step.
[0096] According to some embodiments of the present invention, the recovery
module 2600 may be adapted to perform the recovery process.
[0097] According to some embodiments of the present invention, if no
error was detected during step 2550, the system may be adapted to
update the position of the three dimensional skeleton in accordance with
the position of the fitted skeleton, as shown in step 2700. According to
some further embodiments of the present invention, the updating process
may comprise (1) projecting the 3D skeleton on the fitted skeleton, (2)
associating the 3D skeleton with the fitted skeleton, and (3) updating the
position of the 3D skeleton.
[0098] According to some embodiments of the present invention, the three
dimensional correlation module, 1700 may be adapted to update the position of
the three dimensional skeleton.
23
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[0099] According to some embodiments of the present invention, the three-
dimensional correlation module 1700 and the skeleton prediction module 1800,
may use some or all of the algorithms and processes which were disclosed in
PCT application serial number.PCT/IL2005/000813, filed on 31 July 2005 under
the same assignee as the present application.
[00100] Turning now to Figure 3, there is shown an exemplary HMI system
in accordance with some embodiments of the present invention, which system
may be best described in conjunction with Figure 4, there is shown a flow
chart
depicting the steps of such an HMI system.
[00101] According to some embodiments of the present invention, Figure 3
shows a 3D human machine interface ("3D HMI"), which 3D HMI may include (1)
a Zlens image acquisition assembly 3000, (2) an initializing module 3100, (3)
an
image segmentation module 3200, (4) a segmented data processing module
3300, (5) a fitting module 3500, (6) a scoring module 3550, (7)a three
dimensional correlation module 3700, and (8) an output module 3900.
[00102] According to some embodiments of the present invention, the Zlens
acquisition assembly may be adapted to acquire a set of images, as seen in
step
4000 wherein substantially each image is associated with a different point in
time. According to some further embodiments of the present invention, the
images may be of a single user or multiple users.
[00103] According to some embodiments of the present invention, the Zlens
acquisition assembly may be mounted on another image acquisition assembly,
i.e. element 1000 of Fig.1.
24
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[00104] According to yet further embodiments of the present invention the
Zlens acquisition assembly (3000) may be best described in conjunction with
PCT/IL2006/001254 filed on October 31 2006 under the same assignee as the
present application and with US Patent application 60/731,274 US filed on
October 31 2005 under the same assignee as the present application.
[00105] According to some embodiments of the present invention, the
system is further adapted to enter an initialization phase, as shown in step
4100,
which is performed by the initialization module 3100. Which initialization
module
may be adapted to detect and define the user's (1) colors, (2) organ's
parameters, surroundings, (3) and other parameters which are associated with
the user.
[00106] According to some embodiments of the present invention, the
system may be adapted to extract segmentation data, as shown in step 4200,
which segmented data may comprise:
= Color
= Movement
= Edge detection
= Texture.
[00107] According to yet further embodiments of the present invention, the
image segmentation module 3200 may be adapted to extract the segmented
data from the image.
[00108] According to some embodiments of the present invention, the
system may be adapted to process the segmented data, as shown in step 4300.
According to yet further embodiments of the present invention, the segmented
data may be processed in the following way:
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
= Color ¨ using known color parameters to detect elements and/or light
changes, for example, use skin color to detect palms and face.
= Movement ¨ detecting moving elements in the frame.
= Background removal.
= Edge detection ¨ detect the contours of every organ.
= Texture- using known texture parameters to detect elements.
[00109] According to yet further embodiments of the present invention, the
segmented data processing module, 3300, may be adapted to process the
segmented data.
[00110] According to some embodiments of the present invention, the
system may be further adapted to fit portions of the extracted segmented data
with the acquired image, as shown in step 4500. According to some further
embodiments of the present invention, the fitting process may comprise
associating portions of the extracted segmented data with dedicated areas of
the
acquired image.
[00111] According to yet further embodiment of the present invention, the
dedicated areas may be stored in the system or may be determined during the
initialization phase. According to yet further embodiment of the present
invention,
the dedicated areas may be specific organs of the user (hands, head, feet) or
any other element which may be acquired during step 3000.
[00112] According to yet further embodiment of the present invention, the
fitting process may comprise testing whether the extracted segmented data
defines parameters which are relevant to the dedicated areas.
26
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[00113] According to some further embodiments of the present invention,
the fitting module, 3500, may be adapted to the associate portions of the
segmented data with the acquired image.
[00114] According to some further embodiments of the present invention,
the outcome of this process is a "fitted image".
[00115] According to some embodiments of the present invention, the
system may be further adapted to evaluate the quality of the fitted segmented
data, as shown in step 4550. According to some embodiments of the present
invention, evaluating the quality of the fitted segmented data may comprise of
(1)
examining the processed segmented data, (2) estimating the quality of the
processed segmented data, and according to the estimated quality (3) decide
which portions of the segmented data are reliable enough to be used by the HMI
system, (4) examine the fitted image, (5) evaluate the fitting quality of said
image
and (6) determining whether an error has occurred during the association of
segmented data.
[00116] According to some embodiments of the present invention, the
scoring module, 3550, may be adapted to evaluate the quality of the fitted
segmented data.
[00117] According to some further embodiments of the present invention,
the system may comprise an error detection mechanism and a recovery
mechanism as was described hereinabove.
[00118] According to some embodiments of the present invention, the
system may be adapted to update the position of a three dimensional body in
accordance with the fitted image and the extrapolation of a depth map using
the
27
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
Zlens image acquisition assembly , as shown in step 4700. According to some
further embodiments of the present invention, the updating process may
comprise associating the extracted depth map with the extracted segmented
data, and (3) updating the position of the three-dimensional body of the
output
model.
[00119] According to some embodiments of the present invention, the three
dimensional correlation module, 3700 may be adapted to update the position of
the three dimensional body.
[00120] According to some embodiments of the present invention, the
functionality of the three-dimensional correlation module 3700 and the Zlens
image acquisition assembly 3000 and particularly the extrapolation of depth
from
an image acquired using the Zlens apparatus may best be described in
conjunction with PCT/IL2006/001254 filed on October 31 2006 under the same
assignee as the present application and with US Patent application 60/731,274
US filed on October 31 2005 under the same assignee as the present
application.
[00121] Turning now to Figure 5, there is shown an exemplary HMI system
in accordance with some embodiments of the present invention, which system
may be best described in conjunction with Figure 6, there is shown a flow
chart
depicting the steps of such an HMI system.
[00122] According to some embodiments of the present invention, Figure 5
shows a 3D human machine interface ("3D HMI"), which 3D HMI may include (1)
a Zlens acquisition assembly 5000, (2) an initializing module 5100, (3) an
image
segmentation module 5200, (4) a segmented data processing module 5300, (5)
28
CA 02684020 2009-10-14
WO 2008/126069
PCT/1L2007/000475
a scoring module 5400, (6) a projection module 5450, (7) a fitting module
5500,(8) a scoring and error detection module 5550, (9) a recovery module
5600,
(10) a three dimensional correlation module 5700, (11) a three dimensional
skeleton prediction module 5800, (12) an output module 5900 and an optional
(13) depth extraction module 5050.
[00123] According to some embodiments of the present invention, the Zlens
acquisition assembly may be adapted to acquire a set of images, as seen in
step
6000, wherein substantially each image is associated with a different point in
time. According to some further embodiments of the present invention, the
images may be of a single user or multiple users.
[00124] According to some embodiments of the present invention, the Zlens
acquisition assembly may be mounted on another image acquisition assembly,
i.e. element 1000 of Fig.1.
[00125] According to yet further embodiments of the present invention the
Zlens acquisition assembly (5000) may be best described in conjunction with
PCT/IL2006/001254 filed on October 31 2006 under the same assignee as the
present application and with US Patent application 60/731,274 US filed on
October 31 2005 under the same assignee as the present application.
[00126] According to some embodiments of the present invention, after the
system has acquired one or more images, the system may enter an initialization
phase, step 6100, which is performed by the initialization module 5100. Which
initialization module may be adapted to detect and define the user's (1)
colors,
(2) organ's parameters, (3) surroundings, and other parameters which are
associated with the user.
29
CA 02684020 2009-10-14
WO 2008/126069
PCT/1L2007/000475
[00127] According to some embodiments of the present invention, the
system may be adapted to extract segmentation data, as shown in step 6200,
which segmented data may comprise:
= Color
= Movement
= Edge detection
= Texture.
[00128] According to yet further embodiments of the present invention, the
image segmentation module 5200 may be adapted to extract the segmented
data from the image.
[00129] According to some embodiments of the present invention, the
system may be adapted to process the segmented data, as shown in step 6300.
According to yet further embodiments of the present invention, the segmented
data may be processed in the following way:
= Color ¨ using known color parameters to detect elements and/or light
changes, for example, use skin color to detect palms and face.
= Movement ¨ detecting moving elements in the frame.
= Background removal.
= Edge detection ¨ detect the contours of every organ.
= Texture - using known texture parameters to detect elements.
[00130] According to yet further embodiments of the present invention, the
segmented data processing module, 5300, may be adapted to process the
segmented data.
[00131] According to some embodiments of the present invention, the
system may be adapted to evaluate the quality of the segmented data, as shown
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
in step 6400, the evaluation is performed by (1) examining the processed
segmented data, (2) estimating the quality of the processed segmented data,
and according to the estimated quality (3) decide which portions of the
segmented data are reliable enough to be used by the HMI system.
[00132] According to yet further embodiments of the present invention, the
scoring module, 5400, may be adapted evaluate the quality of the segmented
information.
[00133] According to further embodiments of the present invention, the
system may be adapted to predict the position of the three dimensional
skeleton,
as shown in step 6800, which position will have the best match or correlation
with
the processed image. According to some further embodiments of the present
invention the prediction may be more accurate with the use of constraints
which
derive from the type of skeleton used, for example, if the skeleton is of a
human
figure, the head of the skeleton can't rotate 360 degrees without a motion of
the
shoulders.
[00134] According to some embodiments of the present invention, the
prediction sequence may also use a set of dynamic and motion process.
[00135] According to some embodiments of the present invention, the three
dimensional skeleton prediction module 5800 may be adapted to predict the
position of the three dimensional skeleton.
[00136] According to some embodiments of the present invention, the
system may be further adapted to extract depth using the Zlens acquisition
assembly, as shown in step 6050, the extraction of depth using a Zlens
acquisition assembly is described in (1) PCT/IL2006/001254 filed on October 31
31
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
2006 under the same assignee as the present application and with (2)US Patent
application 60/731,274 US filed on October 31 2005 under the same assignee as
the present application.
[00137] According to some embodiments of the present invention, the
depth extraction module, 5050, may be adapted to extract depth from the
acquired image.
[00138] According to some embodiments of the present invention, the
system may be further adapted to project the skeleton onto the image, as shown
in step 6450. According to some further embodiments of the present invention,
the projection may be applied in the two-dimensional plane.
[00139] According to yet further embodiments of the present invention, the
projection of the skeleton may be applied in the three-dimensional plane if
module 5050 is used.
[00140] According to some embodiments of the present invention, the
projection may be onto a three-dimensional image and/or a three dimensional
cloud of points.
[00141] According to some embodiments of the present invention, the
projection module, 6450, may be adapted to project the skeleton onto the
image.
[00142] According to some embodiments of the present invention, the
system may be further adapted to fit the segmented data with the projected
skeleton, as shown in step 6500. According to some further embodiments of the
present invention, the fitting process may comprise the association of
portions of
the segmented data with portions of the projected skeleton.
32
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[00143] According to some embodiments of the present invention, fitting
the
segmented data may comprise associating portions of the extracted segmented
data with current skeleton parameters, which current skeleton parameters may
support the associated portions of extracted segmented data.
[00144] According to some further embodiments of the present invention,
the outcome of this process is a "fitted skeleton".
[00145] According to some further embodiments of the present invention,
the fitting module, 5500, may be adapted to associate the segmented data with
the projected skeleton.
[00146] According to some embodiments of the present invention, the
system may be further adapted to give score to the fitted skeleton and detect
errors, as shown in step 6550. According to some embodiments of the present
invention, giving score and detecting errors may comprise of (1) examining the
fitted skeleton, (2) evaluating the fitting quality of said skeleton and (3)
determining whether an error has occurred during the skeleton prediction
process or the association of segmented data.
[00147] According to some embodiments of the present invention, the
scoring and error detection module, 5550, may be adapted to give score and
detect errors,
[00148] According to some embodiments of the present invention, if an
error was detected during step 6550, the system may enter a recovery phase, as
shown in step 6600. The recovery process may be a process of multiple
processing layers.
33
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
[00149] According to some embodiments of the present invention, the
recovery phase may comprise re-segmenting the image, re-predicting the 3D
skeleton position, re-projecting and re-fitting the skeleton using extended
effort.
According to yet further embodiments of the present invention, the recovery
module may also decide to skip a frame or more if the image information is
corrupt.
[00150] According to some embodiments of the present invention, during
recovery the system may be adapted to detect that the object its tracking is
not in
the frame. According to yet further embodiments of the present invention, the
system may be adapted to skip one or more frames until the object is back in
the
frame.
[00151] According to yet further embodiments of the present invention, the
recovery phase may direct the system back to the initialization step.
[00152] According to some embodiments of the present invention, the
recovery module 5600 may be adapted to perform the recovery process.
[00153] According to some embodiments of the present invention, if
no error was detected during step 6550, the system may be adapted to
update the position of the three dimensional skeleton in accordance with
the position of the fitted skeleton, as shown in step 6700. According to
some further embodiments of the present invention, the updating process
may comprise (1) projecting the 3D skeleton on the fitted skeleton, (2)
associating the three dimensional skeleton with the fitted skeleton, (3)
extract depth using the Zlens assembly, (4) associating the three-
34
CA 02684020 2009-10-14
WO 2008/126069 PCT/1L2007/000475
=
dimensional skeleton with depth parameters and (5) updating the position
of the 3D skeleton.
[00154] According to some embodiments of the present invention, the three
dimensional correlation module, 5700 may be adapted to update the position of
the three dimensional skeleton.
[00155] According to some embodiments of the present invention, the
three-dimensional correlation module 5700 and the skeleton prediction module
5800, may use some or all of the algorithms and processes which were disclosed
in PCT application serial number.PCT/IL2005/000813, filed on 31 July 2005
under the same assignee as the present application.
[00156] According to some further embodiments of the present invention,
the functionality of (1) the three-dimensional correlation module 5700 (2) the
Zlens image acquisition assembly 5000 the (3) depth extraction module 5050
and (4) particularly the extrapolation of depth from an image acquired using
the
Zlens apparatus may best be described in conjunction with PCT/IL2006/001254
filed on October 31 2006 under the same assignee as the present application
and with US Patent application 60/731,274 US filed on October 31 2005 under
the same assignee as the present application.
[00157] According to some embodiments of the present invention, the
systems described hereinabove may be adapted to receive from an exterior
source depth images and/or three-dimensional images. According to yet further
embodiments of the present invention, if a depth images and/or three-
dimensional images is received the system is adapted to extract its parameters
in the relevant modules.
CA 02684020 2014-05-09
[00158] The processes and displays presented herein are not inherently
related to any
particular computer or other apparatus. Various general purpose systems may be
used with
programs in accordance with the teachings herein, or it may prove convenient
to construct a
more specialized apparatus to perform the desired method. The desired
structure for a
variety of these systems will appear from the description below. In addition,
embodiments of
the present invention are not described with reference to any particular
programming
language. It will be appreciated that a variety of programming languages may
be used to
implement the teachings of the inventions as described herein. One of ordinary
skill in the
art should understand that the described invention may be used for all kinds
of wireless or
wire-line system.
[00159] While certain features = of the invention have been illustrated and
described
herein, many modifications, substitutions, changes, and equivalents will now
occur to those
skilled in the art. It is, therefore, to be understood that the appended
claims are intended to
cover all such modifications and changes to the extent they fall within the
scope of the
apparatuses, systems and methods defined by the limitations recited in these
claims.
36
