Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
Description
This invention is in the field of wearable electronic devices. It deals with
an interface for
controlling user interfaces for different types of electronic devices. This
can include
everyday portable electronic devices like a smartphone, tablet, laptop or
desktop
computer. Another these types of devices are displays like television screens,
smart
TVs, and TV set-top boxes like Apple TV, Google Chromecast, Amazon Fire stick,
or
Roku TV stick, etc. Another type of device are smart home electronics such as
robot
vacuum cleaners, smart blinds, smart home lighting, or smart thermostat, etc.
Still
another type of devices are virtual reality or augmented reality headsets. The
use of
this invention is not limited to these types of device categories, which are
listed to give
examples of applications for this invention.
Consumer electronic devices have advanced greatly in the past decade, taking
on
smaller, low energy form factors, with increased capabilities made possible by
wireless
technologies such as Bluetooth, IEEE 802.11 ("Wifi") and cellular networks.
However,
methods for controlling such devices are still limited to predominantly
tactile methods
such as keyboards, mouse, physical buttons, touchscreens or pointers. Control
when
direct proximity is not available is limited to audio cues such as voice
commands or
rudimentary hand gestures such as waving or clapping. This invention expands
upon
these methods by using a finger mounted wearable controlled wireless
accelerometer
sensor to relay precise and complex finger gestures and interpret them using a
combination of conventional signal processing and deep neural networks. This
allows a
wide array of fine natural and intuitive gestures to be detected and
classified with high
proficiency to allow for a high degree of device control, including operation
of devices
and text entry. The invention allows for intuitive text entry and adds a
method for easily
correcting errors in text entry, while at the same time improving the gesture
recognition
capabilities of the invention.
The concept of the invention is shown in figure 1. It makes use of a sensor
platform, a
compact wireless device with a built-in motion detector (such as an
accelerometer or
gyroscope) worn on the user's finger or hand, such as a ring, or bracelet, or
an
implanted sensor, that can communicate wirelessly to the main computing
device, such
as a laptop computer or smart phone, using a wireless digital communications
protocol
such as Wifi, or Bluetooth. The invention consists of the software operating
on
wearable device and on the computing device and the integration of the two,
which
allows the sensor platform and the main computing device to work together to
recognize
complex hand and figure motions, allowing it to take the user's finger inputs
and take
actions accordingly (open applications, insert or modify text, control an
additional
remote device, etc.).
The general operation of the invention has the wearable finger sensor
platform's
accelerometer data sent to a low power microprocessor on the wearable sensor
platform. The low power microprocessor takes this data and can perform some
basic
signal processing and decision making with this data- such as wake from sleep
mode,
Date Recue/Date Received 2022-01-28
or change to other mode of operation. When appropriate, a stream of data,
which can
include the accelerometer data and data that can be derived from the
accelerometer
data (such as filtered data, or transformed into frequency domain or wavelet
transformation) is sent to the main computing device, where more advanced
signal
processing takes place. This signal processing involves multiple layers of
conventional
signal processing (such as basic time or frequency domain filters, wavelet
transforms
etc.), and deep neural networks (using algorithms such as convolutional neural
networks, recursive neural networks such as LSTM or GRU, as well as fully
connected
neural networks). A high-level decision is made based on the processing of
this data,
which can be to determine if the accelerometer data represents a valid gesture
or
character that should be acted upon, performing the necessary actions based on
the
gesture, and learning from previous gestures to improve the future
functionality of the
device.
The hardware of the sensor platform is shown in Figure 2. It is a small finger
or hand
worn device, which contains several electronic components, such as a motion
detector
(such as an accelerometer or gyroscope), a microprocessor, a power management
circuit, on board data storage (both for storing data, access codes, and
executable
instructions) as well as a radio transmitter for sending the measured motion
data, as
well as for operations such as firmware updates, and device pairing.
The motion data is sent to the main computing device where it is analyzed and
decisions are made based off of the nature of the data. This includes the
detection and
classification of gestures and characters, with actions taken accordingly
(such as the
insertion of a character in a line of text, or the operation of a smart
device). In order to
make such a determination given the complex nature of the motion data, a deep
processing stack is required. Parts of this stack may exist on the sensor
platform and
the main computing device.
A sample of the data processing stack that may be on either or both of the
sensor
platform and the main computing device is shown in Figure 3. Motion data taken
from
the motion sensor on the sensor platform is given to a preprocessing layer,
which can
perform basic operations such as scaling, interpolating, decimating,
normalization.
Additional data streams may be derived from this stream as well, which can
include
differentiation, integration, weighted averaging, etc. This data is fed into a
processing
stack which performs filtering operations on the data. These could be
conventional
signal processing filters such as IIR, or FIR filter, FFT transforms, or non-
linear
operations (such as exponentiation, power, logarithmic, etc.). Alternatively,
they can be
neural network layers to make up a deep neural network, which could include
convolutional neural network (CNN) layers, Recursive Neural Network (RNN)
layers
such as Long Short-Term Memory (LSTM) or GRU units, as well as fully connected
layers. These layers are given numerical weights, bias values, and a non-
linear
activation function, whose values are determined through mathematical training
operations based on a large library of measured and labelled motion data
(including
Date Recue/Date Received 2022-01-28
gesture and character samples). Many of these processing layers are present to
process the data, and the layers are connected one to the other until they
reach the
decision-making layer, which takes the result of the previous layers to make a
determination based on them (such as the detection or classification of a
gesture or
character). These layers may be branched, interconnected, and combined as
needed,
which is determined by a mathematical training operation using the previously
mentioned library of labelled motion data. To improve the efficiency of the
library, lower
level layers can be shared to make different types of determinations, while
the upper
levels can remain distinct.
The invention allows for 4 different modes of operation. These include a
gesture
detection mode, a character detection mode, a fitness tracking mode and a low
power
sleep mode. Figure 4 shows a state diagram of these 4 different modes and how
they
may change from one state to another. One way the state changes may occur is
if
predetermine tap patterns are entered. Once detected, the tap pattern "G" will
cause
the device to enter gesture detection mode, while tap pattern "C" will cause
it to enter
character detection mode, while tap pattern "F" will cause it to enter fitness
tracking
mode, while tap pattern "S" will cause it to enter low power sleep mode. Other
means
of triggering transitions to other modes of operation are possible. The device
sleep
mode can be triggered on a long enough period of inactivity, while context
sensitive
cues can cause the device to move from character to gesture recognition mode
(for
instance, once a character is detected, several candidate characters will be
displayed
for the user to select one, with the device changed to the gesture recognition
mode to
allow the user to use directional gestures to select the desired character).
In the low power sleep mode, most of the device functions, such as the radio
transmitter, are disabled to save power and to prolong battery life. Limited
functionality
exists to detect tap patterns to wake the sensor platform and change it to
another mode
of operation.
In the fitness tracking mode, the device acts as a basic step counter to track
steps or
distance travelled. The motion detector on the sensor platform is used to
monitor the
motions of the user's arm and body to track motions associated with fitness
activity such
as walking or jogging. The signal processing block on the sensor platform can
provide
preliminary signal processing so to keep track of detected steps while
minimizing the
number of radio transmissions to save power.
In the gesture detection mode, gestures detected as shown in the chart in
Figure 5.
Which allows the user to perform many different type of actions such as the
manipulation of remote user interfaces without the need for physical contact.
This can
be when it is impractical, impossible, or undesirable to do so (such as a
screen that is
far away, or only exists virtually in a headset, or if the user's hands are
full, wet or dirty).
The invention can detect a gesture when a predetermined tap pattern is
detected to
start listening for a gesture. The gesture data will be the accelerometer data
(direct and
Date Recue/Date Received 2022-01-28
preprocessed) up until the end of the gesture is detected (such as when the
accelerometer detects no further motion of the wearable sensor platform). This
data is
sent to the main computing device when listens for this gesture and acts
accordingly -
such as opening or closing an application or dialog box, changing an active
selection, or
interacting with a Ul element such as a slider. The gestures include a set to
recognize
commonly performed actives- such as a gesture to accept, enter or start, a
gesture for
cancellation of exit, a directional swipe set of gesture for selection, and
rotational
gestures for incrementing or decrementing values.
The text entry mode of operation operates similarly to the gesture detection
mode of
operation. It is shown in the chart in Figure 6. To make the invention as
intuitive to use
as possible, the use of a virtual keyboard is not desirable. Instead, when the
user
wishes to enter a character (such as a letter, number, or punctuation symbol),
the user
should draw the character directly with their fingers as if they were writing
it in the air.
The invention can detect a character when a predetermined tap pattern is
detected to
start listening for a character. The character data will be the accelerometer
data (direct
and preprocessed) up until the end of the gesture is detected (such as when
the motion
detector detects no further motion of the wearable sensor platform). This data
is sent to
the main computing device when listens for this gesture and acts accordingly.
Due to
the larger set of characters that can be recognized, there will often by a
large margin of
error in selecting the correct character- to accommodate for this, after the
character
entry, the invention will display several top choices based on the character
detection
algorithm and allow the user to choose the desired one with a quick swipe
gesture
(similar to what was described in the gesture detection mode of operation). In
addition
to giving the user an opportunity to correct for errors, this also gives the
system a
chance to save the recorded character entered and the user's actual intent to
a library
of prerecorded characters for user in improving the character detection
algorithm in the
future for all users in general, and for the current user in particular to
recognize their
handwriting style, for instance.
Date Recue/Date Received 2022-01-28
