Note: Descriptions are shown in the official language in which they were submitted.
CA 02531358 2005-12-21
TITLE: Kinesthetic Training System with Composite Feedback
FIELD OF THE INVENTION
This invention relates to the training of individuals carrying out kinesthetic
activities. More particularly, it relates to the provision of feedback from
various
components of movements and the provision of a training environment that can
help automate movements with multi-degrees of freedom. It has application in
the fields of rehabilitation, athletics, recreational activity, and in any
field where
skill in carrying-out a physical motion or procedure must be acquired.
BACKGROUND TO THE INVENTION
The human body can be trained to carry out specific physical motions or
activities through the guided procedure of repeating approximations of the
desired
behavior. In the field of rehabilitation, persons who have lost neuromuscular
functions can relearn how to carry out such basic activities as grasping and
walking, as well as more complex motions, through the process of repetition.
In
the recreational field, skilled players learn through repetition to carry out
critical
motions in the fields of tennis, basketball, golf and baseball to high degrees
of
precision.
It is known to train individuals to carry out a kinesthetic procedure by
providing feedback which is indicative of their level of performance and
progress
towards achieving full competence in respect of the subject activity. A number
of
references in this regard are:
US Patent 6,413,190 to Wood et al
US Patent 6,032,530 to Hock
US Patent 5,989,157 to Walton
US Patent 5,692,517 to Junker
1
CA 02531358 2005-12-21
US Patent 5,679,004 to McGowan et al.
US Patent 5,697,791 to Nashner et al.
US Patent 5,277,197 to Church
US Patent 4,571,682 to Silverman, et al.
US Patent 3,905,355 to Brudney
US Patent 3,641,993 to Gaarder
Such training has been effected through the use of body sensors that
provide signals corresponding to specific body motions and stances. Such
kinesthetic sensors can include goniometers, inclinometers, rotational
sensors,
force sensors, torsiometers, position sensors, bend sensors, tilt sensors,
stretch
sensors, pressure sensors, force sensors, velocity sensors, accelerometers,
and
neuromuscular electromyographic (EMG) pick-ups which identify the activation
of specific muscles or muscle groups. Any sensor which can provide an
indication as to the performance of a physical activity by the human body
arising
from the activation of specific muscles or muscle groups is relevant in this
field.
Hereafter such sensors are referred to as "body sensors
A particular challenge in this field is to train multiple, distinct, body
actions
to operate on a coordinated basis. An example of such a movement that could be
optimized through bio-kinesthetic feedback would be the case of a person
learning to swing a tennis racket. Such a person may have a pressure pad
placed
under one, forward, foot and an inclinometer strapped to their wrist. In order
to
train the individual to rotate their wrist at the same time that their weight
is shifted
onto the forward foot, trainees are coached to carry out the motion of
swinging a
tennis racket while the feedback originating from the two sensors is presented
to
them by a suitable display.
Various types of displays to provide feedback to trainees have been
proposed. One of the simplest is the creation of an image on a computer screen
that presents a bar which lengthens in accordance with the value of the output
2
CA 02531358 2005-12-21
originating from a body sensor. Such scalar values arise from detecting the
output
of a single sensor.
One particular prior art system described in part in US patent 6,413,190 to
Wood et al relies upon creating a video display which provides feedback to a
trainee, indicating the progress that is being made towards achieving
competence
in a specific physical activity. Feedback is provided through the positioning
of a
cursor on a video screen which serves as part of a video game. Integration of
this
type of display into the field of kinesthetic training provides motivation for
trainees to deliver appropriate signals from a body sensor and trains them to
carry
out an activity based on that sensor with the effort or timing necessary to
acquire
a functional skill.
This reference US patent 6,413,190 to Wood et al teaches use of a video
game such as "Pong"TM, which was one of the earliest video games created for
home computers. In this game a cursor in the form of a reflecting wall is
moved
by the game player along one edge of the screen to intersect the image of an
arriving ball. The presence of the reflecting wall or "paddle" causes the ball
to
rebound and eventually return for a second potential interception. According
to
this prior art patent, the output from a body sensor is used to control the
position
of the reflecting wall or paddle on the video screen in various ways. For
example,
body sensor output above a selected threshold releases the paddle to move
across
a border of the screen. In the absence of a signal, the paddle may remain
stationary or, in one version, move automatically to return to a "parked"
location.
Alternately, the paddle may move proportionally to the output from the body
sensor.
In this Wood reference a body sensor provides an output in the form of a
scalar value which, when such output surpasses a preset threshold, allows the
trainee to exercise control and participate in the videogame. Outputs below
this
threshold produce no response. Further, the use of multiple body sensors is
3
CA 02531358 2005-12-21
disclosed. This patent references using the output from at least two sensors
to
control cursor movement on the video screen. Such control can be in both X and
Y directions, encouraging the trainee to manipulate the cursor as part of a
video
game by actuating two body sensors. However, the outputs from such multiple X,
Y sensors have separate, independent effects on the display.
Two types of displays arising from the output from a single sensor can be
contemplated. The first is an "on-off' display wherein achievement of a
minimum level of signal, over a preset threshold, releases a paddle for motion
on
a video display. The second is a "proportional" display wherein the position
of a
paddle on a video display is proportional to the intensity of the signal being
generated by the body sensor.
The claims of this patent are directed towards combining signals obtained
from first and second muscle contractions by processing such signals on the
basis
of "Boolean anding". Thus the position of a gamepiece within a computer game
is controlled on the basis that the user must effect two muscle contractions
in
order to move the gamepiece on the display. Alternately, one sensor must
generate a "null" output for Boolean anding to permit gamepiece movement.
Nevertheless, two sensors are always being monitored and their combined
outputs
release the paddle for motion in an on-off manner.
However, this reference does not teach that the signal strengths from the
respective sensors are mixed. It therefore falls short of producing a
"composite"
output. Rather, Boolean anding only says that the outputs of two sensors have
to
be monitored in order to achieve action in the video display.
This patent teaches that output signals from body sensors can be adjusted
in magnitude so as to size them to fit a video display screen, a process that
may be
referred to as "calibration" or "normalization". Thus the system can establish
upper and lower range limits of signal that fit within the boundaries of the
video
4
CA 02531358 2005-12-21
display screen. The upper and lower limits of the signal from the sensor can
be
converted to represent a scale from 0 to 100%. The signal presented as the
person
performs a muscle contraction or movement can thereby be adjusted to fall
within
this range.
This patent further teaches that an output may be suppressed until a certain
threshold level, controlled by a therapist, has been achieved. Movement of the
game piece can be made non-responsive unless the person providing the body
signal exceeds a selected level of output. This can provide an incentive for
effort
as, for example, in the case where the threshold is set near the upper limit
of a
person's capacity.
A therapist can adjust the limits while the patient is engaged in a therapy
exercise, i.e. moving the goal posts during the game. Thus this reference
observes
that as the patient's movements are improved, the settings can be gradually
changed to require more effort by the patient in order to achieve a given
activation
of the cursor on the screen.
Use of a central dead-band has also been established. Thus crossing one of
two thresholds would be required to move a cursor right or left, with no
cursor motion
resulting from a body position within the dead band between the two
thresholds.
Different sensors can have different limits and different thresholds.
Cursor control can either be based on a position correspondence or a velocity
correspondence. Thus, in one embodiment, if the angle formed by elbow were
midway between the two extremes, then the game piece position would be midway
between the two extreme sides of the display. In another embodiment, the game
piece
position is set by putting the game piece in motion in a specific direction
corresponding to the current position of a body part.
A non-linear correspondence between sensor output and display is proposed
5
CA 02531358 2005-12-21
as a possibility in Wood. This permits a magnified or de-magnified response in
certain ranges. Thus, in some embodiments, cursor movement is not linearly
related
to body movement. For example, movement in the middle of the range may cause
very little corresponding cursor movement while movement towards one extreme
or
the other of the range may cause a greater amount of corresponding movement in
the
display. This feature addresses the form of display associated with a single
output. It
does not address the treatment of multiple outputs to produce a single
display.
A further type of pre-existing video display presents the signal values
originating from a single force sensor in an X - Y graphic format. The Y
direction represents the strength of the signal and the X direction represents
time,
providing a trace which proceeds across the screen. By scrolling the screen to
the
left, the leading end of the trace remains continuously present on the screen,
while
the recent history of the values being presented are shown by the trailing
balance
of the trace.
As a further feature of this specific pre-existing type of video display, the
difference between the output values of two force sensors has also been
displayed
as a single trace on such screen. Thus a composite signal has been created by
subtracting the value of the output from one force sensor from the value of
the
output from another force sensor. If the object were to train a subject to
produce
balanced forces on two force sensors, as with dual force pads placed beneath a
subject's respective feet, then this difference-based composite trace, when
this
objective is achieved, would present a steady, horizontal, line having a value
of
zero on the vertical scale.
United States Patent 6,032,530 to Hock is such a case where feedback
based upon a differential between signals is provided to an individual who is
trying to learn a physical activity. This document refers to separate sensors
being
combined by providing: "a sensor system for sensing rotational kinetic
activity
...(wherein)... The outputs of the sensors are connected differentially for
6
CA 02531358 2005-12-21
measuring rotation of the section about the axis." While this reference
suggests a
mixing of signals from two symmetrically placed sensors, this reference does
not
teach producing a composite output for purposes of a display that provides
feedback to a patient in the form of a single indication derived from
accumulating
values from multiple inputs arising from non-symmetrical muscle sources.
Hock also mentions the case where many transducers may be used:
"While the system works with a single transducer, there may be multiple
transducers, with processing circuitry that can integrate multiple inputs and
generate a composite result as a function of the multiple inputs."
However, in the context of the Hock disclosure the composite result he
refers to appears to be directed to the production of multiple, discreet and
musical
notes rather than a true composite signal wherein the feedback provided does
not
distinguish the signals originating from individual body sensors.
While these various systems represent instances of provision of displays
that can serve to provide feedback to a user, the full potential of the
composite
display concept has not been recognized or adequately exploited. What the
prior
art references fail to address is an enhanced procedure for effecting an
integration
of multiple outputs from the sensory-motor system to provide a single output
composite display arising from different, non-synunetrical muscle origins. A
need exists for a system based upon combining the outputs from two or more
such
sensors, particularly with the added feature of weighting values provided to
such
outputs, to produce a single performance-indicating output.
This invention addresses the object of providing a kinesthetic training
system that provides guidance and motivation for a trainee through more
advanced forms of composite displays.
7
CA 02531358 2005-12-21
This invention also addresses the object of providing a kinesthetic training
system which progressively exposes a subject to the outputs of two or more
sensors. This progressive training methodology goes beyond and is to be
distinguished from mere "Boolean anding" in that it provides subjects with a
more
complex display that serves as a strong incentive for improved physical
performance.
The invention in its general form will first be described, and then its
implementation in terms of specific embodiments will be detailed with
reference
to the drawings following hereafter. These embodiments are intended to
demonstrate the principle of the invention, and the manner of its
implementation.
The invention in its broadest and more specific forms will then be further
described, and defined, in each of the individual claims that conclude this
Specification.
SUMMARY OF THE INVENTION
The present invention is based on an apparatus for training a subject to
perform a specified physical activity or series of physical activities and a
process
of training said subject.
According to one aspect of the invention, the apparatus comprises two or
more body sensors for monitoring a subject's movements. These sensors may be
any type of body sensor that can record data that is indicative of a subject's
kinesthetic performance arising from different muscular origins and
particularly
from muscle sources that are not symmetrical to each other. This asymmetrical
feature is associated with the object of automatising a movement with several
degrees of freedom to provide a display that provides feedback to a patient in
the
form of a single indication derived from accumulating values from multiple
inputs
arising from non-symmetrical muscle sources.
8
CA 02531358 2005-12-21
The sensors need not be mounted on the user, they may be mounted on
equipment used by the user or can be in the form of devices that can sense the
user's movements from a distance. Each such sensor produces a sensor output
signal indicative of an aspect of the physical performance of the subject.
Also provided by the apparatus is a signal processing means which
receives sensor output signals from the two or more body sensors and provides
an
output which is used to drive the display. The output signals from individual
sensors may be "normalized" either by an operator providing input to the
signal
processing means, or by the signal processing means itself.
According to one variant of the invention this signal processing means
generates a single composite signal with a value which is an additive function
of
the individual sensor output signals originating from independent non-
symmetrical muscle sources. The composite signal is preferably formed by
adding values based on such individual sensor output signals to produce a
single
output. This is to be distinguished from simply taking a difference between
two
signals or adding signals from symmetrically placed sensors to cancel or
combine
common signal elements to produce the value for the composite signal. In this
sense, an 'additive" combination is established. Optionally, but preferably,
the
signal processing means assigns weighting factors in respect to each of the
individual sensor output signals before they are combined. After the composite
signal is formed, it may itself then be "normalized" by reducing its scale to
a
range wherein its maximum expected value corresponds to or near the upper
limit
of the available range for display.
The single composite signal is then provided to a display means, which
provides feedback to the user. This feedback is in the form of a display that
is
generated in response to the single composite signal but is derived from
multiple
sensor sources. The display itself for the single composite signal does not
contain
any apparent indication as to the relative contribution of the multiple sensor
9
CA 02531358 2005-12-21
signals contributing to the composition of the composite signal. An option
may,
however, be provided to give the trainee an indication of which particular
sensor
is not within the desired range. This may be a display which is visual, a
display
which is auditory or any form of display which will communicate to the
subject.
Visual displays may be in the form of a dial or linear gauge, and the display
may
be effected through a video terminal. The display may also be in the form of a
game piece which can be positioned on the display of a video terminal in
response
to the value of the composite signal.
In a further aspect, the invention may be effected by providing an
apparatus for training a subject to perform a specified physical activity, the
apparatus comprising:
a) two or more body sensors for monitoring a subject's movements, said sensors
each respectively producing sensor output signals indicative of the
kinesthetic
performance of the subject;
b) signal processing means connected to receive sensor output signals from the
body sensors,
said signal processing means comprising means to:
i. assign weighting factors in respect to each of said individual sensor
output signals to produce individual weighted sensor output
signals, and
ii. generate a single composite signal with a value as a function of the
individual weighted sensor output signals, and
CA 02531358 2005-12-21
c) display means connected to said signal processing means to receive said
composite signal to provide feedback to the subject in the form of a display
that is generated in response to the single composite signal,
whereby a subject may receive feedback as to their physical activity.
While it may appear, initially, that a trainee would be confused by
feedback in the form of a single composite signal that originates from
multiple
individual sensors, it has nevertheless been found that a subject can isolate
and
optimize individual actions that contribute to providing feedback in the form
of a
single composite display.
Optionally, an operator or therapist in attendance can adjust the manner in
which the combined output is produced from the input signals while the patient
is
actually carrying-out an exercise. This may appear equivalent to the
established
procedure of moving the "goalposts" during a game, but in accordance with the
invention this is effected in a sense different from the prior art. The
operator can
adjust the correspondence between the body sensors which detect the patient's
movements and the composite signal driving the display by changing the
weighting applied to specific individual sensor output signals that are being
used
to produce the composite output. The "goalpost", which is provided for the
single
composite signal may remain constant. It is the weighted values of the
individual
sensor signals that may be changed and then combined to form the new composite
signal. This can serve to force the patient to further define and control
certain
aspects of his physical performance in order to achieve a satisfactory output.
Optically, the standards for satisfactory output from the display, "the
goalposts",
appear to remain constant.
Thus, according to one aspect of the invention feedback is generated and
provided to the display system in the form of a single, composite output from
a
plurality of sensors in order to generate a display wherein the composite
output
11
CA 02531358 2005-12-21
includes elements arising from each of the plurality of sensors, some of which
may have optionally been modified by weighting factors.
According to another aspect of the invention, a subject pursuing an
activity with multiple body motions with the object of developing an improved
pattern of motion for one muscle set, including while limiting the activity of
another muscle set (avoiding excessive/improper/insufficient motion), is
exposed
to a training experience that is "progressive". According to this methodology,
exercise commences with a single sensor. And then, once training on the single
sensor has succeeded, exercise moves-on to training on a second sensor. The
first
sensor is still operative to suppress output if minimum thresholds or
satisfactory
values are not being achieved by the first sensor. The output from the first
sensor
is treated as providing an on-off signal. The output from the second sensor is
treated as providing a proportional output. Alternately, the display in this
scenario
may be driven by a composite signal.
Thus the display, according to this aspect of the invention, may either
simply be the output of the second sensor, in proportional mode, or a
composite
output derived from several sensors. The composite output could be based on
the
first and second sensors, or if further sensors are employed, from other
combinations of sensors.
In accordance with this aspect of the invention, a subject is exposed to the
following procedure:
a) providing a first body sensor for monitoring a subject's movements, said
first
body sensor producing first output signals in response to the performance of
the
subject;
12
CA 02531358 2005-12-21
b) providing signal processing means connected to receive the first output
signals
and generate a first display control signal with a value as a function of the
first
output signals;
c) establishing a range of acceptable values for the first output signals
produced
by the first body sensor;
d) providing display means connected to said signal processing means to
receive
the first display control signal to provide feedback to the subject in the
form of a
display that is generated in response to the first display control signal
until the
subject is able to generate first output signals that fall within the pre-
established
range of acceptable first output signal values; then
e) providing a second body sensor for monitoring the subject's movements, said
second sensor producing second output signals in response to the kinesthetic
performance of the subject;
f) delivering the second output signals to the signal processing means and
generating a second display control signal as a function of the second output
signals;
g) providing the second display control signal to the display means to provide
feedback to the subject in the form of a display that is proportional to and
generated in response to the second display control signal, but only while the
first
sensor display signal value falls inside the pre-established range of
acceptable
first output sensor signal values.
Alternately, a method it may be implemented as above but with the last
two sub-paragraphs above providing as follows:
13
CA 02531358 2005-12-21
f) delivering the first and second output signals to the signal processing
means
and generating a second display control signal as a composite function of the
first
and second output signals;
g) providing the second display control signal to the display means to provide
feedback to the subject in the form of a composite display that is generated
in
response to the second display control signal, but only while the first sensor
display signal value falls inside the pre-established range of acceptable
first
output sensor signal values.
This procedure can be carried out by providing an apparatus for training a
subject to perform a specified physical activity, the apparatus comprising:
a) a first body sensor for monitoring a subject's movements, said first sensor
producing first sensor signals in response to the kinesthetic performance of
the
subject;
b) a second body sensor for monitoring a subject's movements, said second
sensor producing second sensor signals indicative of the kinesthetic
performance
of the subject;
c) signal processing means connected to receive the first and second sensor
output signals from the body sensors, said signal processing means comprising
means to:
i) compare the first sensor signals to a range of pre-established acceptable
first sensor signal values, and
ii) generate a single display control signal with a value as a function of the
second sensor signals, and
14
CA 02531358 2005-12-21
d) display means connected to said signal processing means to receive the
single
display control signal to provide feedback to the subject in the form of a
display
that is generated in response to and is proportional to the single display
control
signal,
wherein the signal processing means operates to suppress the single display
control signal if the value of the first sensor signals falls outside the pre-
established range of acceptable first sensor signal values,
whereby the user may receive feedback in the form of a proportional display
arising from the single display control signal, but only when the first sensor
signal
values fall inside the preprogrammed range of acceptable on/off sensor signal
values.
Again, as indicated above with respect to the method, the single display
control signal may, itself, be a composite signal based on the outputs of two
or
more individual sensors. On this basis the last sub-paragraphs above may
instead
provide as follows:
c) signal processing means connected to receive the first and second sensor
output signals from the body sensors, said signal processing means comprising
means to:
i) compare the first sensor signals to a range of pre-established acceptable
first sensor signal values, and
ii) generate a single composite display control signal with a composite
value as a function of the first and second sensor signals, and
CA 02531358 2005-12-21
d) display means connected to said signal processing means to receive the
single
display control signal to provide feedback to the subject in the form of a
display
that is generated in response to the single composite display control signal,
wherein the signal processing means operates to suppress the single composite
display control signal if the value of the first sensor signals falls outside
the pre-
established range of acceptable first sensor signal values,
whereby the user may receive feedback in the form of a composite display
arising
from the single display control signal, but only when the first sensor signal
values
fall inside the preprogrammed range of acceptable on/off sensor signal values.
Further, the composite display control signal may be generated in response
to several sensors other than the outputs from the first and second sensors.
In all of the variants of the invention, the individual sensor output signals
may
optionally be normalized or adjusted through individual gain controls such
that
they each provide a"normalized" signal that falls satisfactorily within the
standard range for the signal processing unit. The signal processing means may
then assign additional weighting factors in respect to each of said normalized
individual sensor output signals to produce individual weighted sensor output
signals, and generate a single composite signal with a value as a function of
the
individual weighted sensor output signals. That single composite signal may
then, in turn, be normalized.
In generating the composite signal, the individual sensor output signals
may be combined by the signal processing unit in a variety of ways. For
example,
the individual sensor output signals may be given equal weight and their
values
simply added to produce the composite signal. Alternately, differing weighting
values may be applied to the respective individual sensor output signals in
order
to provide a greater or lesser emphasis on certain signals that correspond to
body
16
CA 02531358 2005-12-21
sensors that detect a portion of a movement that the user might be having
trouble
with. Such weighting values may be based upon the preferences of an operator
or may be based upon a predetermined weighting system that is preprogrammed
into the sensor signal processing means for a given activity. The processing
means, e.g. a programmed computer, may also be preprogrammed to change the
weighting values as the exercise activity progresses.
The weighting factors provided to the signal processing means may be
changed by an operator in order to shift the focus for the user on certain
physical
activities. An operator might also change the algorithm itself in order to
vary the
manner in which the composite signal is generated. The process of changing the
weighting factors or the algorithm may optionally be performed in real time
while
a user is performing a series of repetitions of an activity. This changing of
the
variables is especially desirable when a user has mastered one or more
components of an action but continues to struggle with the remaining
components.
For example, the user might initially be provided with two sensors whose
individual sensor output signals are given the same weighting factor and are
additively combined to provide the composite signal: Composite signal = la +
lb.
The user would repeatedly practice a motion while being given feedback as to
the
success of that motion. Eventually, the user might become very proficient in
one
of the motions (for example, the motion related to the "a" signal) but not the
other. In this case, the operator would decrease the weighting factor of the
"a"
signal with respect to the weighting factor of the "b" signal. The algorithm
for the
composite signal in this case might become: Composite signal = 0.75a + lb.
Preferably, the composite signal would then be re-normalized. This would then
require that, in order for the user to produce the same composite output
signal as
was produced with the initial weighting factors, the action related to the "b"
signal
would need to be performed with a greater degree of proficiency, e.g. effort.
This
is done through providing greater importance of the action related to the "b"
17
CA 02531358 2005-12-21
signal. Thus the focus, for the user, would become more oriented towards the
"b"
signal.
It may also be desirable to provide a blocking or suppression function that
is incorporated into the algorithm generating the comparison signal in order
to
suspend feedback when either individual sensor signals or the composite signal
fall within a respective range designated as a "dead zone". This would be
equivalent, for example, to being below a minimum threshold, for a specific
signal. One or more "dead zones" may be provided in respect to each of the
sensor signals. When one or more of the individual sensor signals and/or the
composite signals fall into a "dead zone", the feedback would be suppressed.
This would constitute a kind of "on-off' filtering.
An example would be suppressing the display originating from the
composite signal based on training to carry out a tennis stroke that relies on
signals from both shoulder and wrist body sensors. The display based upon the
composite signal would be suppressed if the wrist extension past 30 is not
complete.
A further example would be based on the user's movements producing a
display derived from multiple signals that each produce outputs that fall
within
the respective, acceptable, sensor value ranges. While this is occurring, that
is
when all desired signals are above their respective thresholds, feedback is
provided which is proportional to the composite output of the movement.
However if one component falls outside of the desired range, no display is
provided.
The feedback means may be in the form of a video game display. In the
example of the game "Pong", when the user generates acceptable values for the
desired signal(s), the paddle can begin to move slowly, moving towards one end
of the screen. User control can be limited to simply releasing the paddle for
18
CA 02531358 2005-12-21
movement i.e. on-off control. This is to be contrasted with the case where the
velocity of movement may be proportional to the composite signal. In the
threshold control mode, once the display driving signal surpasses the
threshold
established for the composite signal, the game piece moves at a predetermined
speed. The 'correctness' of the movement, in this case, affects the
persistence of
moving the game piece.
Alternately, by varying the quality demanded of the desired movement,
i.e. the effort or range of motion, the speed of the paddle may be changed.
This
would constitute a proportional display. In the proportional mode of control,
movement within the display based upon the driving signal is more responsive
to
the effort being made by the user.
The foregoing summarizes the principal features of the invention and
some of its optional aspects. The invention may be further understood by the
description of the preferred embodiments, in conjunction with the drawings,
which now follow.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure lA depicts a subject training for performing a desired movement in
the game of tennis whose shoulder sensor is coupled through a signal processor
to
a video display.
Figure 1B is a partial schematic of a portion of the signal processor of
Figure lA.
Figure 1C is a schematic depiction of the subject of Figure lA generating
a scalar display as to his performance effort arising from the shoulder
sensor.
19
CA 02531358 2005-12-21
Figure 2 is a graph showing the performance effort of the subject of Figure
1 C as a function of time.
Figure 3 is a schematic depiction similar to Figure 1 C but wherein the
subject is provided with two body sensors, one for the shoulder and one for
the
hip, providing two display signals.
Figure 4 is a graph showing the performance effort of the subject of Figure
3 as a function of time derived from the second sensor present on the hip of
subject of Figure 3.
Figure 5 is a graph of a type that may optionally be presented to a user
wherein the two sensor outputs of Figures 2 and 4 are overlaid on the same
graphic presentation.
Figure 6 illustrates an embodiment of the present invention wherein the
outputs of the sensors of Figures 2 and 4 are combined additively to produce a
composite output, displayed as a graphic curve which is a function of time,
for
presentation to the user according to the invention either directly or through
an
alternate display.
CA 02531358 2005-12-21
DESCRIPTION OF THE PREFERRED EMBODIMENT
Fig. 1A depicts a subject 1 training for performing a desired movement
related to, for example, the game of tennis. The subject 1 is provided with an
electromyographic (EMG) sensor 2, disposed on the subject's shoulder area 3.
This sensor 2 provides an electrical signal generated by the contraction of
muscles, an electromyographic signal arising in the shoulder area 3 of subject
1.
The EMG sensor 2 can be provided with a preamplifier (not shown) for pre-
amplifying the EMG sensor signal in order to provide an electrical signal
sufficiently strong to be fed to a signal processing means 5 via communication
line 6.
As shown in Fig. 1 A, the signal processing means 5 may include a
personal computer system (PC) 555 having a central processing unit (CPU) -805
coupled via bus 803 to memory 801, interface port 810, disk controller 805 and
input/output controller 806. The disk controller 805 is coupled to storage
device
804 and the interface port 810 is coupled to interface 802 via communication
line
8. The input/output controller 806 is coupled to video controller 807, which
is in
turn coupled to a display means such as video monitor 7, referred to as a
"display"
or "display means". The input/output controller 806 is also coupled to
keyboard
808 and pointing device 809.
As seen in Fig. 1B, interface 802 may include a series of sensor ports 820
connected to an analog-to-digital (A/D) converter 821 which is in turn
connected
to a microprocessor 822. The microprocessor 822 is connected to a port 823
which may be a USB port. Port 823 is connected to PC 555 through
communication line 8. EMG sensor 2 is connected to one of the sensor ports 820
through communication line 6. Interface 802 may also include an interface
memory 825 connected to microprocessor 822.
21
CA 02531358 2005-12-21
A first instruction set residing in storage device 804 and/or memory 801
allows PC 555 to communicate to interface 802 through interface port 810. The
first instruction set, together with a second instruction set residing in
interface
802, allows the signal processing means 5 to determine the EMG signal level
provided by EMG sensor 2. The instruction sets may further allow subject 1, or
a
trainer, to set specified ranges. The specified ranges are used by the signal
processing means 5 to determine to which of the specified ranges the EMG
signal
level corresponds. The first and/or second instruction set further allows for
the
provision of a comparison signal, the level of which will be dependent on the
specified range, to which the EMG signal level corresponds. This comparison
signal is provided to video monitor 7 through video controller 807 in order
for the
video monitor 7 to provide a visual feedback signal to subject 1. The visual
feedback signal may be in the form of a performance bar 9 as illustrated in
Fig.
1C.
This performance bar 9 relates to the passage of time in two aspects. In a
first aspect, the signal provided by a body sensor can vary during the
execution of
the procedure which the subject 1 is being trained to perform. Thus sensor 2
can
generate an output signal that has a value or waveform, which varies over
time.
This waveform is defined within the interval required for the complete
execution
of the desired movement. The performance bar 9 can represent the instantaneous
performance value of this output signal, the maximum level achieved or the
average over a selected interval, as well as other variations.
In a second aspect, the performance of the subject 1 can, and normally
will, change with repetitions of the action. In this case, the passage of time
is
divided into multiple intervals with each interval corresponding to the output
waveform arising from one repetition of the desired action.
Furthermore, over multiple intervals, the performance of subject 1 will
pass through stages corresponding to making progress towards optimum
22
CA 02531358 2005-12-21
performance. These stages can be associated with ranges of values for the
output
signal being provided by the body sensor. The performance bar 9 can represent
advancement in the level of performance over multiple repetitions, for
example,
by preserving a maximum value for a subject to exceed on a further attempt.
Whether the performance bar is presenting the instantaneous performance
value, the maximum level achieved, or the average over a selected interval,
including preserving a maximum value for a subject to exceed on a further
attempt, its value can be set by the output of the composite signal generated
by the
invention, as described further below.
Fig. 2 shows a graph of relative intensity of the EMG sensor signa121
with respect to a time scale that shows progressive stages of performance.
Initially, the shoulder muscle is excessively contracted above the minimum
desired threshold 22 to beyond a maximum desired level 23, and then the
shoulder
muscle maintains a relatively constant level of effort for the period
indicated.
Through the first and/or second instruction set, the signal processing means 5
may
be adjusted by a trainer, or by subject 1, in such a way that when EMG sensor
signal display 21 is within a first specified range comprised, for example,
between
levels 22 and 23, the corresponding output to the display signa121 on the
video
monitor 7 will be such that performance bar 9 of Fig. 1 C would indicate a
performance level 10 at the "excellent" level.
When subject 1 does not sufficiently contract the muscles in the shoulder
area 3, i.e. not within the satisfactory performance band 22, 23 as shown,
then
performance bar 9 would indicate a performance level 10 situated between
"poor"
and "excellent" (this is the exemplary scenario depicted in Fig. 1 C). The
performance bar 9 would indicate a similar decreased performance level 10 in
the
case where the subject 1 excessively contracts muscles in shoulder area 3 and
generate an EMG sensor signa121 comprised above leve123, and therefore
outside of the region of acceptance performance.
23
CA 02531358 2005-12-21
Thus, levels 22 and 23 represent thresholds bounding the "On" region
there between, and the regions between levels 24 and 22 and between 23 and 25
can represent "dead zones". A subject 1 or a trainer can specify the values
for
these levels and EMG sensor signal level ranges associated with each specified
range of performance level. In this manner the signal processing means 5 may
be
adjusted to provide a display signal to the video monitor 7 commensurate with
the
performance level.
As the subject 1 becomes better at performing a contraction of the muscles
of shoulder area 3, the width of the specified ranges may be narrowed, thereby
further training subject 1 to perform a more precise muscle contraction. The
width of the specified ranges could be modified manually via a programming
interface. Additionally, the software within the personal computer PC 555 can
be
programmed to have the width of the specified range vary automatically after
subject 1 performs a certain number of contraction of the muscles of shoulder
area
3 or achieves a preset level of performance.
Figure 3 depicts a scene where subject 1 is again training for performing a
desired movement related to the game of tennis. However, in addition to having
an EMG sensor 2 disposed on the shoulder area 3, subject 1 has a rotation
sensor
41 disposed or fitted to hip area 40. The rotation sensor 41 could be an
accelerometer such as Endevco's Mode17269 or Assemtech's ETS90SS. The
rotation sensor 41 detects a rotation of the hip area 40 with respect to the
base of
support of subject 1. The sensor 41 then provides an electrical signal
indicative of
the hip rotation to the signal processing means 5 via a communication line 42
connected, as communication line 6, to one of the sensor ports 820. Figure 4
provides a graphic display for this output, similar to Figure 2 wherein an
optimal
specified range for rotation sensor signal 51 has a lower limit depicted by
leve158
and an upper limit depicted by leve159.
24
CA 02531358 2005-12-21
In this case, where the subject 1 is equipped with an EMG sensor 2 and a
rotation sensor 41, the signal processing means 5 could process both signals
provided by the EMG sensor 2 and the rotation sensor 41 to provide at or for a
given moment in time two bar-graph displays, 7A, 7B showing respect of
performance levels 10A, IOB. However, the bar graphs do not show the
coordination of these two movements over time.
Allowing for signal output variation over time, Figure 5 shows a graph of
the relative intensities both of EMG sensor signal display 21 and rotation
sensor
signal display 51 with respect to a progressive time scale. The signal for
trace 51
in Figures 4 and 5 shows the value for the rate of angular rotation of the
hips
during the exercise. This rate builds up to a maximum during the swing and,
for a
short interval AA, the satisfactory higher rate of hip rotation of trace 51
(between
limits, 58 and 59) corresponds with the satisfactory level of effort being
made by
the shoulder muscles trace 21 (between limits 22 and 23).
The signal processing means 5 may be adjusted by a trainer, or by subject
1, through the first and/or second instructions sets mentioned above, in such
a
way as to provide to the video monitor 7 a composite comparison signal 60, cf
Figure 6, which is derived, for example, by adding both EMG sensor signal 21
and rotation sensor signal 51. This composite signa160 may then be normalized
for presentation.
Consequently, the curve for the trace 21 in Figure 5 represents the case
where subject 1, as in Figure 2, initially over-contracts and then optimally
contracts the muscles of shoulder area 3 (between specified levels 22 and 23).
At
the same time, the subject is slowly building-up rotation of the hip area 40
into a
satisfactory speed range (between specified levels 58 and 59). The values for
these two traces, 21, 51 can be additively combined to produce a composite
signal
trace 60, shown in Figure 6.
CA 02531358 2005-12-21
In the case where the EMG signa121 and the rotation signal 51 are both in
their respective optimal ranges - during interval AA -, the composite
comparison
signa160 shown in Figure 6, which, for the purposes of this demonstration is
taken to be a relative signal normalized to near its highest possible value,
will
have an intensity of nearly 100%. Optionally, a further performance bar
similar
to those in Figure 3 but showing composite performance could be presented to
the
subject. For the interval AA the performance level of the composite
performance
bar would indicate "excellent".
Such a combined performance bar as in Figure 3, which corresponds to the
composite signal 60, can provide feedback to subject 1 regarding the subject's
combined performance in the contraction of the muscles of shoulder area 3 and
in
achieving a rate of rotation of hip area 40.
Alternately, video monitor 7 can display the graph of Figure 6, presenting
the composite graphic to the subject 1 indicating the achievement of
successful
performance. In this display, the trace in the region AA can be distinguished
by
highlighting, or the balance of the trace can be suppressed on the basis that
either
trace 21 or trace 51 is in a "dead zone". Optionally, other traces 21, 51 can
be
provided with reduced highlighting.
Where a gaming display is used to motivate the user, the achievement of
successful performance in terms of acceptable values for the composite display
signal 60 can be used to advance the subject's participation in the game.
While
either of the traces 21 or 51 is in a dead zone, the game piece can be non-
responsive. Where a game acts in response to a scalar value, such value may be
derived from the intensity or level of the composite trace 60.
While Figure 6 shows the display of a composite signal 60, this display
could also be based on the proportional output 51 of rotational sensor 41.
This
display can be made conditional on the output 21 of EMG sensor 2 being in the
26
CA 02531358 2005-12-21
acceptable region 23, 24 as an "off-on" condition. This mode of display may be
adopted once the subject has mastered the motion associated with EMG sensor 2.
In the foregoing example depicted in Figure 6, the values for the outputs
of the two sensors 2, 40 were simply added to produce the values for the trace
60.
The generation of the single composite signal in accordance with the invention
may be based upon algorithms which are novel in the way that they mix inputs
from individual sensor output signals. The following is an example of how such
an algorithm can be applied to the invention.
A stroke in tennis (e.g., a forehand stroke) requires a weight shift forward.
This can be detected by a pressure sensor "a" in a footpad placed under the
participant's forward foot. It also requires a trunk rotation which may be
detected
by a horizontal rotation sensor "b" mounted on their belt. Additionally, it
requires
a wrist deviation "c" which may be detected by the EMG sensor attached to a
specified muscle associated with such wrist deviation. And finally, such
motion
may require a wrist extension of over 60 degrees, a motion which may be
detected
by a goniometer "d". In the latter case, the "zero-point" for the angular
orientation may be established as the orientation of the wrist determined at
the
stage of the full wrist extension where the action to be trained begins.
Assuming the most important element to be presented to this particular
individual for learning is the trunk rotation, then the relevant algorithm
could be:
Composite signal= 3/4a + 2b + c + d
This composite signal is then used to provide a display to the person being
trained. Receiving feedback from the composite signal, the participant
endeavors
to generate an output that corresponds with the optimum output as
predetermined
by the operator. The user can be motivated to go through the motion while
being
27
CA 02531358 2005-12-21
coached by the operator/technician who might be adding one element at a time
to
ease and graduate the learning process.
Each time the user goes through the motion, he endeavors to produce a
composite signal which more nearly achieves the ideal level which is
predetermined by the operator, by preset norms, or by the computer system.
Meanwhile, the operator, if present for assisting in this exercise, has a
visible display for each of the individual parameters that are contributing to
the
formation of the composite signal. The operator has control over the weighting
factors that are being applied to each of the individual sensor outputs.
Additionally, the operator can help teach a complex movement with
several degrees of freedom by establishing upper or lower thresholds,
essentially
defining "dead zones" for each contributing sensor. One or more dead zones
may be provided in respect of one or more of the independent sensor signals as
well as in respect to the composite sensor signal. If one or more independent
sensor signal(s) falls outside the range of acceptable values (which can be
above,
below or inside the indicated zone), then, according to one variant, no
composite
signal at all is provided to the display. This threshold/dead zone feature in
a
"veto" format serves as an indication to the player that he/she is deviating
substantially from the correct form of behavior with respect to that
particular
sensor associated with the dead zone limitation. The establishment of changes
to
such thresholds/dead zones can occur during training under the control of the
operator. Such changes can also be automatically implemented by the software
provided with the signal processing means. For example, the software can
effect
an 'automatic adjustment' for a particular sensor by changing the threshold or
dead
zone if the percentage of hits in, say, the first 10 attempts is within 20% of
the
ideal target value. In this manner, an automatic system can be established for
scaling the feedback system to suit the abilities of the patient or subject
being
trained.
28
CA 02531358 2005-12-21
The process of adding one element at a time to the display provided to the
trainee is an important contribution to the learning process. By training one
single
motion at first, (in the previous example the proper shifting of the weight to
the
forward foot detected by the pressure sensor "a", for example), the patient is
able
to determine the motion required to obtain a satisfactory score on the display
when only that motion is considered. In the current example, the algorithm
that
controls the composite signal would initially be:
Composite signal= a
The user would perform the required motion repeatedly, while obtaining
feedback on the success of that single motion. Once the patient has mastered
this
movement, the technician might decide to add the required trunk rotation
movement to the system. By placing the horizontal rotation sensor "b" on the
patient's belt, the trainee is then able to master this secondary motion while
still
being required to perform the first. In this case, to add increased importance
to
the trunk rotation (since the patient has already become partially skilled at
shifting
his weight to his forward foot), the algorithm might be something like:
Composite signal= 0.5a + 2b
The signals may be added (and weighted if so desired) until all necessary
motions are integrated. The final algorithm might be something like
Composite signal= 0.75a + 2b + c + 0.5d
where c is a wrist deviation which may be detected by the EMG sensor attached
to a specified muscle associated with such wrist deviation and d is a
goniometer
indicating the degree of wrist extension.
29
CA 02531358 2005-12-21
The operator can establish thresholds or introduce a dead zone in respect
of one or more of the individual sensors or even in respect to the composite
signal. If a sensor signal value with "veto" power falls beneath a selected
lower
threshold, above a selected upper threshold, or in its dead zone, then
according to
one variant of the invention, no composite signal at all is provided to the
display.
By continuing with the aforementioned tennis scenario, a wrist extension
of over 60 degrees may be necessary in order to perform the desired stroke. In
this case, a motion, which may be detected by a goniometer "d", and a
threshold
may be provided such that in order for the display to be activated, the value
for
the goniometer "d" must be above 60 degrees. If a moving paddle within a video
game display provides the feedback, then the value for "d" would need to
exceed
its assigned threshold in order to move the paddle in the game. In this case,
the
algorithm for the composite signal might include the combined limitations:
Ifd<60,
Composite signal = 0
Ifd>60,
Composite signal= 0.5a + 2b + c + d
The operator may even provide multiple thresholds for the "dead zones"
of a single sensor signal, or one or more thresholds for different signals. It
is
useful to add a "dead zone" to the range of motion in which a user might
damage
himself. If the previous example is used, it might be said that a wrist
extension
measured by the goniometer "d" above 90degrees is dangerous. In this case, the
algorithm for the composite signal might be:
If d < 60 or d > 90,
Composite signal = 0
CA 02531358 2005-12-21
If d > 60 and d < 90,
Composite signal= 0.5a + 2b + c + d
The case might arise where the algorithm could require that the composite
signal must fall below or above a given threshold. In this case, the algorithm
might be:
If Composite signal < threshold,
composite signal output = 0
If Composite signal > threshold
Composite signal output = 0.5a + 2b + c + d
The composite signal may be used by the signal processing means and
feedback means to provide feedback to the user in many ways. For example, a
display means whereby "off-on" feedback is provided to the user when the
user's
movements produce a composite signal that falls within the acceptable range.
In
this case the feedback is only an indication of whether or not all of the
desired
sensor signal values fall within the established acceptable sensor value
ranges.
Where the feedback means is a video game this procedure may serve to
provide "off-on" control as follows. In the example of the game "Pong", when
the user's efforts produce signals which fall within the acceptable value
range for
the desired signal(s), the paddle will move with a constant speed towards one
end
of the screen. When the user's effort does not fall within that range - the
paddle
may move automatically with constant speed back towards the opposite end.
Alternately, the strength of the composite signal may control the location
of the paddle on the video display. A level of strength over a certain value
could
move the paddle in one direction, and a level of strength below that value
could
move the paddle in the other direction. Or the value of the composite signal
may
31
CA 02531358 2005-12-21
produce a proportional display wherein the location of the paddle is
determined
proportionally to the strength of the composite signal between zero and
maximum
composite signal.
On this basis, it will be seen that a new and useful means can be
established for providing feedback to users in the rehabilitation and physical
training fields so that they may integrate or tie together and automate
various
aspects of a movement.
CONCLUSION
The foregoing has constituted a description of specific embodiments
showing how the invention may be applied and put into use. These embodiments
are only exemplary. The invention in its broadest, and more specific aspects,
is
further described and defined in the claims which now follow.
These claims, and the language used therein, are to be understood in terms
of the variants of the invention which have been described. They are not to be
restricted to such variants, but are to be read as covering the full scope of
the
invention as is implicit within the invention and the disclosure that has been
provided herein.
32
