Note: Descriptions are shown in the official language in which they were submitted.
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
HEART RATE VARIABILITY AND DROWSINESS DETECTION
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to United States Provisional
Application No.
62/319,854, filed on April 8,2016.
TECHNICAL FIELD
[0002] Example embodiments relate to a seat assembly in a vehicle as well
as
systems and methods for detecting drowsiness of a driver and for alerting a
driver.
BACKGROUND
[0003] Numerous automotive accidents are caused by drowsy, sleeping or
hypo-
vigilant drivers. Others have attempted to address this issue and prevent
accidents by
creating technology to detect a lane departure of a vehicle, such as U.S.
patent No.
8,354,932 to Schmitz. Other systems rely on the use of an electrooculogram
(EOG) and
image capture devices to monitor and interpret blinking patterns of the driver
in order to
detect drowsiness. See, for example, U.S. patent No. 8,306,271 to Yoda et al.
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Advantages of the present invention will be readily appreciated as
the same
becomes better understood by reference to the following detailed description
when
considered in connection with the accompanying drawings wherein:
[0005] Figure 1 is a block diagram illustrating a system in accordance
with one
embodiment of the present disclosure;
[0006] Figure 2 is a schematic perspective view of an automotive vehicle
interior;
[0007] Figure 3 is a perspective view of a dashboard display of the
automotive
vehicle;
[0008] Figure 4 is a close up view of a seat assembly for an automotive
vehicle;
1
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
[0009] Figure 5 is a side view of the automotive vehicle interior and a
driver;
[0010] Figure 6 is a flow chart of a method of detecting a low alertness
level of a
driver;
[0011] Figures 7, 8 and 9 are graphs of received sensor signals and R-R
intervals;
[0012] Figure 10 is a graph of R-R interval values over three time
intervals;
[0013] Figure 11 is a graph with a histogram representation of some of
normalized
data from Figure 10; and
[0014] Figures 12, 13 and 14 are graphs of power spectral analysis
determined for
received sensor signals.
[0015] Like reference numerals are used throughout the Figures to denote
similar
elements and features.
DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS
[0016] Many prior systems for detecting drowsiness of a driver are
primarily
reactive in that they detect or respond to the driver being drowsy, or to the
operation of the
vehicle by a sleeping or drowsy driver. The present application describes
methods, devices
and systems for detecting drowsiness or the alertness level of a driver of
vehicle, preferably
before the driver's state affects the operation of the vehicle. Embodiments
described herein
measure the driver's heart rate variability (HRV) through biometric sensors in
the vehicle
and a drowsy state is predicted by correlating HRV values to activation levels
of the driver's
central nervous system. Although illustrated in the present application with
the example of
an automotive vehicle, the described embodiments may apply to other vehicles
operated by
a seated driver.
[0017] Through the central nervous system, the brain regulates two motor
systems,
the voluntary motor system which provides for muscular control of the limbs,
body and
head; and the involuntary motor system, also known as the autonomic nervous
system
2
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
(ANS), which regulates internal organs like the heart, digestive system,
lungs, bladder and
blood vessels. The ANS is divided into 2 opposing sections: the
parasympathetic nervous
system which is responsible for "rest and digest" functions; and the
sympathetic nervous
system which is responsible for "fight or flight" functions. The interaction
between the
parasympathetic and sympathetic nervous systems is known as sympathovagal
balance. The
sympathovagal balance leads to variations in cardiac output, heart rate, blood
flow, pupil
dilation and digestive system.
[0018] Human heart rates are often calculated by measuring the number of
contractions or beats per minute (bpm). A typical, healthy human is likely to
have a heart
rate (HR) of about 50 to 90 bpm, Heart rate variability (HRV) refers to the
phenomenon of
the variation in the time intervals between heartbeats. HRV rises as the
dominance of the
sympathetic system gains relevance over parasympathetic system, such as when a
person is
more alert and active. HRV falls when the parasympathetic system gains
relevance, such as
when a person is getting drowsy. Thus, the variation of HRV over time can
provide a
physiological indication of a person transitioning from an alert to a drowsy
state.
[0019] A block diagram of a system 100 for detecting drowsiness of a
driver of a
vehicle is illustrated in Figure 1. The system 100 includes one or more
sensors 110, a
computing device 120, and one or more output devices 130. As described further
below, the
computing device 120 is configured to detect drowsiness or low alertness of
the driver based
on analysis of the signals received from the one or more sensors 110 and a
determination of
the HRV of the driver. If a drowsy state is predicted, the computing device
120 sends signals
to activate one or more output devices 130 in order to notify the driver. In
some
embodiments, the system 100 includes a seat assembly of the vehicle, a user
interface of the
vehicle such as a dashboard display, or both the seat assembly and the user
interface.
[0020] The computing device 120 has a processor and a memory which is
configured to store and execute instructions for methods for detecting
drowsiness as
described herein. The computing device 120 may be a separate module or
component, such
as a programmable chip or application specific integrated circuit, or it may
be part of the
3
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
another computing device present in the vehicle. In some embodiments, the
computing
device 120 may be configured to support wired and/or wireless communications
with other
systems of the vehicle and with the sensor 110 and output device 130. The
computing
device 120 may include a user interface and the computing device 120 may be
configured to
support a user programming or configuring the computing device 120 and/or a
user
obtaining data or reports of information stored by the computing device 120
through the user
interface. The computing device 120 may include additional signal processing
circuitry or
functionality in order to filter signals received from the sensor 110.
[0021] Figure 2 illustrates an example view of an automotive vehicle
interior and the
environment for the possible placement of sensors 110 and output devices 130.
The vehicle
includes a seat assembly 200 which includes a generally horizontal seat
cushion 212 and a
generally upright seat back 214 for supporting a seat occupant within the
vehicle. The seat
back 214 is typically operatively coupled to the seat cushion 212 by a
recliner assembly 216
for providing pivotal movement between an upright seating position and a
plurality of
reclined seating positions. The seat occupant is referred to herein as the
driver.
[0022] Each of the seat cushion 212 and seat back 214 commonly includes a
molded
resilient cellular foam pad (not shown) encased in a trim cover assembly 218,
which may be
of cloth, vinyl, or leather. The one or more sensors 110 may be placed in the
seat assembly
200 behind the trim cover assembly 218, either behind/below or in front/on top
of the foam
pad. In some embodiments, the one or more sensors 110 may be integrated into a
layer of
the trim cover assembly 218. In some embodiments, the one or more sensors 110
may be
provided as part of a heating and cooling system for the seat assembly 200. In
some
embodiments, the one or more sensors output devices 130 may be provided as
part of the
heating and cooling system for the seat assembly 200.
[0023] Each sensor 110 comprises a biometric sensor which is mounted in
the
vehicle in order to gather data about the driver. Multiple sensors 110 of the
same type may
be used, or the sensors 110 may comprise multiple sensors of different types.
Example types
of sensors include a capacitive sensor, a radio frequency (RF) sensor, an
impedance sensor,
4
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
a permittivity sensor, or an ultrasensitive pressure transducer. Each type
sensor is described
below.
[0024]
Capacitive sensors monitor a voltage change over time caused by the
polarization and depolarization of the heart muscle when beating. Electrodes
on the sensor
interact with the body of the driver to determine those variations. In one
embodiment,
capacitive sensors are located in or on the seat back 214.
[0025]
Radio Frequency (RF) sensors are composed of a transceiver and an antenna.
The transceiver modulates and issues a radio frequency signal through an
antenna to the
driver's body. The mechanical and electromagnetic changes in the body caused
by the
driver's heartbeat and respiration modulate the signals that bounce back to
the antenna. This
modified signal is received or captured by the RF sensor. In one embodiment,
RF sensors
are located in or on the seat back 214, and/or in or on the seat cushion 212.
[0026]
Impedance sensors monitor the bio-impedance signal from the driver's body
obtained due to blood volume changes and blood resistivity changes between
heart beats. In
one embodiment, impedance sensors are located in or on the seat back 214.
[0027]
Permittivity sensors create an electromagnetic field above a surface of the
sensor.
During polarization and depolarization of the driver's heart muscle, the
electromagnetic field is affected and the disturbances in the field are
recorded by the
permittivity sensor. In one embodiment, permittivity sensors are located in or
on the seat
back 214.
[0028]
Ultrasensitive pressure transducers operate on the principle that every time
the heart beats, a mechanical pulse is generated throughout the driver's body.
This
mechanic pulse creates a signal also known as a ballistocardiogram.
Ultrasensitive pressure
transducers are made of piezo-resistive and piezo-electric materials that
sense the fluctuation
of these ballistic forces over time. Ultrasensitive pressure transducers can
be located in or
on the seat back 214, and/or in or on the seat cushion 212.
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
[0029] In some embodiments, the seat assembly 200 includes an
electrostatic
discharge (ESD) mat 220. The ESD mat 220 is an anti-static device that helps
eliminate
static electricity by having a controlled low resistance. The mat is grounded
to the vehicle in
order to discharge the static electricity at a slow rate. In one embodiment,
the ESD mat 220
is located within the seat cushion 212 as shown in Figure 2.
[0030] Figures 3, 4 and 5 illustrate additional example views of the
automotive
vehicle interior and the environment for the possible placement of sensors 110
and output
devices 130. An output device 130 is a device capable of generating output
signals, such as
alarms or notifications, to alert a driver to a potential state of drowsiness
or to an actual state
of the driver being drowsy or asleep. In some embodiments, the one or more
output devices
130 may be integrated into a layer of the trim cover assembly 218. In some
embodiments,
the one or more output devices 130 may be provided as part of a heating and
cooling system
for the seat assembly 200.
[0031] As shown in Figure 3, an output device 130 may include a dashboard
indicator or infotainment system generating an audio, visual, or audio/visual
notification
such as an alarm and or message 310. Figure 4 illustrates a close-up view of
the seat
assembly 200. An output device 130 may provide a haptic alert such as a
vibration which
may be felt by the driver. The haptic alert may be provided via a vibration or
other sensory
disturbance of the seat cushion 212, the seat back 214, the restraint system
410, and/or the
steering wheel 510 as illustrated in Figure 5, or any combination thereof.
[0032] Multiple output devices 130 may be activated by the computing
device 120 at
or around the same time in order to generate multiple audio, visual,
audio/visual, and/or
haptic signals to alert the driver to a state in which the driver may not have
a sufficient level
of alertness to safely operate the vehicle. The output devices 130 may be
activated if the
driver's measured HRV predicts a low alertness level. In one embodiment, the
output
devices 130 are deactivated once the driver's measured HRV increases and the
level of
predicted alertness is sufficient for operation of the vehicle. In some
embodiments, the
output devices 130 may be deactivated based on other feedback signals, such as
the receipt
6
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
of an input through a user interface of the system 100 or through an input of
the
infotainment system, to acknowledge the alarm or warning signal. Other
feedback may
include the slowing or stopping of the vehicle. These other feedbacks or
inputs may be used
alone or in combination with the measured HRV of the driver to deactivate the
output device
130.
[0033] Methods of determining a low alertness level or potential drowsy
state of a
driver of a vehicle are described in further detail below with reference to
Figures 6 to 14.
Specifically, referring to Figure 6, a method (600) of determining an
alertness level of the
driver and activating an output device is shown.
[0034] The method includes receiving signals (610) from the one or more
sensors
110 which are indicative of or provide information relating to the driver's
heartbeat. Each
received signal is processed to determine intervals (620) of the driver's
heartbeat.
Depending on the type of sensor 110 and signal received, a variation in an
electric,
magnetic, or mechanical property of the sensor signal over time provides
information from
which the intervals of the driver's heartbeat can be determined. In some
embodiments, this
determination includes a preliminary action to filter the received signals in
order to remove
noise and improve or clean up the waveform for further analysis.
[0035] Although the measured property and amplitude of each heartbeat
signal may
vary from sensor to sensor, each signal may be analyzed to detect peaks in the
waveforms.
The three central deflections of a heartbeat waveform which are easiest to see
and detect are
referred to as the QRS complex. The R wave component of the QRS complex has
the largest
positive amplitude and HRV may be determined based on measured intervals in
the peak of
the R wave. The space between peaks may be referred to as the "R-R" interval
and an R-R
interval may be determined for each received signal and/or for the group of
signals from
multiple sensors. Figures 7, 8 and 9 illustrate sample waveforms and R-R
intervals for
signals received from a variety of sensor types. The present application
describes the
determination of HRV based on R-R intervals but other peaks or points in the
heartbeat
waveform may be used to determine the interval between heartbeats.
7
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
[0036] Once R-R intervals are determined for each of the signals from one
or more
sensors 110, the HRV is determined (630). The action of determining the HRV
may include
determining R-R interval outliers and omitting the outliers from the sampled
data. In one
embodiment, only R-R intervals inside the range 0.26 seconds < R-R < 1.2
seconds are
counted. This range of R-R intervals is associated with a heart rate between
50 to 230 bpm.
In one embodiment, normalized data points will be assumed as the average of
the previous
data points.
[0037] The HRV may be determined in two ways. According to the first
method, R-
R intervals are plotted and HRV is determined (630) based on how scattered the
data points
are from an average within specific clusters of time. The data points will be
clustered within
specific time periods and compared with the previous time period to determine
the degree of
variability from time period "n" to time period "n-1", "n-2", etc. as
illustrated in Figure 10.
The data also may be normalized and analyzed based on overlapping histograms
for
different time periods, identified as time periods A and B in Figure 11. The
variance and the
mean change may be used to compare different time periods and determine
changes in HRV.
Since the data is collected and processed in real time, any shift within a sub-
group of a given
sample size may be monitored to anticipate a trend in the data points. In
other words, a
smaller time period inside the broader time period, such as a group of data
points within
time period n, also may be analyzed to predict trends in HRV and a
corresponding alertness
level of the driver.
[0038] A second method to determine HRV (630) is to process the received
data
through power spectral analysis. Any sinusoidal or wave form signal with an
amplitude that
varies over time has a corresponding frequency spectrum. In one embodiment,
finite time
periods can be used to sample the data received from a sensor 110, such as
over 3, 5, 10 or
15 minute periods. From the raw data obtained by the sensor 110, a polynomial
curve fitting
is done to determine a base function over time. A Fourier Series is used to
describe the
function over time, according to the standard expression shown in equation (1)
with
coefficients ao, aõ and bn as shown in equations (2), (3) and (4).
8
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
(1) f(X) = -2ao + Enc i(an cos(nx)) E: i(bnsin(nx))
(2) ao = -12 f +77 f (x) dx
(3) an = -1 f +77 f (x) cos nx
(4) bn = -1 f 4-1T f (x) sin nx dx
n
[0039] An approximation of the Fast Fourier Transform (FFT) may be
achieved by
computing the Discrete Fourier Transform (DFT) and converting the function
from a time to
a frequency domain as shown in equations (5) and (6).
(5) F (k) = f 4-c f (x)21Tikx
-0,
(6) DFT = X, = (xj e-2Triis/N)
i=o
[0040] The function f(x) is obtained by data point interpolation using
the Fourier
series to describe its sinusoidal behavior. The function f(x) describes the
variation in the
signal from the sensor 110 in regards to its independent variable. In the
above equations, the
independent variable x is time, and f(x) is the signal variation in milliVolts
(mV) over time.
[0041] F(k) is the Fourier Transform for the infinite number of data
points the sensor
110 can collect over time. Since the time period evaluation will be finite, to
simplify the
computational analysis, the DFT may be used for a given time period. In that
case, N is the
number of outputs (R-R intervals for example) and s is the continuous variable
x (time)
which may be replaced by a discrete variable s (an integer of "x" -- which is
to be confirmed
by the operational data).
9
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
[0042] Figure 12 represents a typical power spectra determined for a
received signal
from a sensor 110 according to equation (7). The power spectrum is achieved by
squaring
the modulus of F(k).
+.
(7) I F(k)I2 = Ico f(x) e-27rikx A 12
[0043] As seen in Figure 12, data is divided into four frequency bands,
ultra-low
frequency (ULF), very low frequency (VLF), low frequency (LF) and high
frequency (HF).
Figures 13 and 14 illustrate a comparison between two different discrete time
periods to
show the fluctuations in frequency bands. The fluctuations are associated with
transitional
stages from alertness (Figure 13) to drowsiness (Figure 14), especially in the
high frequency
and low frequency bands.
[0044] As HRV levels decrease, a state of drowsiness or low alertness
level is
predicted (640). A state of drowsiness or low alertness may be predicted, for
example, in
response to the HRV level falling below a first predetermined threshold and/or
remaining
below the first predetermined threshold for a first period of time. In
response to this
prediction, an output device 130 is activated (650) in order to send a
notification to the
driver of the vehicle. Audio, visual, audio/visual or sensory notifications
may be generated
by one or more output devices 130 as described above. If a state of drowsiness
or low
alertness is not predicted, the method (600) continues to receive signals from
the sensors
110, determine heartbeat intervals (620), and determine and monitor HRV levels
(630).
[0045] If a termination condition (660)is met, the one or more output
devices 130
may be deactivated (670) in order to stop generating warnings or alarms. The
termination
condition (660) may include the HRV level increasing above a second
predetermined
threshold and/or remaining above the second predetermined threshold for a
second period of
time. The first and second predetermined thresholds may or may not be the
same. The first
and second periods of time may or may not be the same. As described above, a
termination
condition (660) may include the receipt of an input through a user interface
of the system
100 or through an input of the infotainment system, to acknowledge the alarm
or warning
CA 03020345 2018-10-05
WO 2017/177221 PCT/US2017/026787
signal, or the slowing or stopping of the vehicle, or a combination of these
inputs and
conditions along with the measured HRV level.
[0046] Although the exemplary embodiments described herein employ device
memory, other types of computer readable media which can store data that are
accessible by
a computer, such as magnetic cassettes, flash memory cards, digital versatile
disks,
cartridges, random access memories (RAMs), read only memory (ROM), USB or
memory
sticks, a cable or wireless signal containing a bit stream and the like, also
may be used in the
exemplary operating environment. Non-transitory computer-readable storage
media
expressly exclude media such as energy, carrier signals, electromagnetic waves
and signals
per se.
[0047] The invention has been described in an illustrative manner, and it
is to be
understood that the terminology, which has been used, is intended to be in the
nature of
words of description rather than of limitation. Many modifications and
variations of the
present invention are possible in light of the above teachings. It is,
therefore, to be
understood that within the scope of the appended claims, the invention may be
practiced
other than as specifically described.
11
