Note: Descriptions are shown in the official language in which they were submitted.
SELF-IDENTIFYING ONE-WAY AUTHENTICATION
METHOD USING OPTICAL SIGNALS
[0001]
[0002]
[0003]
[0004]
[0005]
[0006]
[0007]
[0008]
[0009]
[0010]
[0011]
[0012]
[0013]
Field of the Disclosure
[0014] This disclosure relates generally to a system and method for
transmitting self-identifying
modulated light signals.
Background
[0015] Indoor positioning services refers to methods where networks of devices
and algorithms
are used to locate mobile devices within buildings. Indoor positioning is
regarded as a key
component of location-aware mobile computing and is an important element in
provided
augmented reality (AR) services. Location aware computing refers to
applications that utilize a
user's location to provide content relevant to location. Additionally, AR is a
technology that
1
CA 2892923 2019-02-20
overlays a virtual space onto a real (physical) space. To successfully enable
AR and location
aware computing, accurate indoor positioning is an important requirement.
[0016] Global Positioning Systems (GPS) loses significant power when passing
through
construction materials, and suffers from multi-path propagation effects that
make it unsuitable
for indoor environments. Techniques based on received signal strength
indication (RSSI) from
WiFi and Bluetooth wireless access points have also been explored. However,
complex indoor
environments cause radio waves propagate in dynamic and unpredictable ways,
limiting the
accuracy of positioning systems based on RSSI. Ultrasonic techniques (US),
which transmit
acoustic waves to microphones, are another method which can be used to
approximate indoor
position. They operate at lower frequencies than systems based on WiFi and
attenuate
significantly when passing through walls. This potentially makes US techniques
more accurate
than WiFi or Bluetooth techniques.
[0017] Optical indoor positioning techniques use optical signals, either
visible or infrared, and
can be used to accurately locate mobile devices indoors. These are more
accurate than the
approaches mentioned previously, since optical signals are highly directional
and cannot
penetrate solid objects. However this directionality limits the potential
reliability of optical
signals, since difficulty in aligning the receiver and transmitter can occur.
[0018] A variety of systems transmit a one-way authentication signal from an
infrastructure
endpoint to a mobile device. Applications that make use of one-way
authentication include
mobile loyalty solutions, ticketing, secure access control, payments, media
sharing, and social
media. A common technology used for one-way authentication is Quick Response
Codes (''QR
codes"). QR codes are two-dimensional barcodes. Mobile devices that include
cameras can take
a picture of the QR code and then decode the information associated with the
QR code.
However, QR codes have the disadvantage of being easily copied. Anyone having
a device with
a camera can take a picture of a QR code, print it out, and replicate it. For
example, when QR
codes are used in mobile loyalty solutions, it is possible for users to cheat
the system by taking
pictures of the QR codes and submitting fake scans. An alternative emerging
technology is Near-
Field-Communication (NFC), which uses radio frequency transmission to transmit
an
2
CA 2892923 2019-02-20
authentication message over a short distance. However, NFC is still in the
nascent stages of
adoption. Many mobile devices do not have NFC capabilities, and NFC terminals
have not yet
been widely deployed.
Summary
[0018a] In an aspect, there is provided a system, comprising: a self-
identifying optical
transmitter, the transmitter comprising: a non-volatile memory for storing an
identifier of the
transmitter as a one-way proximity authentication code; a processor in
communication with the
memory for generating a data signal comprising the identifier of the
transmitter as the one-way
proximity authentication code; a modulator for receiving the data signal and
generating an
electrical signal, wherein the modulator generates the electrical signal by
modulating the data
signal; a light source for receiving the electrical signal, converting the
electrical signal into an
optical signal, and broadcasting the optical signal as an optical data
transmission stream that
verifies that a user holding a mobile device came into proximity of the
transmitter; an optical
surface for dispersing the optical data transmission stream as the optical
data transmission stream
is emitted from the transmitter; and wherein the mobile device includes: an
inductive or
capacitive circuit configured to: respond to contact with the mobile device
from the user holding
the mobile device, and in response to the contact with the mobile device cause
the transmitter to
broadcast the optical signal.
10018b] The present disclosure also discloses a mobile device, comprising: an
image sensor; a
wireless interface configured to communicate through a network over a wireless
medium; a
processor coupled to the image sensor and the wireless interface; a memory;
and software in the
memory to be run by the processor, wherein running of the software by the
processor configures
the mobile device to implement functions, including functions to: operate the
image sensor to
capture one or more images including a modulated visible light signal
transmitted from an
optical transmitter located within a space, wherein the visible light signal
includes a one-way
proximity authentication code; demodulate the modulated visible light signal
from the captured
one or more images to obtain a pattern; determine, based at least in part on
the obtained pattern,
information corresponding to a geographical location of the optical
transmitter; store the
3
CA 2892923 2019-09-16
determined information; and process the determined information to verify that
a user of the
mobile device came into proximity of the optical transmitter.
[0018c1 The present disclosure further discloses a method, comprising steps
of: capturing, via an
image sensor of a mobile device, one or more images including a modulated
visible light signal
transmitted from an optical transmitter located within a space, wherein the
visible light signal
includes a one-way proximity authentication code; demodulating the modulated
visible light
signal from the captured one or more images to obtain a pattern; determining,
based at least in
part on the obtained pattern, information corresponding to a geographical
location of the optical
transmitter; storing the determined information; and processing the determined
information to
verify that a user of the mobile device came into proximity of the optical
transmitter.
[0018d] In another aspect, there is provided a system, comprising: a self-
identifying optical
transmitter, the transmitter comprising: a non-volatile memory for storing an
identifier of the
transmitter as a one-way proximity authentication code; a processor in
communication with the
memory for generating a data signal comprising the identifier of the
transmitter as the one-way
proximity authentication code; a modulator for receiving the data signal and
generating an
electrical signal, wherein the modulator generates the electrical signal by
modulating the data
signal; a light source for receiving the electrical signal, converting the
electrical signal into an
optical signal to verify that a mobile device is in proximity to the
transmitter as verification that a
user has visited a particular location, and broadcasting the optical signal as
an optical data
transmission stream including the identifier of the transmitter as the one-way
proximity
authentication code; wherein the mobile device comprises: an image sensor; a
wireless interface
configured to communicate through a network over a wireless medium; a
processor coupled to
the image sensor and the wireless interface; a memory; and software in the
memory to be run by
the processor, wherein running of the software by the processor configures the
mobile device to
implement functions, including functions to: operate the image sensor to
capture one or more
images including the modulated visible light signal with the one-way proximity
authentication
code, transmitted from the optical transmitter; demodulate the modulated
visible light signal
from the captured one or more images to obtain the one-way proximity
authentication code; and
process the one-way proximity authentication code to verify that a user of the
mobile device
4
CA 2892923 2019-09-16
came into proximity of the optical transmitter; and an inductive or capacitive
circuit configured
to: respond to contact with a mobile device, and in response to the contact
with the mobile device
cause the optical transmitter to broadcast the optical signal.
[0019] Given the security issues with QR codes, and the lack of NFC-enabled
devices, demand
exists for a secure, one-way authentication method that works on mobile
devices available today.
This disclosure describes a device that uses an optical signal for
transmitting a secure, one-way
authentication code. The signal is received using an optical sensor present on
many mobile
devices. This optical sensor can be an image sensor that is part of a mobile
device's camera, a
photodiode, a reverse-biased LED, or any other optical receiver that would be
readily understood
by a person of ordinary skill in the art. The optical signal may be
transmitted via visible light, as
most camera-enabled mobile devices are capable of receiving information
transmitted via a
visible light signal. In some embodiments, the signal may be transmitted in
the infrared
spectrum, using an IR LED or other similar emitter. For mobile device cameras
that include
infrared filters that may prevent such a signal from being received, a visible
light signal is
preferred.
Brief Description of the Drawings
[0020] FIG. 1 is a representation of a mobile device receiving light from a
LED light source,
according to some embodiments of the present disclosure.
CA 2892923 2019-09-16
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
100211 FIG. 2 is a representation of a mobile device receiving multiple
sources of light
simultaneously from multiple LED light sources, according to some embodiments
of the
present disclosure.
100221 FIG. 3 is a representation of the internal components commonly found
in a LED
light source that is capable of being modulated to send digital data.
[0023] FIG. 4 illustrates information which can be optically transmitted
from an LED
light source.
[0024] FIG. 5 is a representation of the components which are commonly
found in mobile
devices which enable them to receive optical signals from. LED sources.
[0025] FIG. 6 is a representation of multiple LED light sources sending
unique
information to multiple mobile devices.
100261 FIG. 7 illustrates the process of a mobile device sending
identification information
and receiving location information via a network to a server.
100271 FIG. 8 illustrates the high level contents of the server which
includes databases
and web services for individual areas enabled with light positioning systems.
100281 FIG. 9 illustrates the components inside the databases.
100291 FIG. 10 illustrates the information contained in the Light IDs
database.
100301 FIG. 11 illustrates the information contained in the Maps database.
100311 FIG. 12 illustrates the information contained in the Content
database.
100321 FIG. 13 illustrates the information contained in the An.alytics
database.
[0033] FIG. 14 illustrates the process of a mobile device receiving
location and content
information via a light-based positioning system.
100341 FIG. 15 is a process illustrating the background services and how
they activate
various sensors contained inside the mobile device.
[0035] FIG. 16 illustrates the process of combining multiple information
sources with a
light-based positioning service.
1.00361 FIG. 17 illustrates how a client accesses multiple light
positioning enabled
locations with multiple mobile devices.
100371 FIGS. 18A-C are representations of a light source undergoing pulse-
width-
modulation at varying duty cycles, according to some embodiments of the
present disclosure.
100381 FIGS. 19A-C are representations of a light source undergoing pulse-
width-
modulation at varying duty cycles with a DC offset, according to some
embodiments of the
present disclosure.
- 6 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
100391 FIG. 20 is a block diagram of a DPR modulator with a dimming control
system
for a light source, according to some embodiments of the present disclosure.
100401 FIG. 21 is a representation of a block diagram of a DPR modulator,
according to
some embodiments of the present disclosure.
100411 FIG. 22 is a block diagram of an encoder for DPR modulation,
according to some
embodiments of the present disclosure.
100421 FIG. 23 is a block diagram for a waveform generator for DPR
modulation,
according to some embodiments of the present disclosure.
100431 FIG. 24 is a block diagram of a symbol selector system module, which
is used to
select an appropriate symbol for use in DPR modulation, according to some
embodiments of
the present disclosure.
100441 FIG. 25 is a plot of a camera sampling function, according to some
embodiments
of the present disclosure.
100451 FIG. 26 is a plot of a modulated illumination function undergoing
DPR
modulation at a frequency of 3001-1z, according to some embodiments of the
present
disclosure.
100461 FIG. 27 is a plot of a convolution of a camera sampling function and
a DPR
modulated light signal, according to some embodiments of the present
disclosure.
100471 FIG. 28 is a model of the CMOS sampling function for a rolling
shutter, according
to some embodiments of the present disclosure.
100481 FIG. 29 is a plot of a sampling function for a CMOS rolling shutter
over multiple
frames, according to some embodiments of the present disclosure.
100491 FIG. 30 is a high level flow chart of an algorithm for configuring a
mobile device
to receive DPR modulated signals, according to some embodiments of the present
disclosure.
100501 FIG. 31 is a high. level flow chart of an algorithm, for minimizing
and locking
camera settings using existing mobile device application programming
interfaces (APIs),
according to some embodiments of the present disclosure.
100511 FIG. 32 is a high level flow chart of an algorithm for receiving DPR
signals on an
image sensor, according to some embodiments of the present disclosure.
100521 FIG. 33 is a high level flow chart of an algorithm fur determining
tones embedded
within a DPR illuminated area, according to some embodiments of the present
disclosure.
100531 FIG. 34 is a high level flow chart of an algorithm for performing
background
subtraction on images gathered from a DPR illuminated scene, according to some
embodiments of the present disclosure.
- 7 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
[0054] FIG. 35 is a high level flow chart of an algorithm for performing
motion
compensation on video frames when performing DPR demodulation, according to
some
embodiments of the present disclosure.
100551 FIG. 36 is a photograph of a surface under illumination from DPR
modulated
signals, according to some embodiments of the present disclosure.
[0056] FIG. 37 is a post-processed image of a DPR modulated scene after
performing
background subtraction, according to some embodiments of the present
disclosure.
100571 FIG. 38 is a post-processed image of a DPR modulated scene after row
averaging,
according to some embodiments of the present disclosure.
[0058] FIG. 39 is a plot of the 1-D spectral content of a DPR modulated
surface,
according to some embodiments of the present disclosure.
100591 FIG. 40 is a plot of the 1-D spectral content of a DPR modulated
surface after
removing DC bias, according to some embodiments of the present disclosure.
100601 FIG. 41 is a 2-D FFT of a DPR modulated surface, according to some
embodiments of the present disclosure.
100611 FIG. 42 is a 2-D ITT of a DPR modulated surface after applying a low
pass filter,
according to some embodiments of the present disclosure.
[0062] FIG. 43 is a 2-D FFT of a DPR modulated surface after applying a
high pass filter,
according to some embodiments of the present disclosure.
100631 FIG. 44 is a representation of mobile devices receiving multiple
sources of light,
according to some embodiments of the present disclosure.
100641 FIGS. 45 and 46 are block diagrams of light sources, according to
some
embodiments of the present disclosure.
100651 FIG. 47 is a representation of mobile devices receiving multiple
sources of light,
according to some embodiments of the present disclosure.
100661 FIG. 48 illustrates a user interface, according to some embodiments
of the present
disclosure.
[0067] FIG. 49 illustrates retrieval of a device profile, according to some
embodiments of
the present disclosure.
[0068] FIG. 50 shows a plot of variance in parameterized phase versus
possible
modulation frequencies, according to some embodiments of the present
disclosure.
100691 FIGS. 51 - 53 illustrate filtering of frequency components,
according to some
embodiments of the present disclosure.
- 8 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
100701 FIG. 54 illustrates a method for intelligently choosing the
appropriate
demodulation, according to some embodiments of the present disclosure.
100711 FIG. 55 illustrates a camera switching algorithm, according to some
embodiments
of the present disclosure.
100721 FIG. 56 illustrates a method for resolving between multiple light
identifiers,
according to some embodiments of the present disclosure.
[0073] FIGS. 57 - 59 illustrates a mobile device receiving signals from
multiple lights
simultaneously, according to some embodiments of the present disclosure.
100741 FIG. 60 is a block diagram of a DPR enclosure module, according to
some
embodiments of the present disclosure.
[0075] FIG. 61 is a representation of a self-identifying one-way optical
transmitter, in
accordance with some embodiments.
100761 FIG. 62 is a representation of a mobile device receiving a self-
identifying one-way
optical transmission, in accordance with some embodiments.
100771 FIGs. 63-66 are representations of self-identifying optical
transmitters, in
accordance with some embodiments.
100781 FIG. 67 is a representation of a waveform for using pulse width
modulation to
transmit a signal, in accordance with some embodiments.
100791 FIG. 68 is a representation of a high-brightness light source
broadcasting a
modulated signal overhead, in accordance with some embodiments.
100801 FIG. 69 is a representation of a one-way authentication system being
used in a
checkout aisle, in accordance with some embodiments.
Description of Example Embodiments
[0081] The present disclosure relates to a self-identifying one-way optical
transmitter that
broadcasts an optical signal via modulation of a light source. The transmitter
is not
necessarily designed for general-purpose illumination. Instead, the
transmitter is designed to
broadcast a signal that can be successfiffly received if a mobile device user
puts their mobile
device in close proximity with the transmitter. The user experience for the
receiver may be
one in which the user "taps" their device onto the transmitter to receive the
ID code. Users
tapping their devices in this way provides an advantage in terms of security,
because users
com.e physically close to the transmitter in order to receive the signal. A
user's device
coming in close proximity to the transmitter also provides an improved user
experience,
because such an action provides tangible feedback. In some embodiments, the
transmitter
- 9 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
can act as a completely stand-alone device. The transmitter may be battery
powered and
preloaded with a specific optical signal. In other embodiments, the
transmitter may be
connected to a power and data network if battery replacement is not viable for
a particular
application or the use of multiple signals is desired.
[0082]On the mobile device side, a camera-equipped mobile device may utilize
its camera to
receive the transmitted signal. An application on the mobile device may be
invoked to
initialize the camera and, once invoked, start to acquire images. These images
can be
captured in succession, in the case of recording video, or can be captured
individually. The
self-identifying one-way optical transmitter may be constantly broadcasting
the signal.
[0083] FIG. 1 represents a mobile device 103 receiving light 102 from a LED
light source
101. The LED light source 101 can be any lighting source used for general
purpose, spot
illumination, or backlighting. The LED light source can come in several form
factors but is
not limited to: Edison. screw in, tube style, large and small object
backlighting, or accent
lighting spots and strips. For the purposes of this disclosure, we consider
any form of LED
light as a potential source capable of transmitting information.
100841 Light 102 is a modulated LED light source 101, and is part of the
visible
electromagnetic wireless spectrum. LEDs are considered digital devices which
can be rapidly
switched on and off, to send signals above the rate which the human eye can
see. This allows
than to be exploited to send digital data through the visible light itself. By
modulating the
LEDs, turning them on and off rapidly, one can send digital information that
is unperceivable
to the human eye, but is perceivable by applicable sensors, including but not
limited to image
sensors and other types of ph.otosensors.
100851 There are many modulation techniques used to send information
through light
102. One technique, "On Off Keying" (00K), is a scheme to transmit digital
data by rapidly
switching a signal source on and off. 00K is the simplest form of amplitude-
shift keying
(ASK) which is a modulation technique that represents digital data through
either the
presence or absence of a carrier wave. When communicating with visible light,
the canier
wave takes the form of the transmitted light signal. Therefore at a
rudimentary level, when
the light signal is turned "on" a digital "one" is perceived, and when the
light signal is turned
"off" a "zero" is perceived. Furthermore the rate at which the light signal is
turned on and off
represents the modulation frequency. Note that regardless of changing the
modulation
frequency, the "carrier wave" remains unchanged as this is an inherent
property of the light
itself. For example the can-ier wave corresponding to a blue light signal is
uniquely different
than the carrier wave corresponding to a red light signal. While these two
signals differ only
-10-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
in the wavelength specific to their perceived color, they can be perceived as
two discrete
signals.
100861 In addition to 00K, another possible technique is defined as
"Digital Pulse
Recognition" (DPR). This modulation technique exploits the rolling shutter
mechanism of a
complementary metal-oxide-semiconductor (CMOS) image sensor. Due to their
superior
energy efficiency, CMOS sensors are preferred to charged-coupled device (CCD)
sensors on
mobile devices. When a CMOS image sensor with a rolling shutter takes an
image, it does
not expose the entire image simultaneously. Instead, the rolling shutter
partially exposes
different portions of the frame at different points in time. Typically, this
causes various
unwanted effects: skew, wobble, and partial exposure. In the presence of an
LED light
driven by a pulse width modulated signal, images received from a CMOS sensor
exhibit
"residual banding" in the form of visible distortions. The image appears to
have alternating
dark/white stripes. The stripes are a direct result of the rolling shutter
mechanism, and their
width is proportional to thc frequency of the pulse width modulated (PWM)
signal. Higher
frequencies correspond to narrower stripes, and lower frequencies result in
wider stripes.
Practical frequency ranges for use with this technique are between 6011z and
500011z. This
technique allows one to exploit the rolling shutter mechanism to recover
digital data from an
optically encoded signal.
100871 DPR has the potential for much higher data rates than both 00K and
frequency
shift keying (FSK). In. FSK. and 00K, the camera's frame rate limits the data
rate. The
highest possible data rate is half of the frame rate, since each symbol spans
over two frames.
In DPR. modulation, a single frame is sufficient for capturing the transmitted
symbol.
Furthermore, symbols are not "binary" - there are can be as many as 30
different possibilities
for a symbol.
[0088] in the DPR modulation scheme, image processing is used to measure
the stripe
width of the recorded image. By successively changing the LED driver frequency
for each
frame, information is essentially transmitted through recognition of the band
widths. In the
current design, 10 separate frequencies are used. For a 30 frames per second
(FPS) camera,
this corresponded to an effective data transfer rate of-100 bits per second
(bps).
[0089] Both of these techniques are interesting because they can allow the
transmission
of information through single color light sources, instead of having to create
lighting sources
which contain multiple color lights. In the world of LED lighting products,
white light is
majorly achieved by layering a phosphorous coating on top of blue LEDs. The
coating
creates the visible perception of "white" light, instead of blue. The
alternative to this can be
-11-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
achieved through combining red, green, and blue LED lights; however this
approach is
expensive and power inefficient as the lumens per watt properties differ
between different
colored LEDs. Blue LEDs are generally more energy efficient than their red and
green
counterparts, which is why they are used in most commercial LED lighting
products. It is
because of this reason that it makes the most sense to use a data modulation
technique that
uses a single wavelength of light, rather than multiple, because this complies
with LED
lighting products.
[0090] In addition to LED light sources, other types of light sources are
also capable of
transmitting information through modulation. Alternative incandescent and
fluorescent
technologies can also be exploited to achieve data transmission, however the
circuitry is more
complex because th.e turn on and turn off times of incandescent and
fluorescent lights are
subject to additional factors.
100911 The modulation frequency of the light source is highly dependent on
the receiving
circuitry. While incandescent and fluorescent technologies generally do not
"flicker" on and
off during the course of normal operation, LED lighting sources are sometimes
designed to
flicker above the rate which the eye can see in order to increase their
longevity, and consume
less power. Most humans cannot see flicker above 60Hz, but in rare instances
can perceive
flicker at 100Hz to 110Hz. To combat this, lighting manufacturers design
flicker above
200Hz into their lighting products.
100921 Mobile device 103 can be a smart mobile device and is most commonly
found in
the form of mobile phones, tablets, and portable laptop computers. In order
for a mobile
device 103 to receive information 102 from the LED light source 101 it has an
embedded or
attached sensor which is used to receive the incoming light 102 signals. One
such sensor is a
camera, which has a typical frame refresh rate between fifteen and sixty
frames per second
(fps). The fps is directly related to the speed at which optical signals can
be transmitted and
received by the camera. The sensor can capture a number of successive image
frames that
can later be analyzed to determine if a light source is providing information
through light.
[0093] Mobile device 103 can include a processor, module, memory, and
sensor in order
to capture and analyze light received from light sources. The mobile device
can analyze the
successive image frames captured by the sensor by using the module. The module
can be
logic implemented in any combination of hardware and software. The logic can
be stored in
memory and run by processor to modify the successive images and analyze the
successive
images to determine information encoded in the light of one or more light
sources. The
module can be built in to the mobile device to provide the capabilities or it
can be
- 12 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
downloaded and installed. The module can be an application that runs on the
mobile device
when selected by a user. The module can also be used to receive content and
other
information related to the position of the mobile device and to provide this
content to other
modules or to the mobile device.
100941 The reception of optically transmitted information is particularly
interesting when
used as an indoor positioning system.. In a light-based positioning system,
the physical
locations of light sources can be used to approximate the relative position of
a mobile device
103 within line of sight. On the mobile side, in addition to a receiving
module, the mobile
device 103 can use information to determine position of the mobile device. The
mobile
device can access a data source containing information about where the lights
are physically
located to determine position. This data source can be stored locally, or in
the case where the
mobile device 103 has a network connection, the data source could be stored on
an external
server 703.
100951 For scenarios where a network connection is not available, before
entering an
indoor space the mobile device 103 could optionally download a "map pack"
containing the
information used to locate itself indoors, instead of relying on an external
server 703. In order
to automate this process, the mobile device 103 would first use an alternative
existing
technique for resolving its position and would use the gained location
information to
download the appropriate map pack. The techniques for receiving geo-location
information
include, for example, GPS, GSM, WiFi, user input, accelerometer, gyroscope,
digital
compass, barometer, Bluetooth, and cellular tower identification information.
These
techniques can also be used to fill gaps between when a position of the mobile
device is
determined using the light-based technique. For example, a mobile device can
be placed at
times so its camera does not capture light sources. Between these times these
alternative
existing techniques can be used for filling in position and location
information that can be
helpful to the user. The map pack would contain a map 902 of the indoor space
the user is
entering, locations of the lights from some sort of existing or third-party
lighting plan 1103,
and any location-dependent content 903 for the mobile device 103 to consume.
Any requests
for location information would simply access data stored locally on the mobile
device 103,
and would not need to access a remote server via a network 601.
100961 In terms of the experience when using a light-based positioning
system, the indoor
location reception and calculation can happen with little to no user input.
The process
operates as a background service, and reads from the receiving module without
actually
writing them to the display screen of the mobile device. This is analogous to
the way WiFi
- 13 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
positioning operates, signals are read in a background service without
requiring user
interaction. The results of the received information can be displayed in a
number of ways,
depending on the desired application. In the case of an indoor navigation
application, the user
would see an identifying marker overlaid on a map of the indoor space they are
moving
around in. In the case of content delivery, the user might see a mobile media,
images, text,
videos, or recorded audio, about the objects they are standing in front of.
100971 In scenarios where the mobile device 103 is in view of several light
sources, it can
receive multiple signals at once. FIG. 2 is a representation of a mobile
device 103 receiving
identification information 102a-102c from multiple LED light sources 101a-
101c. Each light
source is transmitting its own unique piece of information. In order to
identify its position or
receive location-based content, the mobile device 103 can then use the
received information
to access a database 802 containing information about the relative positions
of the LED light
sources 1.01a-101c and any additional content 903. When three or more sources
of light are
in view, relative indoor position can be determined in three dimensions. The
position
accuracy decreases with less than three sources of light, yet remains constant
with three or
more sources. With the relative positions of lights 101a-101c known, the
mobile device 103
can use photogrammetry to calculate its position, relative to the light
sources.
100981 Pbotogrammetry is a technique used to determine the geometric
properties of
objects found in photographic images. In the context of locating mobile
devices using light
sources, photogrammetry refers to utilizing the corresponding positions of LED
light sources,
and their positions in 3-D space, to determine the relative position of a
camera equipped
mobile device. When three unique sources of light are seen by the camera on a
mobile device,
three unique coordinates can be created from the various unique combinations
of 101a-101c
and their relative positions in space can be determined.
100991 For a mobile device 103 equipped with an image sensor we can
consider the
following scenario. When multiple LED light sources appear in the image
sensors field of
view, the sources appear brighter relative to the other pixels on the image.
Thresholds can
then be applied to the image to isolate the light sources. For example, pixel
regions above the
threshold are set to the highest possible pixel value, and the pixel regions
below the threshold
are set to the minimum possible pixel value. This allows for additional image
processing to
be performed on the isolated light sources. The end result is a binary image
containing white
continuous "blobs" where LED light sources are detected, and dark elsewhere
where the
sources are not detected.
- 14 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
101001 A blob detection algorithm can then be used to find separate LED
light sources. A
minimum of three separate LED blobs are used to resolve the 3-D position of a
mobile device
103. Each LED blob represents a "region of interest" for the information
reception, and is
simultaneously transmitting a unique piece of information via the modulated
visible signal
from the light source. For the purposes of reception, each region of interest
is processed
independently of other regions of interest arid is considered to be uniquely
identifiable. A
center of mass calculation for each region can be performed to determine the
pixel
coordinates of the center of each LED light source. This center of mass
calculation is
performed for each frame to track the regions of interest as they move around
the image.
101011 Once the regions of interest are established, a detection algorithm
captures
multiple image frames for each region of interest in order to receive the
visible light signal
contained in each blob. For each frame in a detected region of interest, a
threshold algorithm
determines whether the frame contains a "1" (in. the case of an aggregate
pixel value above
the threshold), or a "0" (in the case of an aggregate pixel value lower than
the threshold).
The threshold algorithm is used since the communication. is asynchronous, so
the camera
receiver period may overlap between the transmission of a "1" and a "0" from
the LED light
source.
101021 The result of converting successive image frames in a region of
interest to binary
values is in essence a down-sampled digital version of the signal received
from the LED light
source. Next demodulation of the down-sampled digital signal is used to
recover the
transmitted bits. This down sampling is used due to the fact that the signal
modulation
frequency should be above the rate at which the human eye can see, and the
image sensor
frame rate is typically limited to 15-30 fps.
101031 At a lower level, the mobile device 103 processes data on a frame-by-
frame basis.
Each frame is split into separate regions of interest, based on th.e detection
of light sources.
For each region of interest, a thresholding algorithm is used to determine
whether a given
region is "on" or "off'. This is done by taking the average pixel value for
the region and
comparing it to the threshold value. If the region is "on", the demodulator
assumes the light
source has just transmitted a "1". If the region is "off', the demodulator
assumes the light
source has sent a "0". The result of this is the equivalent of a 1-bit analog-
to-digital
conversion (ADC), at a sampling rate which is equal to the frame rate of the
camera.
101041 After a frame is processed, the results of the ADC conversation are
stored in a
circular buffer. A sliding correlator is applied to the buffer to look for the
presence of start
bits 402. If start bits 402 are found, the demodulation algorithm assumes it
is reading a valid
- 15 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
packet of information 401 and proceeds to capture the rest of the
transmission. Two samples
are used for each bit, so the algorithm creates a linear buffer that is twice
the size of the
remaining packet. Each subsequent ADC is written sequentially to the linear
buffer. When
the linear buffer is filled, the demodulation algorithm performs a Fast
Fourier Transform
(FFT) on the buffer to recover the transmitted signal.
101051 FIG. 3 describes internal components commonly found in LED light
source 101
with the addition components to allow for the transmission of optical signals.
The LED light
source 101 contains an alternating current (AC) electrical connection 301
where it connects
to an external power source, an alternating current to direct current (AC/DC)
converter 302
which converts the AC signal from the power source into an appropriate DC
signal, a
modulator 304 which interrupts power to the LEDs in order to turn them. on and
off, a
microcontroller 305 which controls the rate at which the LEDs are modulated,
and a LED
driver circuit 303 which provides the appropriate amount of voltage and
current to the LEDs.
[01061 Electrical connection 301 is an electrical source that is used to
supply power to the
LED light source 101. This most commonly comes in the form of a 120 Volt 60 Hz
signal in
the United States, and 230 Volt 50 Hz in Europe. While depicted in FIG. 3 as a
three
pronged outlet, it can also take the form of a two terminal Edison socket
which the bulb is
screwed into, or a bundle of wires containing a live, neutral, and/or ground.
When
considering other forms of lighting such as backlighting and accent lighting,
the electrical
connection can also come in the form of a DC source instead of an. AC source.
101071 Most LED light sources contain an AC/DC converter 302 which converts
the
alternating current from. the power source 301 to a direct current source used
internally by the
components found inside the bulb or light source. The converter takes the
alternating current
source commonly found in existing lighting wiring and converts it to a direct
current source.
LED light sources generally use direct current, therefore an AC/DC converter
is found in
most lighting products regardless of form factor.
101081 LED driver 303 provides the correct amount of current and voltage to
the LEDs
contained inside the lighting source. This component is commonly available and
can have
either a constant current or constant voltage output. The LEDs found inside
most lighting
sources are current controlled devices, which require a specific amount of
current in order to
operate as designed. This is important for commercial lighting products
because LEDs
change color and luminosity in regards to different currents. In order to
compensate for this,
the LED driver circuitry is designed to emit a constant amount of current
while varying the
voltage to appropriately compensate for the voltage drops across each LED.
Alternatively,
-16-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
there are some high voltage LEDs which require a constant voltage to maintain
their color
and luminosity. For these cases the LED driver circuitry provides a constant
voltage while
varying the current.
101091 Modulator 304 serves the function of modulating the LED light source
101 on and
off to optically send light 102 signals. The circuits comprising the modulator
can simply
consist of solid state transistors controlled by a digital input. In essence
the modulator 304
turns the LEDs on and off by allowing or preventing current flow. When current
flows
through the modulator with the switches closed the LEDs turn on, and when the
switches are
open in the modulator no current can flow and the LEDs turn off. When the
modulator is
controlled by an additional logic component, it has the ability to send
repeating patterns of
on/off signals in order to transmit digital data through the visible light
102. The modulator
interfaces directly in between the AC/DC converter 302 and the LED driver 303,
and is
controlled by a microcontroller 305.
101101 The microcontroller 305 provides the digital input signal to the
modulator unit
304. This function can. also be achieved using a field programmable gate array
(FPGA), but
typically consumes more power with added complexity. The microcontroller's 305
task is to
send a pre-determined sequence of signals to the modulator 304 which then
interfaces with
the LED driver 303 to modulate the outgoing visible light from the LED source
101. The
microcontroller contains a nonvolatile memory storage area, which stores the
identification
code of the light signal. Examples of possible nonvolatile memory sources
include
programmable read only memory (PROM), electrically erasable programmable read
only
memory (EEPR.OM), or Flash.
101111 In regards to the microcontroller pins, the microcontroller 305
contains a digital
output pin, which is used to modulate the light output. To generate the output
signal
waveforms, timer modules within the microcontroller 305 are used. Typical
logic levels for
the digital output are 3.3V and 5V. This digital output feeds into the
modulator 304 which
interrupts the driver circuit 303 for the LED light source 101. Alternatively,
if the LED light
source requires lower power, such as backlighting or individual LED diodes,
the output of the
microcontroller 305 could also be used to drive the light sources directly.
10112] The sequence of signals sent from the microcontroller 305 determines
the
information which is transmitted from the LED light source 101. FIG. 4
describes the
information 401 format of the optically transmitted information from the light
102. At the
highest level, each packet of information contains some sort of starting bit
sequence, which
indicates the beginning of a packet, followed by data 403, and some sort of
error detection
- 17 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
identifier. The size and position of each portion of information is dependent
on the
application and is also constrained by requirements of the receiving device.
101131 Each packet of information 401 transmitted from the LED light source
101
contains a sequence of starting bits 402, followed by data 403, and then
terminated with an
error detection code 404. Since the LED light sources 101 are continually
broadcasting
information 401, erroneous packets are simply discarded while the receiver
listens for the
starting bits 402, indicating the beginning of the next packet. In cases where
multiple sources
of light are observed by a mobile device 103, multiple pieces of information
401 are received
simultaneously.
101141 Information 401 describes the encoded information that is
transmitted by the LED
light source 101. The information 401 is contained in a packet structure with
multiple bits
which correspond to numeric integer values. The data 403 portion of the
information packet
can include unique ID codes 701. Currently the data 403 size is set to 10
bits, but can be of
varying length. Each bit represents a binary "1" or "0", with 10 bits of data
103
corresponding to 1024 possible values. This corresponds to 1024 unique
possibilities of ID
codes 701 before there is a duplicate. The 1D code can include location
information in the ID
code that provides a general indication of geographical location of the light.
This
geographical location information can be used to more quickly locate light
source
infomiation that is used in determining indoor positioning on the mobile
device. For
example, the geographical information can point to a database to begin,
searching to find
relevant information for positioning. The geographical information can include
existing
location identifiers such as area code, zip code, census tract, or any other
customized
information.
101151 The ID code 701 is static and is assigned during the calibration
phase of the LED
light source 101 during the manufacturing process. One method to assign the ID
code 701 is
to place instructions to generate a random code in the nonvolatile memory.
Once the LED
light source 101 is powered on the microcontroller reads the ID code 701 from.
the
nonvolatile memory storage area, and then uses this code for broadcasting each
and every
time it is subsequently powered on. Since the ID code 701 is static, once it
is assigned it will
be forever associated locally to the specific LED light source 101 which
contains the
microcontrol ler 305.
101161 FIG. 5 describes the components found in mobile devices 103 that are
capable of
receiving optical information. At the highest level the mobile device contains
an image
sensor 501 to capture optically transmitted information, a central processing
unit 502 to
-18-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
decipher and manage received information, and a network adapter 503 to send
and receive
information.
101171 Photosensors are devices which receive incoming electromagnetic
signals, such as
light 102, and convert them to electrical signals. In a similar fashion.,
image sensors are
arrays of photosensors which convert optical images into electronic signals.
The ability to
receive signals from multiple sources is an important benefit when using image
sensors for
receiving multiple optical signals.
101181 Image sensor 501 is a typical sensor which is found in most smart
devices. The
image sensor converts the incoming optical signal into an electronic signal.
Many devices
contain complementary metal-oxide-semiconductor (CMOS) image sensors, however
some
still use charge-coupled devices (CCD). CMOS image sensors are the more
popular choice
for mobile devices due to lower manufacturing costs and lower power
consumption. There
are several tradeoffs to consider when choosing an image sensor to perform
photogrammetry
on multiple LED light sources 101. One tradeoff is between the camera
resolution and the
accuracy of the photogrammetric process when triangulating between multiple
light sources ¨
increasing the number of pixels will increase the accuracy. There is also
another tradeoff
between the data rate of the transmission and the sampling rate (in frames per
second) of the
camera. The data rate (in bits/second) is half the frame rate of the camera
(e.g., a 30 fps
camera will receive 15 bps). And finally when determining the length of the
information 401
packet, the larger the size the longer the reception period, as more bits
generally requires
longer sampling periods to capture the full message.
101191 CPU 502 is a generic CPU block found in most smart devices. The CPU
502 is in
charge of processing received information and sending relevant information to
the network
adapter 503. Additionally the CPU has the ability to read and write
information to embedded
storage 504 within the mobile device 103. The CPU 502 can use any standard
computer
architecture. Common architectures for microcontroller devices include ARM and
x86.
101201 The network adapter 503 is the networking interface that allows the
mobile device
103 to connect to cellular and WiFi networks. The network connection is used
in order for
the mobile device 103 to access a data source containin.g light ID codes 701
with their
corresponding location data 702. This can be accomplished without a data
connection by
storing location data 702 locally to the mobile device's 103 internal storage
504, but the
presence of a network adapter 503 allows for greater flexibility and decreases
the resources
needed. Furthermore, the network adapter 503 is also used to deliver location
dependent
content to the mobile device when it is connected to a larger network 601.
-19-
CA 02892923 2015-05-28
WO 2014/063150 PCT/ U S2013/065923
[01211 FIG. 6 is a representation of multiple LED sources sending light
102a-d
containing identification information 102 to multiple mobile devices 103a-
103b. In this
instance the light sources are acting as non-nehvorked broadcast beacons;
there are no
networking modules or physical data wires connecting them. This property is
desirable when
looking towards a commercial installation of numerous LED light sources 103a-
103b, as
additional wiring and networking will not be required. However, in order to
receive relevant
information the mobile devices have the ability to send and receive additional
information
from a local source or a network 601. Once the mobile device 103 receives
identification
information 401 from the light sources, it then asks a local or remote source
for additional
information.
[0122] Enclosed area 602 is a spatial representation of an enclosed mom
containing four
LED sources 101a-101d and two mobile devices 103a-103b, meaning that they can
operate
next to each other without interference. As a rule of thumb if the received
image feed from
the mobile device secs one or more distinct bright sources of light, it has
the ability to
differentiate and receive the unique information without interference. Because
the light
capture is based on line of sight, interference is mitigated. In this line of
sight environment,
interference can arise when the light capture mechanism of the mobile device
is blocked from
the line of sight view of the light source.
[0123] Network 601 represents a data network which can be accessed by
mobile devices
103a-103b via their embedded network adapters 503. The network can consist of
a wired or
wireless local area network (LAN), with a method to access a larger wide area
network
(WAN), or a cellular data network (Edge, 3G, 4G, LTS, etc). The network
connection
provides the ability for the mobile devices 103a-103b to send and receive
information from
additional sources, whether locally or remotely.
101241 FIG. 7 describes how the mobile device 103 receives location data
702. In
essence, the mobile device 103 sends decoded ID codes 701 through a network
601 to a
server 703, which sends back location information 702. The decoded ID codes
701 are found
in the information 401, which is contained in the optically transmitted
signal. After receiving
this signal containing a unique ID code 701 the mobile device 103 sends a
request for
location data 702 to the server 703, which sends back the appropriate
responses. Additionally
the request could include other sensor data such as but not limited to GPS
coordinates and
accelerometer/gyroscope data, for choosing between different types of location
data 702 and
any additional information.
- 20 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
[0125] Location data 702 is the indoor location information which matches
the received
information 401. The location data 702 corresponds to indoor coordinates which
match the
ID code 701, similar to how outdoor GPS tags known locations of interest with
corresponding information.. The location data 702 could also contain, generic
data associated
with the light identification information 401. This could include multimedia
content,
examples of which include recorded audio, videos, and images. The location
data 702 can
also vary depending, for example, on other criteria such as temporal criteria,
historical
criteria, or user-specified criteria.
101261 The temporal criteria can include the time of day. The historical
criteria can
include user location history (e.g., locations visited frequently), Internet
browsing history,
retail purchases, or any other recorded information about a mobile device
user. The user-
specified criteria can include policies or rules setup by a user to specify
the type of content
they wish to receive or actions the mobile device should take based on
location information.
For example, the user-specified criteria can include how the mobile device
behaves when the
user is close to an item that is on sale. The user may specify that a coupon
is presented to the
user, or information about the item is presented on the mobile device. The
information about
the item can include videos, pictures, text, audio, and/or a combination of
these that describe
or relate to the item. The item can be something that is for sale, a display,
a museum piece,
or any other physical object.
101271 Server 703 handles incoming ID codes 701, and appropriately returns
indoor
location data 702 to the mobile devices 103. The handling can including
receiving incoming
ID codes, searching databases to determine matches, calculating position
coordinates based
on the ID codes, and communicating indoor location data 702. Since the LED
light sources
101 are acting as "dumb" one way communication beacons, it is up to other
devices to
determine how to use the ID codes to calculate position information and
deliver related
content. In some embodiments, the server 703 can include the information used
to link ID
codes 701 to physical spaces and to deliver location-specific content. The
server is designed
to handle the incoming requests in a scaleable manner, and return results to
the mobile
devices in real-time.
101281 The server can include one or more interfaces to the network that
are configured
to send and receive messages and information in a number of protocols such as
Internet
Protocol (IP) and Transmission Control Protocol (TCP). The protocols can be
arranged in a
stack that is used to communicate over network 601 to mobile device 103. The
server can
also include memory that is configured to store databases and information used
in providing
-21-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
position coordinates and related location based content. The server can
include one or more
modules that can be implemented in software or other logic. These modules can
perform
calculations and perform operations to implement functionality on the server.
The server can
use one or more processors to run the modules to perform logical operations.
101291 To describe the server interaction in more detail, FIG. 8 delves
into location-
specific areas 801 containing databases 802 and web services 803. The areas
801 represent a
subset of databases 802 and web services 803 for individual locations where
there are
installed LED light sources 101. The server 703 directly communicates with
these
installations, which have their own separate sets of information. At a high
level, databases
802 represent the stored information pertaining to a specific area 801, while
the web services
803 represent services which allow users, customers, administrators, and
developers access to
the ID codes, indoor locations, and other information.
101.301 In order to send relevant information, after each received ID code
701, the server
703 requests information pertaining to the specific area 801. Contained in
each area 801, arc
databases which contain information corresponding to the specific ID code 701.
This
information can take multiple formats, and has the ability to be content
specific to a variety of
static and dynamic parameters.
101311 In order to optimize response time, the server 703 can. constrain
its search space
by using existing positioning technologies available to the mobile device 103
or from
information in the light source ID code depending on the embodiment. In
essence the server
looks for the light IDs 901 within a specific radius of the current
approximate position of the
mobile device 103, and ignores those that are geographically irrelevant. This
practice is
known as "geo-fencing", and dramatically reduces the request/response time of
the server
703. As final verification, if the database 802 contains one or more of the
same IDs within
the current search space that match the ID codes received by the mobile device
103 within a
specific time frame, then a successful transaction can be assumed.
101321 As seen in FIG. 9, each database 802 contains numerous sub-
categories which
store specific types of information. The categories are labeled light IDs 901,
maps 902,
content 903, and analytics 904.
101331 Light IDs 901 is a category which contains records of the individual
light ID
codes 701 which are contained in an area 801. In a typical light positioning
enabled
installation, there will be tens to hundreds of unique LED light sources 101
broadcasting
unique ID codes 701. The purpose of the light IDs 901 database is to maintain
and keep a
record of where the ID codes 701 are physically located in the area 801. These
records can
- 22 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
come in the form of but are not limited to GPS (latitude, longitude, and
altitude) coordinates
which are directly mapped into an indoor space. For instance, most indoor
facilities have
information about the number of installed lights, how far apart they are
spaced, and how high
the ceilings are. You can then match this information with building floor
plans or satellite
imagery to create a digital mapping of where each light is positioned.
101341 To expand upon the Light IDs 901 category, additional information
can come in
the form of location-specific maps 902. These maps can take on many physical
and digital
forms, either directly from the management of the location, or a third-party
vendor or outside
source. In addition to mapping information, location-specific content 903 and
analytics 904
are also contained inside the databases 802.
101351 FIG. 10 is a description of the ID log 1001 information contained in
the Light IDs
database 901. It is a representation of the file structure that contains
individual records
corresponding to individual light ID codes 701 found within different areas
801. In a typical
area 801 there is a possibility of having duplicate ID codes 701 since there
arc a finite
number of available codes. The size of the ID code 701 is proportional to the
length of the
data 403 field contained in the optical information 401.
101361 To deal with duplicate ID codes 701, additional distinguishing
information can be
contained inside of the individual log records; ID 11001, ID 2 1003, and ID 3
1004. This
information can contain additional records about neighboring ID codes 701
which are in
physical proximity of the LED light source 101, or additional sensor data
including but not
limited to: accelerometer or gyroscope data, WiFi triangulation or
fingerprinting data, GSM
signature data, infrared or Bluetooth data, and ultrasonic audio data. Each
additional sensor
is an input into a Bayesian model that maintains an estimation of the current
smartphone
position and the uncertainty associated with the current estimation. Bayesian
inference is a
statistical method used to calculate degrees of probability due to changes in
sensory input. In
general, greater numbers of sensory inputs correlate with lower uncertainty.
101371 In order to calibrate the light-based positioning system., a user
equipped with a
specific mobile application will need to walk around the specific area 801.
The mobile
application contains map 902 information of the indoor space, with the
positions of the LED
light sources 101 overlaid on the map. As the user walks around, they will
receive ID codes
701 from the lights. When the user receives an ID code 701, they will use the
map on the
mobile app to select which LED light source 101 they are under. After the user
confirms the
selection of the light, the mobile application sends a request to the server
703 to update the
light location contained in the lighting plan 1103 with the ID code 701.
Additional user-
- 23 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
provided 1104 tinetadata including but not limited to current WiFi access
points, RSSI, and
cellular tower information can also be included with the server request to
update additional
databases.
101381 In addition to manual calibration, calibration of LED light source
101 locations
can also be achieved via crowd-sourcing. In this algorithm, as mobile
application users move
around an indoor space receiving ID codes 701, they will send requests to the
server 703
containing the light ID code 701 received, the current approximate position
(based on other
positioning techniques such as WiFi, GPS, GSM, and inertial sensors) and the
erior of the
current approximation. Given enough users, machine learning algorithms on the
server 703
can be used to infer the relative position of each LED light source 101. The
accuracy of this
calibration method depends heavily on the number of mobile application users.
101391 FIG. 11 is a description of the maps database 902 and map log 1101
information
containing floor plans 1102, lighting plans 1103, user-provided information
1104, and
aggregated data 1105. Map log 1101 is a representation of the file structure
that contains the
information found inside the maps database 902. Information can come in the
form of but is
not limited to computer-aided drafting files, user-provided computerized or
hand drawn
images, or portable document formats. The information residing in the maps 902
database
can be used both to calibrate systems of multiple LED light sources 101, and
to augment the
location data 702 that is sent to mobile devices 103.
101401 Floor plan 1102 contains information about the floor plan for
specific areas 801.
The contained information can be in the form of computer-aided drafting files,
scanned
images, and legacy documents pertaining to old floor plans. The information is
used to build
a model corresponding to the most recent building structure and layout. These
models are
subject to changes and updates through methods including but not limited to
crowd sourcing
models where users update inaccuracies, third-party mapping software updates,
and
additional input from private vendors.
101411 Lighting plan 1103 contains information about the physical lighting
fixture layout,
electrical wiring, and any additional information regarding the lighting
systems in the area
801. This information can also come in a variety of physical and digital forms
such as the
floor plan 1102 information. The lighting plan 1103 information is used in the
calibration
process of assigning light ID codes 701 to physical coordinates within an area
801. In
essence, a location with multiple LED light sources 101 acts as a large mesh
network except,
in this case, each node (light ID 701) is a non-networked beacon of
information that does not
know about its surrounding neighbors. To help make sense of multiple light ID
codes 701,
-24-
CA 02992923 2015-05-28
WO 2014/063150 PCT/US2013/065923
the lighting plan 1103 information is used as one of many ways to tell the
backend server 703
where LED light sources 101 are located.
101421 User-provided information 1104 contains additional data that the
user manually
uploads in regards to building changes, updates, or new information that is
acquired. The
user in this case is most likely the facility manager or staff member, but
could also originate
from an end user of the system who contributes via a crowd sourcing or machine
learning
mechanism. For instance, if an end user was using a light based positioning
system in a
museum and was unable to find a particular exhibit or noticed inaccurate
information in
regards to location or classification of the exhibit, they could red flag the
occurrence using
their mobile device 103. When coupled with data from additional users,
sometimes known as
a crowd sourcing method, this user-provided information 1104 can be used to
update and
repair inaccuracies in the maps 902 database.
101.431 Aggregated data 1105 contains information, that is gathered by the
system that can
be used to augment the current information that is known about the mapping
environment.
This can occur during normal operation of the system where multiple mobile
devices 103 are
constantly sending and receiving location data 702 from the server 703. Over
time the
aggregation of this data can be used to better approximate how light ID codes
701 correspond
to the physical locations of the LED light sources 101. For instance, if
multiple mobile
devices 103 consistently receive a new ID code 701, in a repeatable pattern
with respect to
additional known ID codes 701 and other sources of location information, then
this
information can be recorded and stored in the aggregated data 1105 database.
This
information can additionally be used to recalibrate and in essence "self-heal"
a light-based
positioning system.
101441 FIG. 12 is a description of the content database 903 and content log
1201
information containing static content 1202, user-based content 1203, and
dynamic content
1204. Content log 1201 is a representation of the file structure that contains
the information
found inside the content database 903. Static content 1202 refers to
unchanging information
that is associated with the specific area 801. This can refer to the previous
example where a
facility manger loads specific content into the content 903 database before a
user enters the
specific area 801. This type of information can take the form of but is not
limited to audio
recordings, streaming or stored video files, images, or links to local or
remote websites.
101451 User-based content 1203 refers to content that is dependent on user
criteria. The
content can depend on but is not limited to user age, sex, preference, habits,
etc. For
instance, a male user might receive different advertisements and promotions
than a female
-25-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
would. Additionally, age and past purchase habits could also be used to
distinguish which is
the correct piece of content to be presented to the user.
101461 Dynamic content 1204 refers to content which changes with varying
frequency.
The content can change dependent on a temporal bases, daily, weekly, monthly,
etc. For
instance, seasonal marketing and content could be automatically presented to
the user
dependent on the month of the year, or content in the form of morning,
evening, or nightly
specials could be presented numerous times throughout the individual day.
101471 In addition to content, point of purchase 1205 information can be
delivered as
well. This could be implemented by using the received ID code 701 to a secure
connection
which establishes and completes a transaction linked to a user's selected
payment method.
Additionally, a standalone point of purchase feature could be implemented by
simply linking
ID codes 701 directly to merchandise or services.
101481 FIG. 13 is a description of the analytics database 904 and analytics
log 1301
information containing frequency 1302, dwell time 1303, path taken 1304, and
miscellaneous
1305. Analytics log 1101 is the file structure that contains the information
found inside the
analytics database 904. Frequency 1302 refers to the number of times each end
user visits a
particular location inside of a specific area 801. Separate records are
maintained for
individual users, and the frequency is aggregated and sorted in the frequency
files database
904.
101491 Dwell time 1303 refers to the time spent in each particular location
inside a
specific area 801. Separate records are maintained for individual users, and
the dwell times
are aggregated and sorted in the dwell time file. Path taken 1304 refers to
the physical path
taken by a user in each specific area 801.
101501 Consider an example that combines many of the above descriptions,
involving a
store owner that installed a light-based indoor positioning system and a
customer walking
around the store using a mobile device 103 capable of receiving optically
transmitted
information. The customer drives to the parking lot of the store, parks, and
walks in. Using
the background sensors and location services available to her phone as modeled
in FIG. 16,
the customer's mobile device 103 already knows that she has approached, and
most likely
entered a store outfitted with a light-based positioning system. Once this
information is
known, the application ninning on the customer's mobile device 103 initiates
several
background services and begins to start looking for optical signals as
depicted in FIG. 15.
101511 Prior to the customer entering the store, the store owner has
already calibrated and
preloaded the database 802 with the unique LED light sources 101, map 902
information
- 26 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
pertaining to the store floor plan 1102, user-provided 1104 product locations,
and content 903
in the form of multimedia and local deals in the form of promotions that can
only be activated
by visiting that particular section of the store.
101.521 In the meantime, the customer is walking around the store looking
to find
particular items on her shopping list which she has already digitally loaded
onto her mobile
device 103. Next, the customer is prompted by her mobile device 103 that one
of the items
on her list has moved locations and an image of the store layout is displayed
with a flashing
icon indicating where her desired product has moved. The mobile phone can
guide her to the
new product. Then as soon. as she gets close to the product, an informational
video is
prompted on her screen detailing the most popular recipe incorporating that
product and how
it is prepared. Finally, in addition to finding her desired product, the
customer receives a
discount promotion for taking the time to seek out the new location of the
product.
101.531 In addition to the services offered by this system. to the
customer, the store owner
now gains value from learning about the shopping experiences of the customer.
This comes
in the form of aggregated data that is captured and stored in the analytics
904 section of his
store's database 802. This example is one of many applications that can be
enabled with an
accurate indoor light-based positioning system.
101541 FIG. 14 is a process describing the act of receiving location and
content
infonnation through visible light. User places mobile device under light 1401
corresponds to
the act of physically placing a camera equipped mobile device 103 underneath
an enabled
LED light source 101. The user stands approximately underneath or adjacent the
LED light
source 101, and the mobile device has the LED light source 101 in view of the
camera lens.
101551 The next block, sample image sensor 1402, refers to the act of
turning on and
reading data from the embedded image sensor in the mobile device 103. Receive
ID? 1403 is
a decision block which either moves forward if a location ID is received, or
returns to sample
the image sensor 1402. Get location data corresponding to ID from server 1404
occurs once
a location ID has been received. The mobile device queries the server asking
for location
data 702 relevant to the ID code. This describes the process of a user
obtaining an ID code
701 from a non-networked LED light source 101, and using the unique identifier
to look up
additional information from either the server 703 or a locally stored source.
101561 Finally, content? 1405 is another decision block which determines if
there is
location-based content associated with the received ID code. If content is
available the
process continues on to the last block 1406 where the content is queried; if
not, the process
ends. As described above, the get content data corresponding to ID from server
1405 refers
- 27 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
to the act of retrieving content data associated with a known location from
either a server 703
or local source.
101571 FIG. 15 is a process describing the act of turning on the
application background
services and determining when to sample the image sensor. Initiate background
service 1
1501 is the primary background running service on the mobile device. This
service is tasked
with initiating a function that can communicate wirelessly to determine if the
mobile device
is close to an enabled area. The wireless communication includes radio
frequency
communication techniques such as global position system (GPS), cellular
communication
(e.g., LIE, CDMA, UMTS, GSM), or WiFi communications. Determine position 1502
is the
function that periodically samples the wireless communication signal and based
on distance
parameters decides whether or not the mobile device is close enough to an area
to move
forward to the next service.
101.581 Light positioning enabled? 1503 is a decision block that moves
forward if the
mobile device is close to an enabled location, or repeats the previous
function if not. Initiate
background service 2 1504 is activated once the mobile device enters an
enabled area. The
service is tasked with initiating the functions that receive location
information via the
modulated light.
101591 Sample ambient light sensor 1505 is the first function of the
previous service
which samples the ambient light sensor data as soon as the sensor detects a
change. The
function of this task is to determine if the sensor has gone from dark to
light, if the user takes
the device out of a pocket or enclosure, or from light to dark, the user has
placed the device
inside of a pocket or enclosure. As an alternative to sampling the light
sensor, the algorithm.
could also look for a change in the accelerometer reading. This would
correspond to the user
taking the phone out of their pocket. Detect change? 1506 is the decision
block that moves
forward if the ambient light sensor has gone from dark to light, meaning that
the mobile
device is potentially in view of surrounding modulated light.
1.01601 FIG. 16 is a process describing the act of determining a mobile
device's position
using a variety of information sources. Sample GPS/GSM 1601 refers to the act
of
determining if the mobile device is close to an enabled area. Enabled area?
1602 is a decision
block which moves forward if the mobile device is close to a enabled area, or
returns to the
previous block if not.
101611 Sample alternative sources 1603 refers to the act of leveraging
existing alternative
positioning technologies such as WiFi, Bluetooth, ultrasound, inertial
navigation, or
employing an existing service using one or more of any available services.
Record internal
-28-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
sensor data 1606 is a task which records the current accelerometer data for a
period of time
before returning to the Sample image sensor 1402 block. This task is performed
so that
location information is constantly being collected even when modulated light
is not being
detected. This allows the mobile device and/or server to keep track of the
mobile device's
position.
101621 FIG. 17 is a system diagram describing how a client device 1704
interacts with a
light-based positioning system 1709. Network 601 is a generic local or remote
network used
to connect mobile devices 103 contained in locations A 1701, B 1702, and C
1703 with the
light-based positioning service 1709.
101631 Each location contains multiple LED light sources 101, each of which
broadcast
unique identification codes 701. In order to interact with the system from an
operator's
perspective, a mobile device can use the database service application 1710
which contains
multiple privilege levels for different levels of access. The client privilege
level determines
read/write permissions to each of these databases. These levels include users
1705 which
refer to general front end system. users, administrators 1706 which are
usually IT or
operations management level within an installation, developers 1707 which have
access to
the application programming interfaces of the system for use in custom
application
development, and root 1708 level which contains master control over the users
and access to
everything contained in the system and databases.
101641 Mobile devices in each location 1701, 1702, and 1703 receive
identification codes
701 from lights in their respective locations. They then send the received
identification codes
701 through the network 601 which connects to database service application
1710, through
user application 1705, and has read access to maps 902 and content, and write
access to
analytics 904. A generic client, 1704, connects to database service
application 1710 through
network connection 601.
101651 The client uses a password authorized login screen to access the
respective
permission status. Clients with administrator permissions have read/write
access to light Ills
901, read access to maps 902, read/write access to content 903, and read
access to analytics
904. Clients with developer permissions 1707 have read access to light .I.Ds,
read access to
maps 902, read/write access to content 903, and read access to analytics 904.
A client with
root permissions 1708 has read/write access to databases 901-904.
101661 As an overview, FIG. 17 describes the top down approach to our
current
implementation of a light-based positioning system. At the highest level,
known locations of
installed non-network standalone LED light sources 101 are used to accurately
identify the
- 29 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
relative position of mobile devices 103. In order to obtain identification
information from the
lights, the background processes running on the mobile device 103 have been
described in
FIGs 14, 15, 16. Once the mobile device has acquired a unique or semi-unique
ID code 701
from the light or combination of lights, it uses this information to query a
database 802 for
additional information. This information can come in many forms, and is used
to create a
more personalized experience for the user. As initially mentioned, this local
experience is
used for location aware mobile computing, and augmented reality applications.
In addition to
local personalized information, location based analytics applications can be
enabled from the
aggregated data and traffic running through the server 703.
101671 The use of light-based positioning capabilities provide a number of
benefits. For
example, the positioning information obtained by using light sources is highly
precise
compared to alternative techniques for positioning information. The accuracy
of a light-
based positioning system. can be down to a few centimeters in three dimensions
in some
embodiments. This positioning ability enables a number of useful services to
be provided. In
certain embodiments, additional mobile device information, can be used in
combination with
the positioning information. For example, accelerometer position information
can be used in
conjunction with light source based position to offer augmented reality or
location aware
content that relevant to the device's position. The relevant content can be
displayed to
augment what is being displayed on the mobile device or the display can
provide relevant
information. Applications on the mobile device can also be launched when the
mobile device
enters certain areas or based on a combination of criteria and position
information. The
applications can be used to provide additional information to the user of the
mobile device.
101681 The light-based positioning systems and methods can also be used to
manage and
run a business. For example, the light-based positioning can help keep track
of inventory and
to make changes to related databases of information. In a warehouse, for
example, the light-
positioning system can direct a person to where a particular item is located
by giving
directions and visual aids. The light positioning can even provide positioning
information to
direct the person to the correct shelf the item is currently residing on. If
the person removes
the item., the mobile device can update the inventory databases to reflect the
change. The
same function can be implemented in a store environment as merchandise
locations are
changed or updated. This information can then be used in providing content to
a user. For
example, if a shopper wants more information about an item, the updated
location can be
used to locate the item or direct the shopper to an online website to purchase
an out-of-stock
item. In some embodiments, the mobile device using the light-based positioning
technique in
-30-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
conjunction with a wireless connection and other information can be used to
provide non-
intrusive data collection on customers. The data collection of how customers
move through a
store and where they spend time can be used to improve layout of stores and
displays of
merchandise.
101691 The light-based positioning systems are also easy and low cost to
setup compared
to other location positioning systems. Since each light source operates
autonomously, a
building owner only needs to swap out existing light sources for those that
provide light-
based information to a camera enabled device. The light sources are non-
networked
independent beacons that broadcast identification codes configured when
manufactured. This
allows the light sources to be manufactured at a lower cost compared to
networked light
sources. Further, the non-networked independent beacon light sources in the
light-based
positioning system can be easier for building owners to install.
101.701 The light-based positioning system. can also include optimizations
in some
embodiments. For example, location information obtained from either the
identification code
or from alternative techniques can be used to reduce latency in determining
position
information. This optimization can work through geo-fencing by constraining
the search area
to find information regarding the captured light sources more quickly. This
can reduce the
overall delay experienced by a user from the time the mobile device captures
the light sources
to when relevant position information is provide to the mobile device and/or
relevant content
is provided to the mobile device.
Efficient Light Bulbs for DPR Schemes
101711 One of the biggest challenges facing beacon based light positioning
systems is
managing the additional power consumption of communication-enabled lighting
devices in
comparison to that of non-communicating devices. Lighting sources 101 in
general,
regardless of form factor or technology, are differentiated in part by their
power
consumption; generally, th.e less the better. Accordingly, higher energy
efficiency is one of
the core economic forces driving adoption of Light-Emitting-Diodes (LEDs).
However,
when using light sources 101 as a means for communication devices, the power
requirements
tend to increase depending on the modulation scheme since energy must be
divided between
the carrier wave and the modulation wave. There are many different techniques
for
transmitting data through light, as discussed in Prior Art such as US
12/412,515 and US
11/998,286, and US 11/591,677. However, these techniques have majorly been
pursued
without considering their impact on light source 101 parameters, including
efficacy, lifetime,
-31-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
and brightness. Since light sources 101 are first and foremost illumination
devices, and not
communication devices, the communication function takes a secondary role. The
present
disclosure utilizes Digital Pulse Recognition (DPR) modulation as a technique
for
transmitting data while minimizing the impact on illumination devices.
101721 FIGS. 18A-C represent several digitally modulated light sources 101a-
c with
varying duty cycles; a low duty cycle 1801, a medium duty cycle 1802, and a
high duty cycle
1803. A duty cycle is a property of a digital signal that represents the
proportion of time the
signal spends in an active, or "on," state as opposed to an inactive, or
"off," state. A light
source with a low duty cycle 1801 is inactive for a high proportion of time. A
light source
with a medium duty cycle 1802 is inactive for about the same proportion of
time that it is
active. A light source with a high duty cycle 1803 is active for a high
proportion of time.
The duty cycle of a light source affects the luminosity of the light source. A
light source
having a higher duty cycle generally provides more luminosity than that same
light source
with a lower duty cycle because it is on for a higher proportion of time. Duty
cycle is one
aspect of a modulation scheme. Other aspects include pulse shape, frequency of
pulses, and
an offset level (e.g., a DC bias).
101731 Because DPR modulated light sources 101 rely on frequency
modulation, they are
able to circumvent the limitations of traditional AM based approaches. Note
that frequency
modulation in this context does not refer to modifying the frequency of the
carrier (which is
the light signal), but instead to modifying the frequency of a periodic
waveform driving the
light source. One popular technique for dimming LED light sources 101 is with
pulse width
modulation (PWM), which controls the average power delivered to the light
source by
varying the duty cycle of a pulse. In a DPR modulation system utilizing PWM, a
DPR
modulator would control the frequency of the pulses, with the duty cycle
determined by the
dimming requirements on the light source 101. As used herein, a DPR modulated
light
source, having a DPR modulation frequency, refers to a light source having an
output
modulated in such a manner that a receiver using DPR demodulation techniques
can
demodulate the signal to extract data from the signal. In some embodiments,
the data can
include information in the form of an identifier which distinguishes a light
source from other
nearby DPR modulated light sources. In some embodiments, this identifier may
be a periodic
tone that the light source randomly selects to identify itself. A periodic
tone may be a signal
that repeats with a given frequency. In other embodiments, a light source may
receive such
an identifier from an external source.
-32-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
101741 To determine the maximum duty cycle (D) supported by DPR
demodulation, the
modulation frequency (f) of the transmitter and the sampling time for the
image sensor (Ts) of
the receiver are first defined. Next the duty cycle parameters (Ton) and (Ton)
are defined
which correspond to the on and off times of the light source. Ts is an
important parameter
because the image sensor sampling time defines a minimum amount of modulation
time
required to produce the banding effects which allow for the frequency
detection required for
DPR demodulation. The required modulation time can refer to either the Tõõ
portion 1804 or
the Toff portion 1805 of the signal, however to maximize the brightness of the
light source Toff
is used as the limiting variable (if solving for the minimum duty cycle, T.
can be used). If Ts
of the receiving device is less than twice Toff of the light source, residual
banding on the
image sensor will not take place; therefore the signal cannot be extracted. in
order for
banding to occur, Ts should be greater than twice the value of Toff
(Ts>2*Toff)-
101.751 It is important to note that when designing for the maximum duty
cycle, the
modulation frequency can be defined from the transmitter side and can be
completely
independent of the sampling time T. This is because the sampling frequency Ts
is a property
of the receiver which is defined by the image sensor manufacturer and is
likely not designed
for optimal DPR demodulation properties. Ts varies depending on the specific
image sensor,
and can be expected to change as more advanced image sensors are developed.
Therefore it
is important to optimize such that a broad range of both modulation and
sampling frequencies
can be used. in the next sections the equations and variables for how to
calculate the
maximum duty cycle are described for a variety of test cases.
101761 In order to solve for Toff in terms of duty cycle and modulation
frequency, one can
first start with the fundamental definition of what the duty cycle is: 1 minus
the ratio of
signal on time divided by the combination of signal on and off time. In the
case of a
modulated light source, D=1-1-0ATõ.+Too. Next the modulation frequency (f) can
be
defined as the inverse of the sum of signal on and off times: f=1/(T0n+T0ff).
Substituting f into
the previous equation for D yields D=1-tn0ff. The variable Toff, which was
previously
defined as a value less than twice Ts, can then be used to define the maximum
duty cycle for
any given modulation used in DPR demodulation. After rearranging and
substituting Ts for
Toff (T0111.5*TO, D=1-P(112)*(Ts). With this equation, we can now solve for
the maximum
duty cycle achievable given the modulation frequency of the transmitter, and
the sampling
time of the receiver.
101771 Since the maximum duty cycle is dependent on both the modulation
frequency of
the transmitter and the sampling frequency (F3=1/Ts) of the receiver, its
exact percentage
-33-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
value can change depending on the present conditions. For testing purposes,
the modulation
frequency range was chosen to start at 300 Hz which is above the range which
the human eye
can see. The modulation frequency range may range from 60Hz to 5000Hz. Typical
image
sensor sampling frequencies (Fs-1/Ts) range between 20kHz and 36kHz for high
quality
image settings (640 by 480 pixel resolution), and 4KHz to 7KHz for low quality
image
settings (192 by 144 pixel resolution). In some embodiments, the image sensor
sampling
frequencies may range from as low as 1 KHz to as high as 1 MHz.
101781 When analyzing specific use cases, the duty cycles corresponding to
a modulation
frequency of 300 Hz and sampling frequencies for high quality image settings
in some
embodiments result in D=1-(300Hz)*(1/2)*(1/20Khz) = 99.25% and D=1-
(300Hz)*(1/2)(1/36kHz) = 99.58%. The duty cycles corresponding to a modulation
frequency of 300 Hz and typical sampling frequencies low quality sampling
frequencies in
other embodiments result in D=I-(300Hz)*(1/2)*(1/4kHz) = 96.25% and D=1-
(300Hz)*(112)*(1/7kHz) = 97.86%. In yet other embodiments, a 2000 Hz
modulation
frequency and high quality sampling frequencies of 20kHz and 36kHz results in
D = 95.00 %
and 97.22% respectively, and for low quality sampling frequencies of 4k1lz and
7ktiz results
in D = 75% and 85.71% respectively.
101791 After the maximum duty cycle has been calculated, to compensate for
the
additional power requirements needed for data communication due to the off
portion 1804 of
the modulation signal, the input power can be increased such that the
resulting average power
of the communicating light source 101 is identical to the non-communicating
light source
101. In effect the average power of the two light sources will be the same,
yielding a
perceivably identical luminous output. Take for instance LED source "A" which
is powered
by 6 Watts and modulated where 50% of the time it is "on", and the remaining
50% "off",
effectively resulting in a 3 Watt average power. In order for this light
source 101 to match the
luminous output of the 6 Watt LED source "B" which is not modulating and is on
100% of
the time, one can double the input power from 6 Watts to 12 Watts. While the
input power of
"A" was increased, the average power is halved to equal 6 Watts; therefore
sources "A" and
"B" appear to be identical to the human eye in terms of brightness.
101801 However, there exists a point where increasing the input power can
decrease the
efficiency of a given light source 101. For T.,ED lighting devices it is
important to stay within
the manufacturer specified voltage and more importantly current, otherwise
efficiency
drastically falls with increased supply current. This unwanted effect is known
as LED
"droop", and generally refers to decreased luminous output for any given
individual LED
-34-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
(assuming one or more LEDs per lighting source 101) due to the additional
thermal heating
resulting from the increased current. In the previous example, the input power
to I,ED source
"A" was doubled while the input power to "B" was left unchanged. Assuming that
each
source was supplied by a constant 12 Volts, this means that the input current
to source "A"
had to have doubled in order to achieve the required 12 Watts of power
consumption. This
equates to a 50% increase in current, when moving from. .5 Amps to 1 Amp, and
can only be
performed if within the manufacturers' tolerable input current range for the
LEDs.
101811 Given inputs of drive current (Id) and operating voltage (V), we can
define the
power (P) of a non-modulated light source 101 as P=Id*V, and compare it with
th.e additional
required power (Pwod) of a modulated light source 101. To define the
additional power
needed due to modulation, you can then define the relationship as Pssod=P2-
(D*Id*V). While
the input variables used in this example vary from source to source, this
method can be used
to accommodate for power loss due to modulation.
101821 We can now solve for the power required to support the maximum duty
cycles
that were previously solved for. In this example, the power consumed by the
non-modulated
light source equals P=Id*V=700mA*12V=8.4W. Pmod can then be calculated to
describe how
much extra power is required to support a modulated light source 101 with
regard to the duty
cycle. Recall that for a modulation frequency of 2000HZ and sampling
frequencies of 20kHz
and 41cliz, the maximum duty cycle equaled 99.25% and 96.25%. Therefore the
additional
power needed to detect a 2000Hz signal at a sampling frequency of 20kHz is
defined as
Pmod= 8.4W-(.9925*700mA*12V) = 63mW, a .75% increase in required power on top
of the
basel.in.e 8.4W. For 200011z at a sampling rate of 4kHz, Piood...8.4-
(.9625*70OrnA*12V) --=
315mW, a 3.75% increase in required power.
101831 While finding the maximum duty cycle supported by DPR demodulation
is
important for maintaining the brightest luminous output levels, it is also
important to support
the lowest duty cycle possible in order to support the dimmest luminous output
levels. This
is because the minimum duty cycle corresponds to the dimmest level that a
modulated light
source 101 can operate at while still supporting DPR demodulation from a
receiving device.
In. order to account for this, we now consider the Too portion of the signal
rather than Toff.
The limiting sampling factor now changes to require that Ts is greater than
twice Ton
(Ts>2T0n). Substituting this condition into the previous max duty cycle
equation (replacing
{1-D) with D), the resulting equation yields 1:(1/2)*P'T8.
101841 Repeating the above examples for a modulation frequency of 300Hz and
high
quality sampling frequencies WI's) of 20kHz and 36kHz, D = .75% and .42%,
respectively.
-35-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
For a modulation frequency of 2000Hz with high quality sampling frequencies, D
= 5.00%
and 2.78%. Considering lower quality sampling frequencies at 300Hz and 2000Hz,
D =
3.75% and 2.14% for a 300Hz modulation frequency, and D = 25.00% and 14.29%
for a
2000Hz modulation frequency.
101851 In addition to modifying the overall duty cycle, there also exists
the opportunity to
tune the modulation scheme such that during the "off" portion 1805 of
operation the light
source 101 does not turn completely off. As described in FIGS. 19A-C,
modulation schemes
1901, 1902, and 1903 depict varying duty cycles where a DC bias 1904 has been
added
which correspond to the modulated light sources 101a-101c. Modulation schemes
where the
light source 101 does not turn all the way "off" are important when
considering light source
101 brightness, efficiency, lifetime, and the signal to noise ratio (SNR) of
the
communications channel. The DC bias 1904 during modulation reduces the peak
power
required to drive the light source for a given brightness. A. reduction in
peak power will
reduce the negative impact of overdriving the lighting source, which is known
to cause
efficiency losses known as "droop" for LEDs, in addition to decreasing light
source 101
lifetimes.
101861 As an example, consider that the average power delivered to the
light source is
defined as: Pay=D*Pon+(1 D)*Põif, , where D is the duty cycle and P, - Po.:
are the respective
on
on/off powers. The impact on light source 101 brightness is that increasing
the "off" power
will increase the total power. This reduces the required peak power delivered
to the lighting
source, because the power transferred during the "off" period can make up the
difference. In
a system. operating at a duty cycle of 50%, for a fixed brightness B, a 10%
increase in the
"off' period power translates to a 10% decrease in the "on" period power.
101871 When approaching the above power equation from a constant voltage
(V), average
current and onloff current (LAO standpoint (P=IV), 1õ,*V=D*11õ.*V+(i -
D)*LAT*V.
After removing the constant V, Iav=D*Ion+(l-D)Ioff. For example, in the case
of a light source
101 requiring an average drive current (11.õ) of 700mA and off current of (la)
of OA.
undergoing modulation with a duty cycle (D) of 96.25%, the peak current (Ion)
requirement is
1en-700mA/.9625 = 727mA. If instead the current delivered during the "off'
time is 100mA.
the average current reduces to 1õ,,=.9625*700mA-F(1-.9625)*100mA = 678mA, a
6.7%
decrease in overall required power given constant voltage. In other
embodiments, a constant
current may be applied with differing voltages to achieve a similar effect.
101881 The impact of non-zero Ioff values for the previous example is two-
fold. First, a
reduction in required power is achieved, and second increasing the "off" time
power lowers
-36-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
the required duty cycle to achieve a fixed brightness level. For the previous
example when
solving for D, D=(1õ,4off)/(10,400. The difference in duty cycle can now be
determined for
the reduction in peak current from 727mA to 678mA, as D----(700mA-
100mA)/(727mA-
100mA)=- 95.69%, which is a .56% difference from 96.25%. This essentially
allows for a
brighter light source 101 with a decreased duty cycle, and lower power
requirements.
101891 Another major requirement for DPR modulation is to interface with
existing light
dimmers. There are a variety of light source 101 dimmers employed on the
commercial
market. One popular dimming technique is triac dimming. In a triac dimmer, a
variable
resistor switch is used to control the amount of power delivered to the light
source 101 over
the AC line. For traditional incandescent and fluorescent sources this is a
cost effective and
efficient way to control the power, and thus the brightness, delivered to the
light source 101.
For LED light sources 101, it is necessary to put a special driver between the
trim dimming
circuit and the LED source. This is because LEDs are current driven devices,
and thus
require an AC/DC converter to transform AC from the power lines to a DC
current for
driving the LEDs.
101901 FIG. 20 demonstrates a system by which a DPR modulator can interface
with
existing lighting control circuits. A dimmer controller 2002 sends a dimmer
signal 2003 to a
dimmable LED driver 2006. In the case of an LED light source controlled by a
triac dimmer,
the dimmer signal would be transmitted across the AC power line. The dimmable
LED
driver 2006 then converts the dimmer signal to a pulse width modulated signal
used for
driving the light output 2007 of the source 2001. The configuration of the
system diagram
shows the dimmer signal 2003 going to both the DPR modulator 2004 and the LED
driver
2006, however this does not always need to happen. In some instances the LED
driver 2006
can contain a "master override" input which is designed to supersede any
dimmer signal 2003
input. In this case the dimmer signal 2003 still goes to the LED driver 2006,
but is ignored.
In other cases where there is not an override input, the dimming signal only
goes to the DPR
modulator.
[0191] DPR modulator 2004 is responsible for sending DPR signals 2005 to
the LED
driver 2006 which controls the light output 2007. In. the case of the light
source 2001 being
driven by pulse width modulation as the dimmer signal 2003 from the dimmer
controller
2002, DPR modulator 2004 controls the frequency of the PWM signal and selects
the desired
value. The width of pulses in signals 1801-1803 are determined based on dimmer
signal
2003, which indicates the desired light source 2001 brightness level. Note
that the dimmer
controller 2002 is not contained within the light source 2001, and can output
a variety of
-37-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
dimmer signals 2003 (triac, or a proprietary method). Because of this, the DPR
modulator
2004 is responsible for interpreting these different signals and appropriately
outputting a DPR
signal 2005 which corresponds to the desired brightness level of the inputted
dimmer signal
2003. In cases where dimming is not required and th.e dimmer signal 2003 is
not present, the
DPR modulator 2004 interfaces directly with the LED driver. In some
implementations, the
DPR modulator 2004 can also be contained inside the LED driver 2006 as part of
an
integrated solution instead of as a separate component.
101921 FIG. 21 contains a high level overview of a DPR modulator 2004. Data
2101 is
first sent to DPR tone generator 2102. Data 2101 could contain information
from any source.
In the context of a beacon based light positioning system, data can include
the identifier for
the light. DPR tone generator 2102 converts the data 2101 into a sequence of
DPR tones. A
DPR tone is a periodic digital signal that oscillates between active and
inactive states with a
particular frequency. This process is described further in FIG. 22. Depending
on the
requirements of the data transmission channel, this could either be a single
tone (suitable for a
beacon based positioning system using light identifiers), or a sequence of
tones (if higher data
rates are desired by the end user). The DPR Tone(s) 2203 are then sent to the
waveform
generator 2103, which is responsible for generating the DPR signal 2005 for
driving the
LEDs. Waveform generator 2103 receives a dimmer signal 2003 input from a
dimmer
controller 2002, which controls the brightness of the light source. In the
case of a DPR tone
as a pulse-width-modulated signal, dimmer controller 2002 would control the
duty cycle of
square wave 1802, while DPR Tone(s) 2203 would control the frequency of the
square wave.
The result is an output DPR signal 2005, which is then sent to the LED driver
2006.
101931 FIG. 22 contains a breakdown of DPR Tone Generator 2102. This module
is
responsible for taking a piece of data and converting it to a sequence of DPR
tones. A DPR
tone determines the frequency at which a waveform, such as the square waves
from FIG. 18,
is sent. The range of possible tones, defined here in as T. through T., is
determined by both
the sampling time, Ts, of the image sensor (as discussed in paragraph 0006),
and the
frequency response of the light source 101. Encoder 2201 is a standard base
converter ¨ it
takes a piece of data in binary and converts it into a corresponding DPR tone.
A typical
range for tones created by DPR Tone Generator 2102 is 300Hz-2000Hz, in steps
of 10Hz,
allowing for 170 distinct DPR tones. The step size between tones is selected
to reduce noise,
and depending on the requirements could be much higher or lower than 10 Hz. As
an
example, that data 2101 may contain an identifier of value 10 for light source
101. This
identifier is passed to Tone(s) Generator 2102, which generates (or selects
from memory) a
-38-
CA 02892923 2016-12-07
sequence of tones. Note that the length of a DPR tone sequence could be as low
as 1 (in the case
of a single tone used in a beacon based positioning system). In this example,
an identifier of 10
would map to a DPR tone of 400Hz. DPR Tone Generator 2102 could either store
the identifier
in memory beforehand, using pre-computed mappings of data to tone sequences,
or alternatively
it could compute this on the fly. The exact method of generating the sequence
of tones would be
driven by the resources available on the light source 101. Once one of the
possible tones
sequences 2202 is created, it is sent to Waveform Generator 2103.
[0194] FIG. 23 contains the breakdown of Waveform Generator system 2103, which
combines a
tone sequence 2202 with a waveform from symbol creator 2303 and dimmer signal
2003 to
create a DPR signal 2005 for driving light source 101. The resulting waveform
will be periodic,
with a frequency defined by the sequence of tones, a symbol created based on
the list of possible
symbols in symbol creator 2303, and an average output (brightness) determined
by the dimmer
signal 2003. This desired brightness could either be hard-coded on the module,
or provided as an
external input through a dimming control module. The choice of a symbol is
determined within
Symbol Selector 2301, which generates a control line 2302 for selecting a
symbol from symbol
mux 2402.
[0195] FIG. 24 contains the breakdown of Symbol Creator 2303, which holds
possible symbols
2401a-2401d. These could include a saw tooth wave 2401a, sine wave 2401b,
square wave
2401c, and square wave with a DC offset 2401d, or any other periodic symbol.
Symbol creator
then takes in a selected symbol 2402, and modifies it such that a desired
brightness 2106 is
achieved. In the case of a square wave symbol 2401c, dimmer signal 2003 would
modify the
duty cycle of the square wave. The resulting waveform is then sent to output
signal 2005 for
driving the light source.
[0196] The goal of the output waveform 2105, which drives light source 101, is
to illuminate a
scene in such a way that the DPR modulated signal can be picked up on any
standard mobile
device 103. Reducing flicker on video which is under illumination from
fluorescent lamps is a
well-known problem. The flicker is caused by periodic voltage fluctuations on
the AC line
powering the lamp. For a lamp powered by a 50 Hz AC line, the luminance level
changes at 100
- 39 -
CA 02892923 2016-12-07
Hz. This causes alternating white/dark bands to appear in video recorded with
CMOS imagers.
The bands are a result of the rolling shutter mechanism on CMOS imagers, which
partially
expose different areas of the image at different points in time. The lines on
the image can occur
on both, one, or on multiple frames, and may appear to move in time. See, for
example, U.S.
Patent No. 6,710,818, which describes methods for detecting and removing this
unwanted effect.
Possible algorithms for mitigating flicker include automatic exposure control,
automatic gain
control, and anti-banding. These techniques are common in many mobile devices
as a means to
remove flicker caused by fluorescent lamps.
Advanced DPR Demodulation Techniques
101971 DPR demodulation, instead of removing flicker, exploits the rolling
shutter effects of
CMOS cameras as a means of transmitting data. A CMOS device with a rolling
shutter captures
an image frame by sequentially capturing portions of the frame on a rolling,
or time-separated,
basis. These portions may be vertical or horizontal lines or "stripes" of the
image that are
captured at successive time intervals. Because not every stripe is captured in
the same time
interval, the light sources illuminating the image may be in different states
at each of these time
intervals. Accordingly, a light source may produce stripes in a captured frame
if it is illuminated
in some time intervals and not illuminated in other time intervals. Light
sources that broadcast
digital pulse recognition signals may produce patterns of stripes. Since the
pattern of stripes is
dependent on the frequency of the digital pulse recognition signal, and the
speed of the rolling
shutter can be determined a-priori, image processing techniques can be used to
deduce the
illumination frequency based on the width of the stripes. For example,
consider a room
containing five light sources 101, each broadcasting at 500 Hz, 600 Hz, 700
Hz, 800 Hz, and 900
Hz, respectively. Each distinct frequency, otherwise known as a DPR tone, can
be used to
identify the light source 101. In a beacon based light positioning system, a
mobile device
receiver within view of the transmitting lights can detect the DPR tones,
correlate an identifier
associated with the tone, and then use a lookup table to determine the
location of the device
based on the location associated with the identifier(s).
- 40 -
CA 02892923 2016-12-07
[0198] Modeling the camera sampling function is essential to understanding how
DPR
demodulation works on modern image sensors, and how the impacts of various
hardware-
dependent parameters affect the DPR signal 2105. To represent this, FIG. 25 is
a continuous time
representation 2501 of how an individual row on a rolling shutter image sensor
is sampled. The
exposure time interval 2502 represents the period over which light accumulates
on the photo
sensor. If the exposure time is much lower than the period of the DPR
modulated signal, the light
and dark bands will be clearly defined. If the exposure time is longer, the
light and dark bands
will lose their definition.
[0199] FIG. 26 contains a continuous time example 2601 of a DPR modulated
light signal. In
this example, the signal is a square wave with a 50% duty cycle being driven
at a
- 40a -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
DPR tone of 300Hz. The relationship between the DPR illumination period 2602
and the
exposure time 2502 determines how well defined the bands are on the received
image.
102001 FIG. 27 is the continuous time sampled image 2701, created by
convolving an
individual row sampling function 2501 with a DPR modulated signal 2601. The
alternating
periods of high brightness 2702 and low brightness 2803 are caused by the DPR
modulation
frequency, and appear as alternatin.g white/dark bands on the received image.
102011 FIG. 28 is a representation of a discrete time domain signal model
2801 for
representing how a rolling shutter on an image sensor samples the incoming
light pulses
2601. The rolling shutter is modeled as an. impulse train, containing a
sequence of the Dirac
Delta functions (otherwise knovvn as a Dirac comb). Each impulse is separated
by an
interval, T, which corresponds to the speed of the rolling shutter commonly
found in most
CMOS image sensors. The interval T varies from device to device which causes
the bands
on scenes illuminated by DPR. modulated signals to vary in size. The mobil.e
device 103
needs to account for hardware dependent factors (rolling shutter speed) to
properly determine
the DPR tone. FIG. 29 contains a discrete time representation. 2901 of the
rolling shutter
sampling functionality over multiple frames.
102021 Because rolling shutter speeds are typically faster than frame
rates, DPR
demodulation on current imaging technology is capable of much higher data
rates than
modulation schemes that sample on a per frame basis. In a DPR modulated system
using a
640x480 pixel image sensor, the sensor would capture 480 samples per frame
(represented as
480 consecutive delta functions in sensor model 2801). A demodulation scheme
using a
global shutter would only be capable of taking one sample per frame. This is a
key advantage
for indoor positioning using beacon based broadcasting schemes because the
time-to-first-fix
is orders of magnitude faster than competing technology, which can take
several seconds to
receive a signal. For example, consider a typical mobile device 103 camera
which samples at
30 frames per second (FPS). Using DPR demodulation, time-to-first-fix can be
achieved
with as little as a single frame, or 1/30 of a second, versus 1 second for a
demodulation
scheme that samples on a per frame basis. This compares to a time-to-first-fix
of up to 65
seconds for UPS, 30 seconds for assisted UPS, and 5-10 seconds for WiFi
positioning.
102031 This order of magnitude improvement opens the door for applications
in which
latency for time-to-first-fix must be minimized. Furthermore, computation for
DPR
demodulation can be performed on the mobile device itself, versus the server
side processing
required for WiFi fingerprinting algorithms. In a mobile environment, where
connection to a
network is not guaranteed, client side processing provides a major advantage.
In the future, it
-41-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
is expected that image sensors will have much higher frame rates. In this
scenario, DPR
demodulation can be adjusted to sample on a per-frame basis, instead of a
rolling shutter
basis. The key principle is that the demodulator can be adjusted in software,
allowing future
mobile devices to tune their receiving characteristics to receive DPR signals.
The software
adjustments that need to be applied are the subject of the following sections.
Configuring a Device for DPR Demodulation
102041 In order to prepare a mobile device 103 to receive the modulated DPR
signals
2105, the device must first be configured. This is to counteract the flicker
mitigation
algorithms typically applied in mobile device image sensors. FIG. 30 describes
the method
by which mobile device 103 is configured to receive DPR modulated signals.
First, the
initialize sensors 3001 function initializes and activates the available
sensors capable of
receiving data. For typical modern mobile devices these would include both the
front and
rear facing cameras. Determine sensors to modify 3002 then decides which
sensors need to
be modified. A number of possible factors determine whether or not a
particular sensor
should be initialized then modified, including power consumption, accuracy,
time since last
reading, environmental conditions, required location accuracy, and battery
state.
102051 Modify sensors 3003 then passes a list of the appropriate sensors
which need to be
modified to a function which has additional information about the mobile
device 103 and
adjusts the demodulation scheme for device specific limitations 3004. In the
case of using an
embedded mobile device 103 camera to demodulate DPR signals, possible sensor
parameters
to modify include exposure, focus, saturation, white balance, zoom, contrast,
brightness, gain,
sharpness, ISO, resolution, image quality, scene selection, and metering mode.
As part of the
modification step 3003, sensor parameters such as exposure, white-balance, and
focus are
locked to prevent further adjustments.
102061 After the sensors are modified 3003, specific hardware limitations
are adjusted for
in the demodulation scheme by using a device profile. The most important of
these is the
rolling shutter speed. Because different models of mobile device 103 will, in
general, have
different camera sensors, the line width of the DPR tone measure on an image
sensor will
vary across hardware platforms for a fixed frequency. For this reason, it is
necessary to
adjust the stripe width one is looking for depending on the specific
characteristics of the
device. In the Fourier Techniques discussed later on in the application,
modifying the stripe
width corresponds to modifying the sampling frequency of Dirac Comb 2801.
-42 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
[02071 There are a number of challenges associated with controlling the
camera
parameters to optimize for DPR demodulation. One challenge is overriding the
automatic
parameter adjustments that mobile operating systems typically provide as part
of their camera
application programming interfaces (APIs). In the case of an embedded image
sensor, the
sensor settings are adjusted automatically depending on factors such as but
not limited to
ambient light conditions, areas of focus, distance from objects, and
predetermined scene
selection modes. For instance, when taking a picture with an image sensor, if
the scene is
dark then the exposure time is automatically increased. When taking picture of
a scene mode
with fast moving objects, the exposure time is usually decreased.
102081 When using an image sensor for DPR demodulation, these automatic
adjustments
can introduce noise into the signal, causing higher error rates. Specifically
in the case of
exposure, longer exposure times correspond to lower data rates, which
correspond to a
decreased amount of available light IDs 901. At the edge case, if the exposure
time is
sufficiently long, then the sampling rate will drop so low that DPR
demodulation becomes
extremely challenging as the signal is severely under sampled. Furthermore, if
the camera is
constantly adjusting, then the performance of background subtraction
(discussed later), which
isolates the moving stripes from the rest of the picture, will be
significantly impaired. This is
because the automatic adjustments are constantly changing the pixel values. In
order to
successfully transmit DPR signals, these automatic adjustments need to be
accounted for.
102091 Practically speaking, many mobile device 103 APIs do not allow for
the
modification of sensor parameters in the top level software. The proposed
method in FIG. 31
describes a method for working around the provided APIs to control the
exposure. Current
API's do not allow for manual exposure control, so instead of manually setting
the exposure,
we present an algorithm that exploits the metering functionality to minimize
the exposure
time.
102101 FIG. 31 contains a process for modifying the various sensor
parameters contained
in a mobile device 103 in a way that overcomes the limitations imposed by
curren.t camera
APIs. In the algorithm, the first step is to initialize the required sensors
3001. For the case of
an image sensor, this involves settin.g the frame rate, data format, encoding
scheme, and color
space for the required sensors. After the image sensors have been initialized
3001, the
algorithm searches for regions of interest 3101. In the case of setting the
exposure using
metering, these regions of interest 3101 would be the brightest regions of the
image. Set
metering area 3102 then sets the metering area to the brightest portion,
effectively "tricking"
the mobile device 103 into lowering the exposure time. Lock parameter 3103
then locks this
-43 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
exposure time to prevent the auto adjustment feature of the camera from
overriding the
manual setting. Next, adjust for hardware dependent parameters 3104 accesses a
lookup
table and adjusts the demodulation algorithm based on hardware and software
differences.
For the case of an image sensor, one example of this is changing the sampling
time based on
the rolling shutter speed of the device. This rolling shutter speed can either
be loaded from a
lookup table beforehand (using predetermined values) or measured on the fly.
Each device
only needs to measure its rolling shutter speed once per image sensor. Once
parameters set?
3105 is satisfied the algorithm ends; otherwise, it returns to identify
regions of interest 3101.
102111 The method of exploiting the metering area on a mobile device 103
can be used to
optimize many of the required parameters in addition to the exposure,
including white
balance, contrast, saturation, ISO, gain, zoom, contrast, brightness,
sharpness, resolution,
image quality, and scene selection. Furthermore, these parameters could
already be known
beforehand, as each mobile device 103 will have its own "device profile"
containing the
optimal camera settings. This profile could be loaded client side on the
device, or sent over a
server. Note that although the method of using the metering area to control
the exposure can
improve the performance of DPR demodulation, it is not strictly necessary.
Simply locking
the exposure 3103 is often sufficient to prevent the automatic camera
adjustments from
filtering out the DPR signals.
Advanced Techniques for Decoding Information in DPR Modulated Signals
102121 Once the sensors have been initialized 3001 and parameters have been
set 3104,
FIG. 32 describes a process for decoding the information contained inside a
DPR modulated
signal. Identify regions 3201 is used to separate different regions on the
image illuminated
by DPR signals. At the base level, the region of interest is the entire image.
However, when
one or more light sources 101 are present, there exists an opportunity to
receive multiple DPR
signals simultaneously. In this scenario, the sensor effectively acts as a
multiple antenna
receiver. Such multiple antenna systems, more generally referred to as
multiple-input
multiple-output (MIMO), are widely used in the wireless networking space. This
is an
example of spatial multiplexing, where wireless channels are allocated in
space as opposed to
time or frequency. The implications of MI:MO for D:PR demodulation in a beacon
based light
positioning system is that frequencies can be re-used in a space without worry
of interference.
When a mobile phone user receives DPR. modulated signals on a photodiode array
(such as
an image sensor, or any imaging technology that contains multiple spatially
separated
sensors), the DPR signals will each appear at different locations on the
sensor. Each region
-44-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
3201 of the image can then be processed independently, in the same way that
each mobile
phone user in a cell network only connects to the cell they are closest to.
102131 This works in a way analogous to cellular phone networks. With
cellular
networks, mobile phone users only communicate with cellular towers that are
close to them.
This allows multiple mobile phone users to share the same frequency, provided
they are all
on different cells. In DPR modulation, each light acts as its own cell
transmitting unique
frequencies. However, different lights can also use the same frequency
provided that they are
far enough apart. Re-using the same frequencies in different space allows for
greater system
scalability, since lighting sources 101 can be installed at random without
requiring the
installer to worry about frequency allocation.
102141 After sensors have been initialized 3001, and regions of interest
3201 have been
identified, detect frequency content 3202 identifies the presence of DPR tones
from the
sensor data. We describe here multiple methods for extracting the frequency
content from a
DPR signal. One possibility is to use line detection algorithms to identify
the pixel width of
the stripes, which directly corresponds to the transmitted frequency. This
stripe width is then
used to access a lookup table that associates width and transmitted frequency
and determines
the transmitted tones. Possible methods for detecting lines include Canny edge
detection,
Hough Transforms, Sobel operators, differentials, Prewitt operators, and
Roberts Cross
detectors, all of which are well developed algorithms, known to those of skill
in the art.
Adjust for dependent parameters 3004 then modifies the appropriate camera
sensors for
optimal DPR demodulation. In the case of line detection, this corresponds to a
linear
adjustment for the line width lookup table. Determine tones 3203 uses the
adjusted line
width to determine the DPR tone sent. This process is performed for each
region on the
image, until there are no more regions 3204 remaining. A data structure
containing all the
regions, with their associated identifiers, is then returned 3205.
102151 An additional method for performing DPR demodulation is described in
FIG. 33.
One or more light sources 101 illuminates a scene 3301. When the image sensor
on mobile
device 103 acquires a sequence of images 3302, the brightness of any given
pixel depends on
both the details of the scene as well as the illumination. In this context,
"scene" refers to the
area within view of the camera. The scene dependence means that pixels in the
same row of
the image will not all have the same brightness, and the relative brightness
of different image
rows is not solely dependent on the modulated illumination 3301. If one were
to take the
Fourier transform of such an image, both the frequency content of the
illumination, as well as
the frequency content of the underlying scene, will be present.
-45 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
102161 In order to recover the frequency content of the modulated
illumination
independently of the scene, the contribution of the scene may be removed using
a background
subtraction algorithm 3303. The 'background' is the image that would result
from un-
modulated illumination as opposed to the effects of modulated illumination
3301. Subtracting
the background from an image leaves only the effects of illumination
modulation. One
possible implementation of a background subtraction method uses a video
sequence. If a
video of a scene illuminated with modulated light is recorded, the light and
dark bands may
appear at different locations in each frame. For any modulation frequency that
is not an exact
multiple of the video frame rate, there will be a resulting beat frequency
between the video
frame frequency and the illumination modulation frequency. The illumination
signal will be
in a different part of its period at the beginning of each frame, and the
light and dark bands
will appear to be shifted between video frames (i.e. the bands will appear to
move up or down
across the scene while the video is played). Although this algorithm is
described with the use
of a video sequence, other embodiments may perform background subtraction
using still
images.
102171 Because the bands move between video frames, the average effect of
the bands on
any individual pixel value will be the same (assuming that in a long enough
video each pixel
is equally likely to be in a light or dark band in any given frame). If all
the video frames are
averaged, the effects of the bands (due to the illumination modulation) will
be reduced to a
constant value applied to each pixel location. If the video is of a motionless
scene, this
means that averaging the video frames will remove the effect of the bands and
reveal only the
underlying scene (plus a constant value due to the averaged bands). This
underlying scene
(the background) may be subtracted from each frame of the video to remove the
effects of the
scene and leave only the effects of illumination modulation 3301.
102181 FIG. 34 contains an implementation of a possible background
subtraction
algorithm 3304. A frame buffer 3402 accumulates video frames 3401. The size of
this buffer
can vary, depending on the memory capacity of mobile device 103 and the
required time to
first fix. Frame averaging 3403 computes the average based on the frames in
the buffer 3402.
The average of these frames is used to generate background frame 2704. The
background
frame can be acquired using a number of diffirent averaging techniques 3403,
including a
simple numerical average, a normalized average (where each frame is divided by
the sum of
all the frames), Gaussian averaging, or by doing a frame difference between
subsequent
frames. A frame difference simply subtracts subsequent frames from one another
on a pixel-
by-pixel basis.
-46-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
102191 For video of a scene with motion, simple averaging of video frames
will not yield
the underlying scene background. FIG. 35 describes a technique for dealing
with motion
between frames, which is a likely scenario when demodulating DPR signals on
mobile device
103. Motion compensation. 3501 is necessary to best determine the underlying
scene. By
determining the motion between video frames (for example, shifting or rotation
of the whole
scene due to camera movement), each video frame may be shifted or transformed
such that it
overlies the previous frame as much as possible. After performing these
compensatory
transforms on each frame in motion compensation 3501, the video frames are
averaged 3403
to get the scene background 3404. Phase correlation is one possible method of
estimating
global (i.e. the whole scene moves in the same way, as in the case of camera
motion while
recording video) translational motion between frames. The 2D Fourier transform
of a shifted
image will be the same as that of the original image, except that a phase
shift will be
introduced at each point. Normalizing the magnitude of the 2D Fourier
transform. and taking
the inverse transform yields a 2D image with a peak offset from the center of
the image. The
offset of this peak is the same as the shift of the shifted image. Those
skilled in the art will
recognize that additional methods for motion compensation 3501 include Kernel
Density
Estimators, Mean-shift based estimation, and Eigenbackgrounds.
102201 After removing the background scene, Fourier Analysis can be used to
recover the
DPR tone based on signals received from modulated light source 103. Specifics
of this
method are further described in FIG. 36-43. FIG. 36 contains a sample image
3601 of a
surface illuminated by a light source undergoing DPR modulation. The image is
being
recorded from a mobile device using a rolling shutter CMOS camera. The stripes
3602 on
the image are caused by the rolling shutter sampling function, which is
modeled in by the
sequence of Dirac Combs 2801 in FIG. 28.
102211 FIG. 37 shows the result 3701 of performing background subtraction
on the raw
image data from FIG. 36. Background subtraction is used to extract the stripes
from the raw
image data. The result is an image of alternating black/white stripes that
represents the
discrete time-domain representation of the transmitted DPR signal. The stripes
3702 are
much more pronounced than in the raw image data from FIG. 36 due to the
improvement
from background subtraction.
102221 Illumination modulation affects each row of a video frame
identically, but
imperfect background subtraction may lead to non-identical pixel values across
image rows.
Taking the Fourier transform of row values along different image columns,
then, may
produce different illumination signal frequency content results. Because the
true illumination
-47 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
signal frequency content is the same for the entire image, a technique to
reconcile these
different results may be employed. One possible method is to assign the
average pixel value
for any given row to each pixel in that row. This method takes into account
the information
from each pixel in the row, but by yielding uniform row values gives a single
illumination
signal frequency content result when taking the Fourier transform of row
values along an
image column. FIG. 38 displays the results of applying row averaging 3801 to
the
background subtracted image 3701. The stripes 3802 are much more visible as a
result of the
row averaging, and they are also more consistent across rows.
102231 FIG. 39 shows the Fourier transform 3901 of the row averaged image
3801 from
FIG. 38. There is a peak frequency at the DPR tone of 700 Hz, as well as a DC
component at
0 Hz. The peak frequency is used to identify the sequence of tones, and thus
the transmitted
identifier.
102241 FIG. 40 shows the Fourier transform. 4001 from FIG. 39 after
applying a high-pass
filter. The DC component of the signal is removed, which allows a peak
frequency detector
to move to detection at the DPR. tone frequency.
102251 FIG. 41 shows a 2-D Fast Fourier Transform 4101 of the post
processed DPR
modulated signal data 3701. In comparison to the 1-D Fourier analysis
performed in FIGS.
38-40, 2-D Fourier analysis of the DPR modulated signal 3601 could also be
performed. 2-D
Fourier Analysis is a popular and widely used technique for image analysis.
Because there
are a number of software libraries that are highly optimized for performing
multidimensional
FFTs, including OpenCV, multidimensional Fourier analysis is a viable
alternative to the 1-D
analysis. The DPR tones 4102 can be easily seen across the vertical axis 4103
of the 2-D
FFT. Brighter areas on the FFT image 4101 correspond to areas on the image
with higher
spectral content. A peak can be seen at the origin 4104, which corresponds to
the DC
component of the DPR signal.
102261 FIG. 42 shows a low-pass filtered version 4201 of the 2-D FFT 4101.
The filtered
image 4201 contains dark areas 3502 at the higher frequencies on the image.
The low pass
filter rejects the higher frequencies. This is a key component of successful
DPR
demodulation. As discussed previously, DPR modulation relies on transmitting
digital
signals at different frequencies. When using Fourier analysis on these
signals, higher
frequency harmonics appear, in particular at higher duty cycles. These higher
frequency
components act as noise in the signal, so removing them with filtered image
4201 is one
technique for recovering the transmitted tones.
-48 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
102271 When performing spectral analysis in the case of a 1-D FFT 3901 in
FIG. 39, it
was necessary to remove the DC component of the DPR signal. PWM signals 1901-
1903 will
contain a significant DC component, which needs to be filtered before moving
on to extract
the transmitted DPR tone. FIG. 43 shows a high-pass filtered version 4301 of
the 2-D FFT
4101. The dark area 4302 at DC demonstrates the result of the high-pass
filter, which rejects
the DC noise component. The higher frequency bands 4303 are still contained in
the signal,
allowing the demodulator to determine the peak frequency.
102281 An important consideration in the design of a beacon based light
positioning
system is the choice of light identification codes with respect to the system
layout. In a
system which uses individual digital pulse recognition (DPR) tones to identify
the positions
of light sources, it is desirable to lay out the light sources such that
adjacent sources have
frequencies that are spaced far apart. For example, consider FIG. 1, which
contains light
sources 10 la-d, each with its own DPR tone. When a mobile device user 102
moves
underneath beacon based light sources 101a-b, each of which is emitting a DPR
tone, there is
the possibility that the user's mobile device 103 will receive multiple light
signals
simultaneously. In such a situation, if the spacing between the tones is too
low (for example,
in the case of light source 101a emitting a DPR tone of 700 Hz, and 101h
emitting a tone of
705Hz), then spectral leakage may occur. Spectral leakage is a well-known
phenomenon
associated with Fourier analysis. It is a consequence of the finite
observation time over
which a signal is measured. For DPR demodulation that exploits the rolling
shutter, this
finite time corresponds to the period of the rolling shutter. Spectral leakage
negatively
impacts the ability to distinguish two or more frequencies, so in the case of
user 102
receiving multiple DPR tones, the ability to distinguish tones will be
diminished. In some
cases, the tones will be unresolvable, which could possibly cause dead spots
in the beacon
based positioning system presented in FIG. 2.
102291 The impact of spectral leakage can be mitigated in a number of ways.
One
technique is the use of a digital filter, which is sometimes referred to as a
window function.
Popular choices for window functions include Rectangular, Hann, Hamming,
Turkey, Cosine,
Kaiser, and Gaussian windows. Using a window function allows one to improve
the spectral
resolution of the frequency-domain result when performing Fourier analysis.
102301 A simple approach to reduce the impact of spectral leakage is to
control the spatial
distribution of bulbs. For example, one could come up with a rule which says
that no two
adjacent bulbs can have a DPR tone within 50 Hz of another adjacent bulb. For
most lighting
topologies 4403, mobile device users 4402a-b will see at most two adjacent
lights at the same
-49-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
time. If the DPR tones are distributed such that adjacent bulbs are dissimilar
enough, then the
individual DPR tones can be resolved.
102311 Furthermore, in cases in which the DPR signal is more complex, and
possibly
composed of several tones transmitted simultaneously, adjacent light sources
could
destructively interfere with each other. For example, consider that the two
light sources
4401a and 440 lb in FIG. 44 each transmit multiple DPR tones. Light source
4401a emits
300 Hz, 400 Hz, and 500 Hz, while light source 440 lb emits 500 Hz, 600 Hz,
and 700 Hz.
Note that both sources share a common tone of 500 Hz. Since the bulbs are
typically non-
networked, their emissions are not synchronized. Accordingly, if the common
500 Hz tones
are out of phase with one another, they will destructively interfere, possibly
resulting in a
missed detection.
102321 FIG. 47 depicts a user commissioning a beacon based light
positioning system.
Commissioning refers to the process of acquiring of acquiring DPR signals
being emitted
from light sources 4401a-4401b, and then georeferencing the light sources.
Mobile device
user 4702a invokes an application running on their mobile device 4703. The
application
listens for the DPR signals using sensors on the device. When the mobile
device detects a
valid DPR signal, the user is prompted to georeference the light source. This
could be done
by simply entering in coordinates (which could take the form of latitude,
longitude, and
altitude), or some arbitrary coordinate system. The user could also assign the
coordinates by
dragging arid dropping the light location on a map. Once the coordinates are
assigned, they
can be sent to a remote database 802 for storage. The database contains
records which
associate the light identifier with the light location. A user interface
developed for this task is
presented in FIG. 48.
102331 After a user georeferences light source 4401a, they proceed in a
path 4704a
underneath additional light sources 4401a-c. For each light source, as they
successfully
detect the source, they are prompted to georeference the light. If the user
attempts to add a
light source that violates the rules for the lighting topology (as in the
previous example,
where adjacent lights are not allowed to contain DPR tones less than 50 Hz
apart), the user
would be prompted to change the location of the light source before adding it
to the map.
102341 FIG. 47 also contains a situation by which multiple users 470 la-b
can commission
a space simultaneously. Note that the number of users could be much larger
than two,
especially in the case of a large space. Having multiple users operate at the
same time greatly
speeds up the commissioning process. As users 4701a and 470 lb move about
their
respective paths 4704a and 4704b, they georeference the light locations in the
same manner
- 50 -
CA 02992923 2015-05-28
WO 2014/063150 PCT/US2013/065923
that an individual user would. Each time the user adds a light, the light
could be sent to a
remote server 703 for storage. The connection to the server could be done
through any
standard network connection. Mobile devices 4702a-b would each write their
respective
portions of the georeferen.ced light positions to the server.
102351 The user workflow for multiple users working in conjunction to
commission a
space could come in a variety of forms. In one embodiment, each mobile device
user 4702a-
4702b would maintain a synchronous representation of the current light
georeferencing. In
this scenario, when mobile device user 4702a adds a light to the database,
mobile device user
4702b would see that light location appear on their device. This helps to
avoid the need for
the users to maintain close communication during commissioning, thereby
speeding the
process along. In another embodiment, mobile device users 4702a-b would each
maintain
their own separate records of the light locations. When the remote server 703
received the
commissioning data from each user, it would then take all of the data as input
and determine
the best result. For example, in one embodiment the server would take a simple
numerical
average of the coordinates for each light. By combining data from many users
when
commissioning a space, the accuracy of the system is improved.
102361 Note that in this embodiment the users are depicted as the ones
commissioning the
space. However, in another embodiment this commissioning could be performed
automatically by a non-human entity, or robotic agent that patrols the space.
The robot can
use forms of inertial, navigation techniques and sensor fusion to maintain an
estimate of its
current position. The robot must first be assigned an initial location fix,
which could be done
through a manual input, or a location fix acquired from. an alternative
positioning technique
such as WiFi, Bluetooth, GPS, A-GPS, Ultrasound, Infared, NFC, RF-ID, a priori
known
location, or markers in the visible or non-visible spectrum. When the robot
receives a light
identifier from light source, it then uploads the identifier along with its
estimated position to
the light location database. Furthermore, the robot can also send an estimate
of the strength
of the current signal, along with an error estimate of its current position,
as well as any
additional relevant information. The server can then combine all of this
information to get a
more accurate estimate of the light position. This is analogous to the way the
server
combined information from multiple users in FIG. 47.
102371 FIG. 48 contains one possible user interface for commissioning a
space in a light
positioning system. As depicted in FIG. 47, mobile device users 4702a-b
commission a
space using mobile devices 4703a-b. The mobile devices 4703a-b presents
location
information, such as a map 4806, of the space. When a user 4702a acquires a
signal emitted
-51-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
from a light source 4401a, a user notification, such as an add button 4801,
becomes active on
the user interface. The user can then add the light onto the map by "dragging
and dropping,"
using either a point and click gesture with a mouse, keyboard based commands,
a
touchscreen, or other such user interfaces that are available. After a light
has been placed on
the map 4806, an icon 4804, or identifier, appears to mark its location.
Additional user
feedback, such as making the device vibrate or change color, can be used to
make the
experience more tactile. The user can use the move button 4803 to move the
location of the
light in the space. In addition to manual location edits, additional
information such as an
architectural, electrical, or mechanical building plan can be used to assist
in light placement
or to automate fine tuning of the process. Furthermore, the user can edit and
delete lights
from the map using the Edit button 4802. Once a light is placed on the space,
and the marker
4804 has been assigned a location, the marker may change its visual look 4805
when the user
walks underneath the appropriate light and successfully receives the DPR.
signal. The visual
feedback could include changing the color, shape, or moving the marker around
on the
screen. This makes it easier for the user to verify the position and
identification code of the
light after they have commissioned the space.
102381 One aspect of DPR demodulation in a light positioning system
includes adjusting
receiver parameters depending on specific characteristics of mobile device
4901. As
discussed previously in the applications cited above, parameters such as
rolling shutter speed,
frame rate, and exposure time can all vary across different hardware
platforms. In order to
compensate for these differences, adjustments can be created in software. One
method of
dealing with this involves creating and maintaining a database of device
profiles, as depicted
in FIG. 49. A device profile can include all the relevant information
regarding the receiver
parameters of a mobile device 4901. For example, this device profile could
include the
exposure time, sampling frequency, number of image sensors, or other hardware
dependent
parameters for a specific version of mobile device 4901. When the mobile
device performs
DPR demodulation as part of a light positioning system, the device profile is
first loaded.
The profile could either be fetched from a remote server, or stored locally on
the device. The
parameters contained within the device profile are then fed as inputs into the
demodulation
algorithm.
102391 Device profiles can either be created in a number of different ways.
In one
embodiment, the device profile could be created via manual input of values. A
system
administrator could acquire device information using sensor datasheets, or
other sources, and
then manually input the parameters into the database. In another embodiment,
the mobile
- 52 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
device can utilize a self-calibration algoiithrn to figure out its own device
profile. For
example, consider a mobile device, with an unknown. device profile that is
positioned within
view of a DPR modulated light source emitting a known tone. The mobile device
could enter
as an input the tone which it is supposed to be seeing, and then adjust its
own device
parameters such that the output of the detection algorithm is equal to the
tone that is
"known."
[0240] Self-calibration is another technique for resolving differences
between device
hardware and software. In this method, a special calibration frequency can be
transmitted at
the beginning of a DPR packet, allowing the mobile device 103 to measure its
own sensor
parameters, such as the shutter speed, and adjust accordingly. The downside to
this approach
is increased read time on the 'receiver, because the calibration frame carries
no positioning
information.
Alternative Algorithms for DPR Demodulation
10241] In addition to the DPR demodulation algorithms presented earlier,
there are a
number of alternative techniques for demodulating :DPR. signals. These
algorithms are
presented in the sections that follow, and include both Fourier and non-
Fourier based
methods. It should be understood that these algorithms illustrate exemplary
DPR
demodulation algorithms, and similar equations are also within the scope of
this disclosure.
10242] When captured by a rolling shutter image sensor, a DPR modulated
sigial
produces periodic horizontal bands that overlay the entire 21) content of a
given scene. A.n
M x N frame (M columns, N rows) can be viewed as a set of M ID signals y0M1 of
length N. Since each column is subject to the same modulation signal, the
phase of
frequency components Yo,...,,[kõ] , where kõ is the vertical spatial frequency
of the
periodic horizontal bands, is expected to be identical across all columns
i.e.
Var 0. in practice, due to noise factors, phase variance exhibits
the
following behavior in absence of periodic scene patterns:
Var(ZY .......... ILO ¨> 0, k=kBL
(1)
Var Yo, .. ,m i[k]) 0, k
where k e [k1o,Ichigh] and k and khv denote the lowest and highest possible
vertical
frequencies of the periodic horizontal bands, respectively. Consequently, when
kõ is
- 53 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
unknown, it can be identified by determining which frequency component has
minimal phase
variance across all columns y0 , i.e.
kBL arg min (Var(ZY0,...,õ ,[k]))
(2)
Phase Parameterization
102431 When measuring variations in phase, it is important to preserve
phase circularity.
This is achieved by parameterizing real-valued phase values ZY,,...,õ ,[k]
using unit
magnitude complex numbers
PO,...,M 1 ¨ exp( j ,[k])
(3)
where j is the imaginary unit. Subsequently, the variance in parameterized
phase is given by
Var(pAl1) ¨ E i])' D
(4)
where E[X] denotes the expected value of X, and X denotes the complex
conjugate of X.
The spatial frequency k õ of the periodic horizontal bands is then given by
k = arg min (Var
ke[k,,,k,igh]
(5)
102441 FIG. 50 shows a plot of variance in parameterized phase vs. possible
modulation
frequencies, in the presence of a 755 Hz modulating source. Note that the dip
5002 at
modulation frequency 755 Hz indicates the signature of the DPR modulated
signal.
Fourier Magnitude Peaks
102451 Because periodic horizontal hands produced by the modulation signal
extend
throughout the whole frame, 2D Discrete Fourier Transform (DFT) of the bands
yields two
compact (narrow spectral support) peaks in DFT magnitude along the vertical
frequency axis.
This fact is used in conjunction with phase consistency to increase detection
accuracy and
eliminate periodic scene patterns,
- 54 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
102461 In order to filter out frequency components with compact magnitude
peaks, the
magnitude spectrum in FIG. 51 is convolved with a 3 x3 filter kernel with the
general
structure presented in FIG. 52.
102471 The motivation for this choice of filter coefficients is to remove
'magnitude peaks
of wide spectral support and replace them by negative-valued dips. On the
other hand,
'magnitude peaks of narrow spectral support remain positive-valued. FIG. 53
shows the result
of applying the above filter kernel to vertical frequency components of the
magnitude
spectrum in FIG. 51.
Detection Algorithm
102481 In what follows, f [m, n] denotes an M x N frame (M columns, N
rows).
i[ni,n] denotes a vertically-windowed version of f[nt,n] .
1) Windowing
102491 Obtain l[m,n] by multiplying each column y0 of f[m,
n] by an N -length
Hann window.
2) 1D DFTs
[0250] Compute N -length DFT for each column of the windowed frame 'hin,n]
to
obtain M ID spectra fo÷..,,
Phase parameterization
102511 Parameterize the phase of Y0 with accordance to Eq. 3 to obtain
^ k
, = exp( j , [k])
3) Phase consistency
102521 Compute the variance of across all columns to obtain Var( ,)
with
accordance to Eq. 4. This provides a phase consistency measure for every
vertical spatial
frequency k in the range k c .
- 55 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
4) Choose candidate banding frequency based on phase consistency
102531 Determine the vertical spatial frequency with lowest variance Var()
in the
frequency range
= arg min (Var ("kbok. 1))
(6)
5) Dip quality
Var (13 ok P )
102541 Define left-handed and right-handed dip quality measures as q, =
Var (pon,õ
Var(kµok:F..;
and q, = respectively.
Var(A,A7 õ
102551 Define dip quality measure as
qd4,=(q,+q,)1 2
(7)
6) 2D DFT of original frame
102561 Compute the M x N DFT of the original (non-windowed) frame f[m,n] to
Obtain F[kh, , Ensure that
the DET spectrum is centered around the de component.
7) Magnitude filtering
102571 Apply magnitude filtering, as described in Sec. 3, to vertical
frequency
components. The result is a ID filtered magnitude sequence FTO, kõ11.
8) Choose candidate banding frequency based on filtered magnitude
102581 Determine the vertical spatial frequency with highest filtered
magnitude value in
the frequency range k, E[k,0õ, khigh]:
= arg max OFTO,
(8)
9) Rejection criteria
102591 If at least one of the following conditions is true, the frame under
question is
rejected and no "hit" is registered:
- 56 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
a. qdip <1.5 (low dip quality)
b. IFIO, < 0
(negative-valued filtered magnitude)
c. ¨k > 2 (significant disagreement between
phase-based and magnitude-based
candidate frequencies)
Note that the above threshold values were experimentally determined to provide
good
performance.
102601 If none of the above conditions hold, define the low-precision
estimate of the
banding frequency as kBLl = k and proceed to Steps 11-12 or Steps 13-14.
10)Interpolated ID DFTs (Zero-padding)
102611 Columns of the windowed frame :f[m,n] are zero-padded to length
N'> N, yielding M N' -length columns 5'....M-1 . Choice of N' depends on the
desired
level of precision in banding frequency detection. Apply Step 2 to 5 1 to
compute M
N' -length spectra
11) Phase consistency in the neighborhood of low-precision estimate
k,õ ¨ 4
102621 Let k", = round N' __ and k'b = round r k + 4
BL, ___________________________________________ 1 . Apply
Steps 3-5 to
N
f70' m 1[11 specifically for the frequency range Ica' k' .
102631 Define the high-precision estimate of the banding frequency as
kBL, h = arg min (Var i))
k:E[K,14] (9)
where po.'õ,= exp (j = I [Al) .
12) Interpolated 1D DFT (zero-padding) of row-averaged intensity
102641 Compute the average intensity of each row in the windowed frame
j[m,n] to
obtain .pays Zero-pad j) to length N'> N yielding an N' -length column _Pay:.
Choice of
- 57 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
/V' depends on the desired level of precision in banding frequency detection.
Compute the
N' -length DFT of ji'avg1 to obtain
13) Peak magnitude in the neighborhood of low-precision estimate
k ¨4 ( k +4
10265] Let ka' = round N' __ and k' = round N' __ Define the high-
precision estimate of the banding frequency as
kBLh arg max (
k'G[k,kL] (10)
14) Convert banding frequency to DPR modulation frequency
[02661 Let A denote the row sampling frequency of the rolling shutter image
sensor.
The DPP. modulation frequency 13 corresponding to kBL is then given, by
kBL h
fl= (11)
ps can be determined experimentally by calibrating detection results obtained
for a known
modulation signal.
Camera Switching
10267] When utilizing mobile device receiver 103 to receive DPR modulated
signals as
part of a light positioning system, one aspect of the process is to identify
which sensors to
use. Because many mobile devices 103 contain multiple sensors (for example,
consider
sm.artphones that contain both front and rear cameras), some embodiments
include an
algorithm to decide which camera sensor to use at a particular point in time.
For example,
consider the situation in FIG. 47, where a user 4702a is walking underneath
light source
440 la, emitting a DPR modulated spot signal onto the floor 4701. The user's
mobile device
4703a contains both a front and rear camera. As the user walks past the DPR
illuminated
spot 4701, the rear camera is the first sensor to capture the image. When the
user walks
further along, the front facing camera is positioned underneath the light. In
this scenario,
some embodiments use an algorithm to decide which camera to use when receiving
the light
signals.
102681 FIG. 35 contains an implementation. of a smart camera switching
algorithm.
Select sensors 5501 first decides which sensors to use. The decision for what
sensor to use
- 58 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
depends on a number of factors. In one embodiment, the mobile device could use
information about its orientation, using sensor readings gathered from the
gyroscope,
compass, accelerometer, or computer vision, and then use that to determine
which camera to
choose. For example, if the device were to recognize that it was oriented such
that the rear
camera was facing the ceiling, select sensors 5501 would first choose the rear
sensor to
receive the beacon based light positioning signal, as opposed to defaulting to
the front
camera. After selecting a sensor 5501, initialize sensors 3001 sets the
required hardware and
software sensor parameters as described in FIG. 30. Once the sensors have been
selected and
initialized, Tone (s) Detected? 5502 implements a DPR demodulation algorithm
to analyze
incoming image/video frames for the presence of DPR tones. An output of the
DPR
demodulation algorithms is a confidence score representing the likelihood that
a DPR tone is
present in the received images. If Tone(s) Detected? 5502 returns TRUE, then
the algorithm
terminates. If Tone (s) Detected? 5502 returns false, Continue Checking? 5503
decides
whether or not to try the remaining camera sensors. The decision is governed
by factors
including battery life, location refresh time, and accuracy requirements. If
Continue
Checking? 5503 continues the algorithm the cycle repeats, otherwise it is
terminated.
Hiding Camera Preview
102691 The primary functionality of most image sensors on mobile devices is
recording
multimedia content in the form of video or photographs. In those use cases, it
is desirable to
display a preview feed of what the camera is recording before actually
shooting the
video/photo. On most mobile device's 103, displaying this preview feed is the
default mode
when using the camera. On some mobile device API's, the software actually
requires the
application to display a preview feed such that if a feed is not being
displayed, the device is
not allowed to record data.
102701 For a mobile device 103 that is receiving DPR modulated light
signals, it is
desirable to hide the camera feed during DPR demodulation. For example,
consider the case
of a user walking around a retail store, using an in-store map to discover
where items are
within the store. If the mobile device application framework required the
camera preview to
be displayed, it would interrupt the user's interaction with the mobile app.
Furthermore,
since there are a number of different camera tweaks that can be applied during
DPR
demodulation (for example, modifying the exposure, focus, zoom, etc.), the
camera preview
image quality would be low.
- 59 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
102711 The present disclosure has explored a number of workarounds to the
restrictions
imposed by device .APIs around the camera preview requirements. In one
embodiment, the
mobile device creates a surface that is a single pixel in size, and then
passes this surface as
the camera preview. The surface is an area on the mobile device for displaying
information
to a user. This fulfills the requirement of the mobile device API that it is
presented a surface
to write to. Because the surface is only a pixel wide, the present system
effectively tricks the
API, since the mobile device user cannot see the individual pixels surface. In
another
embodiment, the mobile device simply does not create a camera preview surface.
Location Dependent Demodulation Algorithms
102721 One aspect of DPR demodulation algorithms is that the performance of
such
algorithms varies from location to location. For example, consider a situation
in which the
floor of a building contains stripes or other noise generating artifacts. The
presence of
artifacts on the floor adds significant noise into the DPR signal. As
discussed previously,
both the front and the rear camera can be used to recover DPR modulated
signals. However,
in a situation with the presence of noise on the rear camera, it would
undesirable to use the
rear camera at all. To account for situations such as this, the present
disclosure could use
location-dependent demodulation algorithms. Using algorithm performance
information
gathered from mobile device users 4402a-b, the remote server 703 maintains
records of
which algorithms perform the best in various situations. These algorithms also
can be tied to
specific device profiles, such that different mobile phones utilize different
location-dependent
algorithms. Furthermore, the location-dependent algorithms also take into
account which
camera is being used. Typically, the performance characteristics for
algorithms vary
depending on the image sensor type and the position on the mobile device. In
the case of
typical smartphones, it is desirable to use a different algorithm on the front
versus the rear.
For example, in one embodiment of the present disclosure the phase variance
algorithm has
better performance on the front camera versus the rear camera.
102731 A method for intelligently choosing the appropriate demodulation to
use in a
given scenario is presented in FIG. 54. First, an image sensor is selected and
sampled 5401.
This image sensor could be one of many sensors on the device (for example,
either the front
or rear sensor). An algorithm for actually selecting the sensor is presented
in FIG. 55. After
the sensor is selected, a demodulation algorithm is chosen 5402. There are a
variety of
factors that can go into selecting the appropriate demodulation algorithm. The
most
important parameters are which image sensor is currently being used, and which
location the
- 60 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
mobile device 103 is within. The mobile device 103 can send a request to a
remote server
703 to find out which algorithm is appropriate for its current environment.
Upon selecting an
algorithm, the algorithm is implemented and the quality of the signal is
reported 5403. The
quality of the signal is measured using a number of different metrics, as
described previously.
After the quality of the signal has been determined, the algorithm might be
adjusted 5404.
Adjustments to the algorithm could include changes to algorithms parameters
such as the
sampling rate, exposure time, or other such parameters as described
previously. The
adjustments also can include changing the actual choice of algorithm.
102741 After the algorithm has been adjusted 5404, Movement
Detected'En.vironment
Change? 5405 is used to determine whether or not the mobile device has changed
its location.
If the mobile device changes location, the image sensor is initialized and the
sampling
algorithm resamples at the beginning.
Interpolation
02751 When using a light positioning system, it is often the case that a
mobile device
receiver can receive signals from multiple lights simultaneously. Consider the
situation
presented in FIG. 57. Mobile device user 5701 is within view of light spots
5702a, 5702b,
and 5702c. Because the user 5701 is not positioned directly underneath any
particular light,
it is desirable to use the relative signal strengths for each light to
interpolate the position of
the mobile device receiver 5703 from the lights. A mobile device 5703 detects
quality
signals from two light sources 5702a and 5702b. For each quality signal, the
magnitude of
the signal is taken into account before resolving the exact location. A simple
algorithm is to
take the average of the light locations, each weighted by the total signal
strength. For a given
set of n magnitudes ma and light coordinates xa, the position of the mobile
device 5703 is x,
where:
E:=oxama
Ent
102761 Each coordinate is weighted against the total sum of signal
strengths as part of the
averaging process. The result is a set of coordinates that indicate the user's
position.
102771 In addition to weighting the total signal strength, the mobile
device 5703 also can
use information gathered from multiple image sensors, as well as information
about the
geometry of the room and the orientation of the device, to improve the
positioning
calculation. At this point, the device will have resolved a separate
coordinate for the rear
camera, and for the front camera. The device lies somewhere in between these
coordinates.
-61-
CA 02892923 2016-12-07
Using the geometry in Fig. 6 described U.S. Pat. App. No. 13/526,773 filed
June 19, 2012
entitled "Method And System For Digital Pulse Recognition Demodulation,''where
xi, is the
coordinate resolved from the front camera, x r is the coordinate resolved from
the rear camera,
and = h1lh2 is the ratio of the height of the device to the height of the
ceiling, we can resolve
the actual coordinate of the device, xa using:
¨ ri,(X Zn)+ Xh
Repeat Identifiers
102781 One potential issue that arises in a light based positioning system is
the possibility of
repeat light identifiers. Because the identification code on each light source
is not guaranteed to
be globally unique, additional techniques can be to resolve which identifier
is currently being
identified. In order to "narrow down" the possible candidates for which light
bulb is being
currently detected by the camera, an initial, broad location can be determined
using geofencing.
This is done via the GPS sensor of the device, or alternative sensors such as
WiFi, Bluetooth,
ultrasound, or other location technologies that would be readily understood by
a person of
ordinary skill in the art. Once the location sensor returns a location, a list
of nearby light sources
and their corresponding identification codes is retrieved. These lights will
become the
"candidates." Any identification code detected by the mobile device's 103
image sensor will be
compared against this list of candidate lights in order to determine the
device's location. This
process happens every time the mobile device 103 detects a major location
change.
102791 FIG. 56 presents an algorithm for resolving between multiple light
identifiers. Acquire
Candidates 5601 fetches a list of nearby light identifiers. The list of nearby
identifiers becomes
the candidates new light identifiers will be compared to. Examine recent
identifiers 5602 then
looks at the list of light identifiers that the mobile device 103 has recently
seen. Determine
distance from candidate 5603 then computes the distance between the current
recently seen
identifier and the second most recently seen identifier. This distance is
defined as the difference
between the signals. The concept is quite similar to a Hamming Distance, which
measures the
- 62 -
CA 02892923 2016-12-07
minimum number of substitutions required to change one codeword to another.
Each of these
candidate distances are stored in a buffer 5604. Candidates Remaining 5605
continues to
evaluate light signatures until all the candidates have been added to the
candidate distance buffer.
Alter all the signals have gathered and their distances
- 62a -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
are computed, Analyze Candidate Set 5606 looks at the candidate signals, their
distances, and
then ascertains which candidate is most likely.
Qlla I ty Score Determination
102801 A challenge when using a light based positioning system is
determining whether
or not a current signal contains the light identification signature. In order
to quantify this, the
present disclosure contains a quality score that measures the quality of the
current detection.
This quality score is analogous to a signal to noise ratio. A signal to noise
ratio can be
defined as the ratio of the power between a signal and the background noise.
This can be
measured in a number of ways, including taking the average power of each, the
root mean
square (RMS) of the respective powers, or the ratios in decibels. The present
disclosure
contains multiple methods for determining signal quality. In one embodiment,
the quality
score can be defined as the ratio of magnitudes of peaks of the signal
spectrum. An algorithm
first checks that all peaks of the spectrum are above a specified threshold,
and then measures
the ratio of the highest peak to the second highest peak. Another component of
the quality
score can be the distance of the peaks from the expected location of the
peaks. For example,
if a mobile device 103 were demodulating a DPR signal that was known to use
the
frequencies 1.000Hz, 11.00Hz, and 1200Hz, and the receiver detected a signal
of 800Flz, the
mobile device receiver's quality score would take into account the 200 Hz
difference between
the detected frequency 800 Hz and the closest known frequency of 1000 Hz.
(02811 in addition to using analytical techniques such as ratio of peaks
and distance
between peaks, quality score determination also can be performed using pattern
recognition
algorithms. Pattern recognition algorithms are concerned with providing a
label for a given
set of data. Such algorithms output a confidence score associated with their
choice, which
can be used as the quality metric. If the quality metric/confidence score is
too low, then the
algorithm flags the result as not containing a signal. In the context of a
light positioning
system, this refers to a situation where the output of the demodulation
algorithm is ignored.
There are a number of techniques for pattern recognition, including neural
networks, Hayes
classifiers, kernel estimation, regression, principal component analysis,
Kalman filters, or
other such algorithms that would be readily understood by a person of ordinary
skill in the
art.
- 63 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
Lighting Control System
102821 In a variety of use cases, it is desirable to control certain
aspects such as the
brightness, color, schedule, or direction of a light. There are a number of
technologies for
lighting control, including both wireless systems such as Zigbee, WiFi, and
6LowPan, as well
as wired technologies such as DMX or other standard lighting control
protocols. An
important component of a lighting control system is the ability to know the
physical locations
of the lights such that they can be controlled.
102831 For example, if a user wanted to remotely control a light in a
particular room, the
central controller would need to know the locations of all lights within a
building.
Furthermore, a user might want to only control lights in their direct
vicinity. Consider the
situation in FIG. 58, where a mobile device user 5803 is standing underneath
DPR enabled
lights 5802b and 5802c. Each light 5802a-c is broadcasting a DPR modulated
signal, which
mobile device 5804 detects using its sensors. The mobile device then computes
its indoor
location, using the algorithms discussed previously. The mobile device user
5803 is granted
permission to control the lights 5802a-c around them. In one embodiment of the
present
disclosure, the mobile device user has a "lighting profile" that is associated
with them. This
lighting profile is used to adjust the lights in the room automatically,
depending on the user's
preferences.
Lighting Control System Commissioning
102841 A common problem when configuring a lighting system is identifying
the physical
locations of lights for the purposes of lighting control. There are two steps
when configuring
a networked lighting control system. One step is network discovery, which is
the problem of
identifying devices on the lighting network. In one embodiment, based on wired
connections,
devices on the network may be connected through power line communication
(PLC),
Ethernet, fibre optics, or other wired communication as would be readily
understood by one
of ordinary skill in the art. In another embodiment, the light sources can be
connected
wirelessly using protocols such as WiFi, Zigbee, Bluetooth, 6LowPati, or other
protocols that
would be readily understood by a worker skilled in the art. In both
embodiments, the
problem of network discovery is accounted for in the underlying protocol,
during which each
device within the network is assigned a unique identifier.
102851 The second step of configuring a networked lighting control system
is determining
the physical location of each light within the system.. There are a number of
existing
- 64 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
techniques for doing this, all of which rely on lots of manual labor on the
part of the
individual tasked with performing the commissioning. The present disclosure
provides a
method by which beacon based light sources broadcast a self-identifying
pattern that is
recognizable by a mobile device receiver. This self-identifying pattern can be
used to assign
the location of the light source when commissioning a light positioning
system. In one
embodiment, the signal. is demodulated using digital pulse recognition (DPR)
techniques on
an image sensor present on mobile device 5804.
102861 Consider the situation presented in FIG. 58. Mobile device user 5803
stands
underneath light sources 5802a-c. Each light source 5802a-e is broadcasting
DPR modulated
signals, and is connected to a network 5801. The lights be can networked using
any standard
networking technique. As the mobile device user 5803 receives information
through the light
signals, they can use the received signals to commission the system. Mobile
device user
5803 receives the light signals, and then assigns them a physical location
within a building.
The mobile device user can do the physical position assignment using a user
interface like the
one described in FIG. 48, or via other inputs that would be readily available.
102871 For a networked lighting system, like the one presented in FIG. 58,
it is desirable
to provide a mobile device user 5803 with the ability to control the lights
both locally and
remotely. If the lights are networked, the mobile device user 5803 could
control the lights by
selecting them manually via the user interface terminal. The lights could also
adjust
depending on the time of day, occupancy sensors, current energy prices,
ambient light
readings, or user preferences. Transmitting an identifier through a modulated
light 5802e,
which could be modulated using DPR modulation, to a mobile device 5804, allows
the
mobile device user to directly control the light source. For example, consider
the case where
mobile device user 5803 is standing underneath light source 5802c, receiving a
light identifier
for the light source. The mobile device user could then send the identifier to
a lighting
control module, along with a set of instructions to control the light source
5802c. This set of
instructions could include the color, brightness, light direction, time
varying signal, on/off
schedule or any other parameters of the light source that a user would
control. The light
identifier could either be the network identifier of the light source (for
example, the .IP
address if the light source were controlled over TCP/IP), or an identifier
that could be
correlated with the light source through the lighting controller.
- 65 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
Smart Light Bulb Renorting
[0288] Acquiring metrics for light sources is typically a very cumbersome
process. In
most cases, a person wishing to measure light source metrics needs a special
handset
equipped with a multitude of sensors. This reporting process is labor
intensive, expensive,
and limited in the amount of data that it can gather. Data about how a light
source performs
over time is of large interest to light source manufacturers, who want to
track how their
products perform outside of their internal test facilities.
102891 There are a number of metrics that a bulb may want to report. These
include total
time on, age, color shift, temperature history, current fluctuations, voltage
fluctuations,
individual LED performance, barometric pressure readings, dimming statistics,
color
statistics, manufacturer origin, parts included within the bulb, network
status, or other such
information that could be connected to the light source.
102901 FIG. 46 presents a system by which a light source uses a light
based.
communication channel to report information about itself. A processing unit
4602 interfaces
with sensors 4603. Possible sensors include temperature, color temperature,
electrical,
barometric, network, or any other sensor that could be added by one skilled in
the art. The
processing 4602 acquires readings from the sensors at regular intervals, and
then broadcasts
those readings through modulator 4607. Modulator 4607 is responsible for
modulating the
light output of light source 103. In one embodiment of the present disclosure,
modulator
4607 is a DPR modulator, as described previously in FIG. 21. Data 4602 is
passed from
sensors 4603 to processing unit 4602, before being sent to Encoder 2102. The
packet
structure could be determined either in the processing unit 4602 or the
modulator 4607.
Within the packet, in addition to the standard start bits, data bits, and
error bits, there could
also be metadata tags that indicate the type of information in the packet. For
example, there
could be a special header that would indicate that the packet contains
information about bulb
longevity. The header could contain the length of the bulb longevity portion,
or the length
could be set as a predetermined portion. The header could also include an
identifier to
indicate which packet was being transmitted in a series of packets. This can
be used for
situations in which multiple pieces of data are being transmitted in
succession. The design of
a packet is well understood by workers of ordinary skill in the art. Th.e
present disclosure
includes a packet structure that contains an identifier for the light source,
light source
dimming information, longevity, temperature, and color temperature.
- 66 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
[0291] A timestamp is added to the data by the mobile device receiver 103.
After the
mobile device receiver completes demodulation of the packet, the receiver can
either send the
data to a remote server for storage, or store it locally. In the case of a
poor network
connection, the receiver device 103 would store the light source information
before
transmitting the data to a server once a sufficient connection is formed. In
this way, mobile
device receivers can crowd source the collection of information about light
source
performance, without having to do manual surveying. This could be done in
conjunction
with light sources providing an indoor positioning service.
Bar Code Scanner
102921 A common problem for employees of retail stores, hospitals,
warehouses, or other
such locations is physically tagging the location of objects, products, and
supplies. For
example, retail employees are frequently tasked with scanning barcodes when
performing
inventory in store. Typically, after an employee scans a code, they must
manually input the
location of the product into the system. To speed up this process, the
presentsystem utilizes
light based positioning in conjunction with barcode scanning. FIG. 59 contains
a depiction of
a mobile device user 5901 standing underneath light sources 4401a-c, each
broadcasting a
modulated light signal. When the mobile device user successfully scans a
product code 5902,
the product code is uploaded to a remote server along with its current
position, which is
derived using light positioning. This reduces the manual labor required on the
part of the
mobile device user. The product code could be a bar code, QR code, product
photo, RF-ID,
photo, or any other tag that would be readily understood by a person skilled
in the art.
Alternative Sensors for DPR Demodulation
102931 Using an image sensor which utilizes a rolling shutter mechanism for
exposing
the image is a convenient method for performing DPR demodulation on today's
mobile
devices. This is because the vast majority of devices contain CMOS sensors for
capturing
multimedia content. However, other image sensors can be used. The CMOS sensor
is
convenient because the rolling shutter functionality acts as a time-domain
sample of the
illuminated DPR signal. However, this functionality can be replicated on any
type of optical
sensor. For example, if instead of rolling shutter CMOS a global shutter
charge coupled
device "CCD" sensor was used, the sampling function would simply change from a
single
fram.e to multiple frame analysis. Light intensity fluctuations of individual
pixels on the
recovered image can be sampled across multiple frames. If a photodiode were
used as the
- 67 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
sensor, the photodiode would be sampled by an analog to digital converter and
then sent to a
processing unit. In all of these cases, the only aspect that changes is the
response function of
the receiving device. One of ordinary skill in the art would readily
understand the
requirements to support DPR demodulation on alternative receivers, including
but not limited
to photodiodes and global shutter image sensors.
DPR Enclosure Module
102941 FIG. 60 describes a physical DPR enclosure 6001 which contains the
DPR
modulator 6004 and stress relieved 6003a-b incoming connections 6002 and
outgoing
connections 6005.
102951 The techniques and methods disclosed for use in light based
positioning systems
can be used with a variety of camera equipped mobile or stationary devices,
such as: mobile
phones, tablet computers, netbooks, laptops, desktops, or custom designed
hardware. Further,
the scope of the present invention is not limited to the above described
embodiments, but
rather is defined by the appended claims. These claims represent modifications
and
improvements to what has been described.
Electrical Design
102961 FIG. 61 depicts a Self-Identifying Optical Transmitter 6101 in
accordance with
some embodiments, which transmits an identifier via an optical signal. In some
embodiments, the Self-Identifying Optical Transmitter 6101 may also transmit
additional
data using the optical signal. Processing Unit 6104 sends a signal to
modulator 6105.
Processor Unit 6104 can come in the form of a microcontroller, FPGA, series of
logic gates,
or other such computing element that would be readily understood by a person
of ordinary
skill in the art. The processing unit can also be a series of logic
elements/switches controlled
via analog circuitry. To minimize power consumption, the processing unit can
be configured
to operate in a low power state. Minimizing power consumption can be important
in the case
of a battery-powered system. The Self-Identifying Optical Transmitter 6101
receives power
from Power Source 6103. In some embodiments, Power Source 6103 is a battery-
powered
system.. Power Source 6103 can. also be in the for.. of an external power
source. The
Processing Unit 6104 contains a Data Interface 6109, which can be used to
program the
device, exchange data, or load information to be transmitted across Light
Output 6110. An
executable program for Processing Unit 6104 may be pre-loaded before the Self-
Identifying
Optical Transmitter 6101 is deployed. This program may be in the form of one
or more
- 68 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
software modules. The processing unit contains a non-volatile memory storage
area 6102 for
storing information to be transmitted across the Light Output 6110. This
information can
include an identifier for the Light Output 6110. The identifier can be a quasi-
unique
identifier, or globally unique if the identifiers were centrally-managed or
pre-configured.
This quasi-unique random identifier is received on the mobile device, which
then verifies that
the mobile device came into close proximity with the self-identifying one-way
optical
transmitter. In some embodiments, the data transmission may include data in
addition to the
identifier, such as meta-data. For example, the optical data transmission
stream can include
information about the current battery state of the device. This information
can be used to
inform the owner of the device that it is time to change the battery.
102971 Modulator 6105 is used to control the power transmitted through the
light source,
thus driving the optically encoded signal in Light Output 6110. Modulator 6105
can come in
the form of a MOSFET, Rif, transistor switch, discharge circuit, or any such
device for
controlling current/voltage that would be understood by a person of ordinary
skill in the art.
102981 Light source 6106 may take an electrical input from modulator 6105
and convert
it into an optical signal. In some embodiments, light source 6106 may include
light emitting
diodes (LEDs). LEDs may be current-driven devices, in which case the
brightness of the
LED is proportional to the current being driven through it. In other
embodiments, the LEDs
may be voltage-controlled, or other light sources aside from LEDs may be used
that have
different current / voltage characteristics. The light sources pulse the
electrical signal
received from the modulator in an optical format.
102991 Optics 6108 disperse the optical sigma! 6107 emitted from. the light
source 6106 to
improve the ability of a mobile device 6203, as represented in FIG. 62, to
receive and
interpret the Light Output 6110. There are a number of reasons for this. For
example, in the
case of light source 6106 including LEDs, the optical signal 6107 may be
emitted from a
single point source. Because it is desirable to allow a mobile device user to
quickly scan their
mobile device past the self-identifying optical transmitter, increasing the
surface area of the
optical signal may helpfully reduce the time used by a mobile device user to
align their
mobile device with the light source 6106. Furthermore, because mobile devices
6203 can
come in many different form factors, it is desirable to have a wide surface
area in order to
support many different mobile devices. Dispersing the optical signal also
reduces the
likelihood that the brightness of individual point LED sources will wash out
the image sensor
on mobile device receiver 6203. Furthermore, viewing an LED with the naked eye
without
dispersing the light may annoy users.
- 69 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
103001 FIG. 62 represents the use of self-identifying optical transmitters
by museum
patrons in a museum space, in accordance with some embodiments. This
representation
includes a museum space 6205, museum patrons 6201 and 6204, mobile device 6203
held by
museum patron 6204, and self-identifying one-way optical transmitters 6202a-
6202d. In
these embodiments, museum patrons 6201 and 6204 equipped with mobile devices
browse a
museum space 6205 and may desire additional information about the exhibits of
the space.
Self-identifying one-way optical transmitters 6202a-6202d may be positioned
throughout the
exhibit 6205. These transmitters may be strategically placed near points of
particular interest,
near points frequently visited by museum patrons, near points that are easily
accessible by
users of mobile devices, or in any other configuration within a space. The
self-identifying
optical transmitters may be mounted or placed vertically, horizontally, or at
any other angle
within the space. The self-identifying optical transmitters may be mounted or
placed on
walls, pedestals, displays, ceilings, or any other place visible by users of
mobile devices.
Self-identifying optical transmitters may also be positioned in other types of
spaces, such &s
grocery stores, shopping malls, or transit stations. One or many self-
identifying optical
transmitters may be placed in these sites.
103011 Within the museum space 6205 that is equipped with self-identifying
optical
transmitters 6202a-6202d, museum patron and mobile device user 6204 may tap
their mobile
device 6203 onto the self-identifying one-way optical transmitter 6202d. The
museum patron
may do this because he or she desires more information about the portion of
the museum
space in which he or she is located, because he or she wishes to report an
error about the
space, or for any other reason. This self-identifying one-way optical
transmitter 6202d is
mounted on a wall within the museum space 6205, and the light output from the
transmitter
may be received by an image sensor of the mobile device 6203. A user does not
necessarily
need to tap their mobile device onto one of the self-identifying optical
transmitters, but
instead may place their mobile device in any position where it is able to
receive the light
output from. a self-identifying optical transmitter. Upon receipt of the light
output, the mobile
device 6203 may interpret the light output as a code. This code can be used to
pull up
additional content associated with the current location of self-identifying
optical transmitter
6202d within the museum space 6205. In the case of a retail store, the code
may be used to
depict an advertisement, acquire a coupon, receive credit for visiting a
location, order an
item, call for customer service, or pull up an online order form. Any other
information by be
received and interpreted by the mobile device 6203 and then related to the
location of the
mobile device 6203.
- 70 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
103021 In addition to using a self-identifying transmitter such as the one
described in FIG.
61 to receive information, as described above, a mobile device user 6201 or
6204 may use the
self-identifying optical transmitters 6202a-6202d to assign content to a
particular location.
For example, a common scenario when taking photographs is to tag the
photograph to a
particular physical location. The EXIF format, which defines meta-data tags
that are added to
images, specifically builds in a standard set of tags for location
information. These tags are
commonly added when taking photographs with a GPS enabled mobile device.
103031 A mobile device user can utilize, for example, the self-identifying
transmitter
6202a to geo-tag media, in the same way that GPS is used to geo-tag a picture.
Note that the
word geo-tag refers to associating a piece of digital media with a particular
location. This
location can be derived by the self-identifying transmitter, or by using
position derived from
light as described above. In the case of mobile device user 6201 taking a
picture, this user
may first tap their mobile device onto self-identifying transmitter 6202a and
thus associate
that picture with the transmitter 6202a. The picture can then be uploaded to a
remote server,
with the corresponding identifier used to associate the picture with the self-
identifying
transmitter 6202a. In some embodiments, the gee-tagging information may be
derived from
an overhead light positioning system, such as the one described previously in
U.S. Pat. App.
No. 13/526,773 filed June 19, 2012 entitled "Method and System For Digital
Pulse
Recognition Demodulation." Note that the media is not necessarily limited to
photos, but can
take the form n of audio recordings, movies, video, user generated text, or
any such media that
can be generated.
103041 FIG. 63 is a representation of a self-identifying optical
transmitter 6301. The
transmitter includes light sources 6303a and 6303b that emit light towards a
surface 6302.
The self-identifying optical transmitter 6301 may be placed in a museum, a
retail store, or
any other location where the optical signal emitted by the light sources may
be accessible by
a mobile device user. Although FIG. 63 depicts one embodiment of a self-
identifying optical
transmitter, a variety of housing shapes, material types, and light source
placements may be
used for altering or improving the optical performance of the self-identifying
optical
transmitter 6301. A. different number of light sources may also be used. In
some
embodiments, the light sources may be LED point sources. In these and other
embodiments,
the light sources may be are reflected off of a surface 6302. Reflecting the
light from the
light sources in this way may increase the area across which the light may be
received. The
surface is ideally colored white, but can be any color that sufficiently
reflects the light from
light sources 6303a and 6303b. The light sources 6303a and 6303b can also have
lensing on
-71-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
them in order to disperse the optical signal they emit. The lensing may be
placed directly
over the light sources, or on some other portion of the self-identifying
optical transmitter.
For example, in the case of a red light source being used, the surface can be
colored in a
different way in order to increase the reflectiveness. Note that the surface
6202 can be
angled, flat, rounded, or another such shape that would be understood by a
person of ordinary
skill in the art. In some embodiments, a lens may be placed at an angle to
surface 6202 such
that the light sources are enclosed within the self-identifying optical
transmitter and the lens.
103051 FIG. 64 is another representation of a self-identifying optical
transmitter 6401,
according to some embodiments. The self-identifying optical transmitter 6401
may contain
one or more light sources (not shown) and be enclosed by lensing 6402 to
disperse the light
from the light sources. A mobile device user may place their mobile device
near the self-
identifying optical transmitter 6401 or tap their mobile device onto the
lensing 6402 itself.
The shape of the lens may be round, flat, or any other shape that disperses
the light source.
Frosted lenses are a popular and effective choice for light dispersion, but
any lens may be
used. In some embodiments, lensing can be combined with a reflective surface
within the
self-identifying optical transmitter 6401 to improve the signal dispersion
characteristics even
further. In yet other embodiments, the top of the lens 6402 may be covered
with a film or
partially transparent surface to further improve optical dispersion.
103061 FIG. 65 depicts a self-identifying optical transmitter 6501 with a
rounded lens
6502, in accordance with some embodiments. As with the optical transmitters
described
above, the self-identifying optical transmitter 6501 may contain on or more
light sources and
a user may place their mobile device or near the transmitter to receive an
optical signal
broadcast from the light sources. The rounded lens can be of any dimension. A
rounded lens
holds advantages in terms of appearance and user experience. Since the rounded
lens is
raised, it forms an easier surface onto which a user may tap their mobile
device. This is
because the flat form factor of a typical mobile device will rest tangential
to the curved
surface of the rounded lens, making it unlikely that the camera lens will rest
directly on the
surface of the lens. Some separation between the lens and the surface is
desirable to avoid
washing out the image sensor.
[0307] FIG. 66 illustrates another possible implementation of a self-
identifying optical
transmitter 6601, in accordance with some embodiments. The sell-identifying
optical
transmitter 6601 includes four light sources 6602a-6602d arranged inside of a
housing. A
different number of light sources may also be used. The light sources 6602a-
6602d may be
LED light sources. These light sources project a signal within and around the
transmitter
- 72 -
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
device. Having multiple light sources may improve the optical power of the
transmitter.
Arranging the light source at different position and angles within, the self-
identifying optical
transmitter may increases the range and areas within which the optical signals
broadcast may
be received by a mobile device. However, some embodiments with multiple light
sources
have the disadvantage of possibly lowering the battery life because more light
sources may
draw a higher current. Each light source 6602a-d may contain individual
lenses.
103081 Conserving power, and reducing the light output for aesthetic
purposes, is a
desired feature of a self-identifying broadcast light source. One way to
achieve this is by
having the Processing Unit 6104 operate in a low power state. Another method
for
improving the lifetime and power consumption of a stand-alone optical-
transmitter is to
power cycle the device when it is not being used. There are a number of ways
to achieve
this. In some embodiments, there may be a proximity sensor on the transmitter
device 6101.
This proximity sensor is responsible for powering up the transmitter device
6101 when a user
comes close to it. This ensures that the transmitter broadcasts the optical
signal when there is
a user nearby to receive it. Other embodiments have a mechanical button or
switch that the
user taps to turn on the light. Other embodiments may include an inductive or
capacitive
circuit that is triggered upon contact with the mobile device 6101. All of
these possible
external inputs can connect to the self-identifying optical transmitter 6101
through data
interface 6109.
Modulation
103091 In some embodiments, digital pulse recognition (DPR) modulation is
used to
modulate the light source 6106 to transmit the light output 6110. As described
previously in
U.S. Pat. App. No. 13/526,773 filed June 19, 2012 entitled "Method and System
For Digital
Pulse Recognition Demodulation," DPR modulation pulses the light source with a
light
pattern. A mobile device receiver demodulates the signal by exploiting the
rolling shutter
mechanism on a CMOS image sensor. The receiver examines the incoming images
for
patterns encoded within the light pulses.
103101 In other embodiments, changing color patterns can be used to
transmit an
identifier. In embodiments where light source 6106 includes RGB :1..EDs, the
information
may be encoded into a series of changing color patterns. A mobile device
receiver 6203 can
examine the pattern of color shifts and decode the corresponding identifier.
Note in both
embodiments that use DPR modulation and embodiments that use color modulation,
the
information transfer is not necessarily limited to the transmission of an
identifier. Additional
- 73 -
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
information can also be encoded into the series of light pulses. This can
include the battery
state of the transmitter, total time on, or data sent via Data Interface 6109.
Note that Data
Interface 6109 can connect with any arbitrary data source.
103111 In some embodiments, pulse width modulation (PWM) may be used to
transmit an
arbitrary signal. PWM is a desirable choice of modulating LED sources because
it reduces
color shift in the LEDs. Because the color-shift of an LED changes depending
on the amount
of current that is driven through it, it is desirable to transmit a fixed
current through the
LEDs, and control the brightness by varying the time the LED is "on." For this
reason, many
LED light sources use PWM dimming techniques, as opposed to constant current
techniques
that can cause color shift. By using PWM to construct an arbitrary waveform,
information
transfer can be facilitated across an optical channel while preserving the
color quality of the
LED output. PWM modulation is used to transmit an arbitrary signal by varying
the duty
cycle of a pulse at a set frequency. The width of the pulse represents a
quantization of the
amplitude of the arbitrary signal at a point in time.
103121 To facilitate the transfer of an arbitrary signal using PWM
modulation, first a
carrier frequency is chosen. The carrier frequency is the frequency of the PWM
wave. The
choice of carrier frequency depends on a number of factors. These include the
quantization
resolution of the transmitter, the clock speed of the transmitter, and the
sampling rate of the
receiver.
103131 FIG. 67 illustrates an example waveform for using PWM to transmit an
arbitrary
signal. As discussed above, this can be used to drive an LED light source. A
carrier wave
6703, representing a PWM wave with a set frequency, is used to represent an
arbitrary signal
6702. At any point on the arbitrary signal, the width of the pulse 6701 is
used to represent
the amplitude of the signal 6702. In this manner, the PWM wave may act as an
analog-to-
digital converter for an arbitrary signal. For example, consider the waveform
presented in
FIG. 67. The PWM carrier wave has a frequency of 64 Khz. The period of the
wave is
1/64Khz, or 15.6 microseconds. At each point along the arbitrary signal 6702,
the amplitude
of the signal is converted to a corresponding pulse width. So if the amplitude
of the signal is
50% of its max value, the pulse width may be 50%. With a 641(hz wave, this
corresponds to
a pulse width of 7.8 microseconds. If the amplitude is 25% of its max value,
the pulse width
may be 25%, or 3.9 microseconds. In these embodiment, an analog signal is
converted to a
digital representation as a pulse width. Provided that the carrier frequency
is chosen to be
lower than the Nyquist rate of the mobile device receiver, any arbitrary
signal can be
transmitted to the mobile device.
-74-
CA 02892923 2015-05-28
WO 2014/063150 PCT/US2013/065923
[03141 In some embodiments, this PWM wave can be used to construct a sum of
sinusoids. The combination of frequencies can then be mapped to an encoding
scheme to
uniquely identify a light source, or transmit other such data. Possible
modulation techniques
that can be constructed from PWM waves include Orthogonal Frequency-Division
Multiplexing, Quadrature amplitude modulation, amplitude modulation, phase-
shift keying,
on-off keying, pulse-position modulation, spread spectrum, or other such
modulation
techniques that would be understood by a person of ordinary skill in the art.
103151 On the receiver side, there are a number of techniques for improving
the
capabilities of the mobile device. As described previously in U.S. Pat. App.
No. 131526,773
filed June 19, 2012 entitled "Method and System For Digital Pulse Recognition
Demodulation," image sensor settings such as exposure, white balance, frame
rate, and
resolution may be modified in ways that improve the signal-to-noise
characteristics of the
receiver. In general, increasing the frame rate of the receiver increases the
sampling rate, and
consequently the data rate of the receiver. Furthermore, increasing the
resolution of the
received image also improves the receiver capabilities. This is due to the
discrete nature of
the sampling process. In some embodiments, when using a Fourier transform
based approach
to recovering the transmitted signal, increasing the resolution of the image
sensor on a
receiver effectively increases the number of samples. Increasing the number of
samples
when performing Fourier analysis may increase the frequency resolution of the
Fourier
transform. In applications that utilize the transmission of individual tones,
or sets of tones,
to identify light sources, increasing the frequency resolution is important in
increasing the
number of identifiers that can be transmitted. Another benefit of increasing
the frequency
resolution is that, in some cases, this overrides optimizations done at the
image sensor level.
For example, a number of image sensors may compress an image before returning
it to the
higher layers of the software stack. Because this compression often happens at
the driver
level, some camera APIs may not be able to override the compression. Data
compression can
cause information within the image to be lost. In the case of a DPR modulated
signal, this
may cause the image to lose features that carry information transmitted via
the optical signal.
Thus, increasing the resolution of the image sensor 501 has the dual benefit
of increasing the
bandwidth of the receiver 103, as well as reducing the possibility of
information loss through
compression and other optimizations at the image sensor level.
-75-
CA 02892923 2015-05-28
WO 2014/063150
PCT/US2013/065923
Applications
103161
Associating the media with the self-identifying transmitter, or a position
derived
from light, allows for users to overlay digital content onto a physical space.
Future visitors to
the same location can scroll through the experiences of previous visitors to
the space. Users
watching remotely would be able to view a live feed of activity in the space,
and see digital
content as it is generated. As an example, consider the situation presented in
FIG. 68. A
high-brightness light source 6801 is broadcasting a modulated signal overhead.
The signal
6802 may be reflected off the clouds, or other large objects. Mobile device
users, such as
user 6804, can. point their devices 6803 at the reflected signal 6802, which
is broadcasting an
optical signal. Mobile device users can then upload text, status updates,
audio, video,
pictures, or other such media, which is tagged with the identifier received
via the visible light
signal. Remote users can then subscribe to a media feed that is tagged with
the particular
physical location. Users who view the cloud reflected signal 6802 can also be
redirected to a
web link, which can present them with advertisements, directions to a nearby
restaurant,
deals, additional information about a venue, a ticketing system, or other such
data that would
be understood by a person of ordinary skill in the art.
103171 A common
usage scenario for one-way authentication systems is verifying that a
user has visited a particular location. For example, consider the situation in
FIG. 69. Mobile
device user 6902 is passing through a checkout aisle 6901. The user may be
participating in a
customer loyalty program, where they receive a credit for walking through the
checkout aisle.
In order to verify the user has actually gone through the checkout aisle, a
mechanism may be
used to tie the user's location to that particular location. The user 6902 may
tap their mobile
device 6904 onto a self-identifying optical transmitter 6903 placed n.ear the
point of interest.
This transmitter 6903 transfers an identifier to the mobile 6904, which stores
the identifier to
verify their position. This identifier can be used on an application running
locally on the
device, or transmitted to a remote server. The identifier provides a secure
method by which
the user's position may be verified. Combined with information about the
user's identity,
previous purchase history, demographics, and previous locations visited, the
authenticated
location transmission can be used to administer the customer loyalty program.
Note that this
approach has significant security advantages over QR codes, because it is very
difficult to
replicate the optical signal transmitted by transmitter 6903.
- 76 -
