Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
SYSTEM AND METHOD FOR ANALYSIS OF LIGHT-MATTER
INTERACTION BASED ON SPECTRAL CONVOLUTION
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to the following applications, each of
which is hereby incorporated by reference in its entirety: United States
Provisional Patent
Application No. 61/019,440, filed January 7, 2008; United States Provisional
Patent
Application No. 61/061,852, filed June 16, 2008; and United States Patent
Application
No. 11/970,448, filed January 7, 2008.
BACKGROUND
[0002] Field:
[0003] Embodiments of the invention may include methods and apparatus for
enabling the collection of dermal and non-dermal images using a non-invasive
imaging
device, the development of a skin state based at least in part on analysis of
such images,
and the monitoring of the skin state by, at least, a collection and analysis
of subsequent
images. Embodiments of the invention may further pertain to the field of skin
care
devices and systems capable of facilitating skin care decisions, more
specifically the field
of devices for skin condition assessment, skin care regimen recommendation,
and skin
care regimen effectiveness tracking.
[0004] Embodiments of the present invention may also relate to an image
processing technique. More particularly, embodiments of the present invention
may relate
to determining a skin photo type of a captured image in Red Green Blue (RGB)
color
imaging system and is also applicable in classification of other skin
characteristics (e.g.
elasticity, melanin, oil concentration etc.), melanoma, skin related tumors
and skin related
disorders.
[0005] Description of the Related Art:
[0006] Skin is the largest organ of the integumentary system which further
includes skin's accessory structures, such as hair, nails, scales, feathers,
sweat glands and
their products. Skin comprises multiple layers of epithelial tissues that
guard underlying
muscles and organs. Since skin is subject to constant attack from various
external and
Page 1 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
internal factors, it can be afflicted by numerous ailments. Thus, it is
critical to monitor
skin health and the effect of any treatments, skin care products, or cosmetics
applied to
the skin.
[0001] A few known methods of skin care decision systems involving
conventional photography are also known as "clinical imaging." Some of the
known
methods involve illuminating the skin surface with white light when the
digital image is
acquired by the camera. The effectiveness of clinical imaging is, however,
compromised
by specular (mirror-like) reflection of the skin.
[0007] The existing processes of making skin care decisions for both medical
and cosmetic purposes are generally based on scattered information collected
from
multiple sources in different time periods. The existing processes are
cumbersome and
inefficient, resulting in delayed and sub-optimal skin-care decisions.
Examples of skin-
care decisions that can be significantly enhanced by the principles of the
invention
include diagnosis of skin conditions, selection of skin-care products and
regimens, and
tracking the effectiveness of skin-care products and regimens over a period of
time. A
skin-care regimen includes both selection of appropriate skin-care products
and the entire
procedure of applying the selected skin-care products, including the dosage,
timing,
methodology, and frequency of application of the skin-care products.
[0008] While various methods exist for determining and monitoring skin
health, most require access to a dermatologist or a dermatological facility,
thus, there may
be difficulty, inconvenience, and prohibitively high cost in accessing the
necessary
resources. There is a need for a simple solution for skin health determination
and
monitoring that may be operable by an untrained or trained user, and, in the
absence of an
in-person consultation, where dermal images may be submitted to an expert, an
analysis
facility, or for automated analysis.
[0009] Skin characteristics are typically determined using Fitzpatrick
classification. Skin phototype is categorized according to a conventional
Fitzpatrick Skin
Typing Test questionnaire (skin type scale), which ranges from very fair (skin
type I, for
example) to very dark (skin type VI, for example). Conventionally, various
image
processing techniques are disclosed for determining skin characteristics using
captured
skin images. In a conventional digital image processing technique it is often
useful to
Page 2 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
detect multiple features in the captured skin image, such as skin color. This
information is
used, for example, to adjust the skin colors in the image to be comfortable to
perceive.
The location of skin color is also used in face detection and face recognition
algorithms,
automatic image retrieval algorithms, and red-eye correction algorithms.
[0010] Less work has been done on objective measurement of human skin
coloration to enable its color classification. Classification of a person's
skin coloration
would be useful, for example, in the medical field for quantification of skin
erythema,
lesions, ultra-violet radiation effects, and other skin coloration phenomena.
In the field of
computer graphics images of people could be rendered more accurately in video-
conferencing and their appearance could be improved or altered. While the
science of
digital skin imaging analysis has identified various skin responses that are
useful
indicators of various skin condition parameters, it would still be desirable
to identify and
use additional specific responses of the skin that are indicative of skin
conditions and skin
characteristics.
[0011] Cosmetic appearance is one of the top priorities to most of the humans
in the modem world. Techniques or systems existing in the known art analyze
the skin
conditions and suggest suitable products to improve the cosmetic appearance of
a human
being.
[0012] There is a need for a minimal error and speed efficient method and
system to determine the phototype of skin.
SUMMARY
[0013] Real-time analysis of digitally captured skin characteristics
facilitates
timely skin condition assessment, skin regimen recommendation, and skin
regimen
effectiveness tracking.
[0014] The problem of generating a skin condition assessment in real-time is
solved by having a skin condition analysis module capable of doing real-time
analysis of
digital skin data, acquired partly using diffused reflectance spectroscopy
and/ or detecting
the red-green-blue components of re-emitted white light.
[0015] In an aspect of the invention, a skin care device may include an
electromagnetic radiation source capable of directing incident electromagnetic
radiation
Page 3 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
to a location on the skin of a user, a radiation detector for measuring
various parameters
of radiation re-emitted from the location, and a skin condition analysis
module coupled
to the detector, the analysis module capable of generating a skin condition
assessment in
real-time, based partly on at least one of RGB analysis and diffused
reflectance analysis
of the radiation parameters. In the device, the incident electromagnetic
radiation may
include radiation in at least one of the visible, near-infrared, and near-
ultraviolet
spectrum. The incident radiation may include white light. In the device, the
radiation
parameters may include at least the degree of polarization of the re-emitted
radiation. In
the device, the radiation source may be a set of light emitting diodes. In the
device, the
skin condition assessment may also be partly based on analysis of a
photographic image
of a skin region surrounding the location. In the device, the device may be a
miniature
device. Miniature may mean that no dimension of the detector exceeds six
inches. The
device may further comprise a memory module for storing the skin condition
assessment.
The device may further comprise a user interface. The user interface may be
operated
using voice commands. In the device, skin assessment data of locations may be
overlaid
on an image of a larger skin region and displayed on the display surface. The
device may
further comprise an access restriction module used for restricting access to
authorized
users only. The access restriction module may be based on biometric access
control. The
device may be capable of generating alerts about abnormal skin conditions in
real-time.
The device may further comprise a skin care regimen recommendation module that
generates a displayable skin care regimen recommendation. The skin care
regimen
recommendation may be based at least partly on determination of a skin profile
of the
user and use of skin care regimen recommendations of persons with a similar
profile.
The skin care regimen recommendation module may be linked to a product
database.
The product database may include products available in a point-of-sale
location. The
availability of a specific product recommended by the skin care regimen
recommendation
module may be indicated by an audio-visual signal. The device may further
comprise a
skin care regimen effectiveness module that generates a displayable skin care
regimen
effectiveness report. The device may further comprise a communication module
for
communicating with a remote computer. The communication may occur wirelessly.
The
Page 4 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
communication may occur over an internet. The remote computer may be operable
by a
physician. The device may be wand-shaped. The device may be wearable by the
user.
[0016] In an aspect of the invention, the skin care device may include an
electromagnetic radiation source capable of directing incident electromagnetic
radiation
to a location on the skin of a user, a detector for measuring various
parameters of
radiation re-emitted from the location, a skin condition analysis module
coupled to the
detector, the analysis module capable of generating a skin condition
assessment in real-
time, based partly on at least one of RGB analysis and diffused reflectance
analysis of the
radiation parameters, and a display panel for reflecting the image of the
user. In the
device, the display panel may be touch-sensitive such that touching the
location in a skin
region image displayed in the display panel triggers display of a magnified
image of the
location. The device may further comprise a camera. The camera may be integral
with
the display panel. The camera may be wirelessly linked to the display panel.
In the
device, the display panel may be a mirror. In the device, a stored image of
the user is
used to automatically identify the person. The device may further comprise a
user
interface for controlling the skin care device. The user interface may be
operated using
voice commands. The device may further comprise a skin care regimen
recommendation
module capable of generating a displayable skin care regimen recommendation.
The skin
care regimen recommendation may be based at least partly on determination of a
skin
profile of the user and use of skin care regimen recommendations of persons
with a
similar profile. The device may further comprise a skin care regimen
effectiveness
module capable of generating a displayable skin care regimen effectiveness
report.
[0017] In aspects of the invention, an imaging device permits a user to take
high magnification pictures of the skin in the vicinity of an area of concern
and submit
those pictures, optionally along with textual and data responses, for medical,
non-
medical, and cosmetic analysis, diagnosis and treatment recommendation and
follow-up.
[0018] In an aspect of the invention, a method and system of a non-invasive
imaging device may comprise an illumination source comprising an incident
light source
to direct light upon skin; and a detector for detecting the degree of
polarization of light
reflected from the skin. In the method and system, the illumination source may
be
positioned to direct light at a selected angle alpha. Varying alpha may vary
the depth of
Page 5 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the measurement of the layers in the skin. Each depth may have a specific
angle which
produces a full polarized reflection. In the method and system, the incident
light source
may be an unpolarized light source. The unpolarized light may be white light,
multiple
selected wavelengths, or a single wavelength. The method and system may
further
comprise a sensor for capturing an image of the reflected or re-emitted light.
The method
and system may further comprise an optical facility for detecting reflected or
re-emitted
light from the skin. The method may determine both reflected or re-emitted
light, and
newly emitted light, through the process of absorption and re-emission. The
method and
system may further comprise a communication facility for transmitting the
detected
information. The method and system may further comprise a storage facility for
storing
information collected by the device.
[0019] In an aspect of the invention, a method and system for determining a
skin state may comprise illuminating skin with an incident light source,
detecting the
degree of polarization of light reflected from the skin, and determining a
skin state based
on an aspect of the polarization of the reflected or re-emitted light. In the
method and
system, the incident light may be directed at a selected angle alpha. Varying
alpha may
vary the depth of the measurement of the layers in the skin. Each depth may
have a
specific angle which produces a full polarized reflection. In the method and
system, the
incident light source may be an unpolarized light source. The unpolarized
light may be
white light, multiple selected wavelengths, or a single wavelength. In the
method of
claim, the aspect of the polarization may be at least one of an orientation,
an amplitude, a
phase, an angle, a shape, a degree, an amount, and the like. In the method and
system,
determining may be done using an algorithm. The algorithm may involve
artificial neural
networks, non-linear regression, genetic algorithms, fuzzy logic, fractal and
multi-fractal
analysis, and the like. The methods and systems may further comprise filtering
the
reflected or re-emitted light to obtain polarized light of at least one
wavelength defined
by the filter output. The algorithmic analysis may be performed on the
filtered image. In
the method and system, determining may involve creating an image from the
difference
between the reflected diffusion light and the reflected polarized light. In
the method and
system, determining may involve comparing the aspect of the polarization of
the reflected
Page 6 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
or re-emitted light to a calibration signal. In the method and system,
determining may
further comprise considering at least one of user input and a visual analysis.
[0020] In an aspect of the invention, a non-invasive imaging device may
comprise an illumination source comprising an incident light source to direct
light upon
an area of concern and a detector for detecting the degree of polarization of
light reflected
from the area of concern. In the method and system, the illumination source
may be
positioned to direct light at a selected angle alpha. Varying alpha may vary
the depth of
the measurement of the layers in the skin. Each depth may have a specific
angle which
produces a full polarized reflection. In the method and system, the incident
light source
may be an unpolarized light source. The unpolarized light may be white light,
multiple
selected wavelengths, or a single wavelength. The method and system may
further
comprise a sensor for capturing an image of the reflected or re-emitted light.
The method
and system may further comprise an optical facility for detecting reflected or
re-emitted
light from the skin. The method and system may further comprise a
communication
facility for transmitting the detected information. The method and system may
further
comprise a storage facility for storing information collected by the device.
[0021] In an aspect of the invention, a method of determining moisture levels
in the skin may comprise emitting incident light towards a skin structure,
detecting a
degree of polarization of the light induced by the skin structure, and
determining a
moisture level based on the amount of polarized and reflected or re-emitted
light. The
method and system may further comprise combining the assessment of moisture
level
with skin color measurements to determine luminosity. In the method and
system, the
incident light may be unpolarized light. The unpolarized light may be white
light, light
of multiple selected wavelengths, or of a single wavelength, or one or more
monochromatic lights. In the method and system, determining may involve use of
an
algorithm. In the method and system, determining a moisture level may be based
on the
ratio of polarized and reflected or re-emitted light.
[0022] In an aspect of the invention, a method and system of determining
elasticity of the skin may comprise emitting incident light towards a skin
structure,
detecting an aspect of polarization of the light reflected by the skin
structure, correlating
the aspect of polarization with a concentration of elastin, and determining
elasticity level
Page 7 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
based on the concentration of elastin. In the method and system, determining
may
involve use of an algorithm. In the method and system, the incident light may
be
unpolarized light. The unpolarized light may be white light, light of multiple
selected
wavelengths, or a single wavelength of light.
[0023] In an aspect of the invention, a method and system of determining
firmness of the skin may comprise emitting incident light towards a skin
structure,
detecting an aspect of polarization of the light reflected by the skin
structure, correlating
the aspect of polarization with a concentration of at least one of an elastin,
a collagen, and
an activity of a sebaceous gland, and determining firmness based on the
concentration of
at least one of elastin and collagen and sebaceous gland activity. In the
method and
system, the sebaceous gland activity may be indicated by at least one of a
number of
glands, percent of glands open/closed, and level of clog/ fill. In the method
and system,
correlating may involve use of an algorithm.
[0024] In an aspect of the invention, a method and system for obtaining
dermal biophysical properties may comprise performing a spectral analysis of
image data
acquired from the degree of polarization of reflections and absorption and re-
emission of
incident light from skin structures, wherein the property is at least one of a
structure,
form, concentration, number, size, state, and stage of at least one of a:
melanocyte,
melanin, hemoglobin, porphyrin, keratin, carotene, collagen, elastin, sebum,
sebaceous
gland activity, pore (sweat and sebaceous), moisture level, elasticity,
luminosity,
firmness, fine line, wrinkle count and stage, pore size, percent of open
pores, skin
elasticity, skin tension line, spot, skin color, psoriasis, allergy, red area,
general skin
disorder or infection, tumor, sunburn, rash, scratch, pimple, acne, insect
bite, itch,
bleeding, injury, inflammation, photodamage, pigmentation, tone, tattoo,
percent burn/
burn classification, mole (naevi, nevus), aspect of a skin lesion (structure,
color,
dimensions/asymmetry), melanoma, dermally observed disorder, cutaneous lesion,
cellulite, boil, blistering disease, congenital dermal syndrome, (sub)-
cutaneous mycoses,
melasma, vascular condition, rosacea, spider vein, texture, skin ulcer, wound
healing,
post-operative tracking, melanocytic lesion, non-melanocytic lesion, basal
cell
carcinoma, seborrhoic keratosis, sebum (oiliness), nail- and/or hair-related
concern, and
the like.
Page 8 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[0025] In an aspect of the invention, a system and method may comprise
providing an interface that includes a social networking domain or rating-and-
ranking
system and at least one of a skin state determination facility and a
recommendation
engine, and enabling users, either all or a selected few, of the interface to
perform a skin
state determination within the interface. In the method and system, the skin
state
determination facility may comprise capturing images with a non-invasive
imaging
device comprising an illumination source comprising an incident light source
to direct
light upon skin, and a detector for detecting the degree of polarization of
light reflected
from the skin, and determining a skin state based on an aspect of the
polarization of the
reflected or re-emitted light. The method and system may further comprise
receiving
product and regimen recommendations from the recommendation engine based on
what
other users with similar skin states are using as well as data regarding
ingredients,
effectiveness, safety, and the like. The method and system may further
comprise
comparing skin states, products, regimens, and recommended products or
regimens with
peers within the social networking domain of the interface. Comparing may
comprise an
analysis of similarity based on the spectral analysis of the degree of
polarization of
reflected or re-emitted light from users' skin. In the method and system, the
interface
may comprise a regimen tracker. The regimen tracker may be populated using a
drag-
and-drop or click-to-add functionality. In the method and system, the
interface may
comprise a rating facility or a product information facility. The product
information
facility may enable a user to obtain product information by search. Search may
be a
search of product identifiers, product ratings, drag-and-drop items, images,
barcode
scans, skin states, and profiles.
[0026] In an aspect of the invention, a method and system for determining a
skin state may comprise obtaining the answers to a series of subjective
questions
regarding the skin, obtaining an objective skin analysis using a dermal
imaging device,
and combining the subjective and objective results algorithmically to obtain a
skin state.
[0027] In an aspect of the invention, a system and method for providing
recommendations for skin care based on a skin state and a skin care goal may
comprise
obtaining a skin state of an individual, categorizing the individual by skin
state, and
recommending products and regimens that are effective for other individuals of
the
Page 9 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
category in achieving the skin care goal. In the method and system, the system
may be
operable over a network. In the method and system, the skin state may be
determined
based on analysis of the degree of polarization of light reflected from the
skin of the
individual.
[0028] In an aspect of the invention, a method for tracking the effectiveness
of a skin care product or regimen may comprise obtaining a baseline skin state
assessment, recommending a monitoring interval based on at least one of the
skin care
goal, product, and regimen, obtaining a second skin state assessment,
comparing the
second assessment to the baseline assessment to determine progress towards a
skin care
goal, and optionally, optimizing the regimen or product in order to improve a
skin state.
In the method and system, the skin assessment may be based on analysis of the
degree of
polarization of light reflected from the skin of the individual.
[0029] In an aspect of the invention, a personalized skin condition analysis
system and related methods may comprise an imaging device, comprising an
illumination
source comprising an incident light source to direct light upon skin, and a
detector for
detecting the degree of polarization of light reflected from the skin, and a
user interface
for controlling the device. In the methods and system, the device may be
adapted to
interact with a physical interface to download image data to update a record
of at least
one of a practitioner, a spa, a salon, cosmetic sales, a cosmetics
manufacturer, a clinical
trials database, and a third party database. In the method and system, the
illumination
source may be positioned to direct light at a selected angle alpha. Varying
alpha may vary
the depth of the measurement of the layers in the skin. Each depth may have a
specific
angle which produces a full polarized reflection. In the method and system,
the incident
light source may be an unpolarized light source. The unpolarized light may be
white
light, multiple selected wavelengths, or a single wavelength. The method and
system
may further comprise a sensor for capturing an image of the reflected or re-
emitted light.
The method and system may further comprise an optical facility for detecting
reflected or
re-emitted light from the skin. The method and system may further comprise a
communication facility for transmitting the detected information. The method
and
system may further comprise a storage facility for storing information
collected by the
device.
Page 10 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[0030] In an aspect of the invention, a non-invasive imaging device may
comprise an illumination source comprising an incident light source to direct
light upon
skin; and a detector for detecting a characteristic of the light reflected
from the skin. In
the device, the illumination source may be positioned to direct light at a
selected angle
alpha. Varying alpha may vary the depth of the measurement of the layers in
the skin.
Each depth may have a specific angle which produces a full polarized
reflection. In the
device, the incident light source may be a polarized light source or
unpolarized light
source. The unpolarized light may be at least one of white light, light of a
single
wavelength, and light of multiple single wavelengths. The device may further
comprise a
sensor for capturing an image of the reflected or re-emitted light. The device
may further
comprise an optical facility for detecting reflected or re-emitted light from
the skin. The
device may further comprise a communication facility for transmitting the
detected
information. The device may further comprise a storage facility for storing
information
collected by the device. In the device, the reflected or re-emitted light may
be at least
one of polarized light and unpolarized light.
[0031] In an aspect of the invention, a method and system for determining a
skin state may comprise illuminating skin with an incident light source;
detecting a
characteristic of the light reflected from the skin; and determining a skin
state based on at
least one characteristic of the reflected or re-emitted light. In the method
and system, the
incident light may be directed at a selected angle alpha. Varying alpha may
vary the
depth of the measurement of the layers in the skin. Each depth may have a
specific angle
which produces a full polarized reflection. In the method and system, the
incident light
may be unpolarized or polarized light. The unpolarized light may be at least
one of white
light, light of a single wavelength, and light of multiple single wavelengths.
In the
method and system, the reflected or re-emitted light may be at least one of
polarized light
and unpolarized light. In the method and system, the characteristic may be at
least one of
light source, light intensity, wavelength of light, angle of light, electrical
and magnetic
properties of the light, and polarization state of the light. An aspect of the
polarization
may be at least one of an orientation, an amplitude, a phase, an angle, a
shape, a degree,
and an amount. In the method and system, determining may be done using an
algorithm.
The algorithm may involve artificial neural networks, non-linear regression,
genetic
Page 11 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
algorithms, fuzzy logic, or fractal and multi-fractal analysis. The method and
system
may further comprise filtering the reflected or re-emitted light to obtain
light of a
wavelength defined by the filter output. The analysis may be performed on the
filtered
image. In the method and system, determining may involve creating an image of
the
difference between reflected diffusion light and reflected polarized light. In
the method
and system, determining may involve comparing the aspect of the polarization
of the
reflected or re-emitted light to a calibration signal. In the method and
system,
determining may further comprise considering at least one of user input and a
visual
analysis.
[0032] In an aspect of the invention, a non-invasive imaging device may
comprise an illumination source comprising an incident light source to direct
light upon
an area of concern; and a detector for detecting a characteristic of the light
reflected from
the area of concern. In the device, the illumination source may be positioned
to direct
light at a selected angle alpha. Varying alpha may vary the depth of the
measurement of
the layers in the skin. Each depth may have a specific angle which produces a
full
polarized reflection. In the device, the incident light source may be a
polarized light
source or unpolarized light source. The unpolarized light may be at least one
of white
light, light of a single wavelength, and light of multiple single wavelengths.
The device
may further comprise a sensor for capturing an image of the reflected or re-
emitted light.
The device may further comprise an optical facility for detecting reflected or
re-emitted
light from the skin. The device may further comprise a communication facility
for
transmitting the detected information. The device may further comprise a
storage facility
for storing information collected by the device. In the device, the reflected
or re-emitted
light may be at least one of polarized light and unpolarized light.
[0033] In an aspect the invention, a system and method may be used to
determine healthy and melanocytic skin. The first, reflected spectrum and/or
emission
spectrum from sample which is skin malformation (SM), subtract reflected
spectrum
from normal healthy skin (SN). The second, from obtained resulting spectral
plots (SM -
SN) subtract reflected spectrum from adequate comparing screen, which
represents
spectral plot of the light source (SO). In that path appeared pure
characteristics of change
generated by skin. For differentiation between melanoma, other malignant or
benign
Page 12 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
nevus and healthy skin can be used data on maxima, minima and zero positions,
in
wavelength scale and data on maxima and minima intensities.
[0034] In an aspect of the invention, a system and method may comprise
capturing an image of a material illuminated with incident non-angled white
light and
angled white light, generating a normalized red and blue color channel
histogram for each
image, correlating the normalized red and blue color channel histograms to a
wavelength
scale to obtain red and blue color channel spectral plots, and convoluting the
spectral
plots by subtracting the spectral plot for angled light from the spectral plot
for non-angled
light for each color channel to generate red and blue normalized, composite
color channel
spectral plots, and subtracting the normalized, composite blue channel
spectral plot from
the normalized, composite red channel spectral plot to generate a spectral
signature for
the material. In the system and method, the illumination source may be
positioned to
direct light at a selected angle alpha. Varying alpha varies the depth of the
measurement
in the material. In the system and method, the unit scale on the spectral
signature may be
a difference of wavelength. In the system and method, the material is
inorganic and/ or
organic matter. In the system and method, the spectral signature may be
analyzed for at
least one of number of peaks and troughs, amplitude and shape of peaks and
intermediate
structures and patterns. In the system and method, the spectral signature may
be analyzed
for metal composition, identification, purity, and strength. In the system and
method, the
spectral signature may be analyzed for water quality, composition, and purity.
In the
system and method, elements of the spectral signature may be tagged and
tracked over
time in order to track changes in the characteristics of the material. In the
system and
method, the spectral signature may be analyzed to measure, track or monitor a
skin state.
In the system and method, the spectral signature may be useful for the
counterfeit
analysis of money. In the system and method, the spectral signature may be
analyzed for
at least one of sweat gland activity and anti-perspirant effectiveness. In the
system and
method, the spectral signature may be analyzed for Mad Cow disease. In the
system, the
spectral signature may be analyzed for food, all epidermal diseases, melanoma
and skin
cancers, rheumatoid diseases, and all diseases that show on the skin. In the
system and
method, the spectral signature may be useful for monitoring post-operative
cosmetic
concerns. In the system and method, the spectral signature may be useful for
predicting
Page 13 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
and monitoring secretion from the mammary glands of lactating women. In the
system
and method, the spectral signature may be fed into a recommendation engine to
provide
feedback and modifications to aspects of a regimen. In the system and method,
the
wavelength position of ideal blue in Maxwell's color triangle is aligned with
the
wavelength position of ideal red in Maxwell's color triangle when convoluting
the
composite spectral plots to obtain the spectral signature.
[0035] A method and a system are disclosed for determining skin
characteristics and cosmetic features. A minimal error output is generated. In
accordance
with exemplary embodiments of the present invention, according to a first
aspect of the
present invention, a method for determining skin characteristics and cosmetic
features
using color analysis may include a step of analyzing color of skin images in a
pixel by
pixel manner in a Red Green Blue (RGB) color system for an acquired digital
image. The
step of analyzing color of skin images in a pixel by pixel manner in a RGB
color system
for an acquired digital image may include analyzing a picture of a part of a
person's skin
by generating a table of most frequent colors appearing in the picture.
[0036] According to the first aspect, a method for determining skin
characteristics and cosmetic features using color analysis includes a step of
generating a
sample of most frequent standard RGB (sRGB) colors responsive to analyzing
color of
skin images in a pixel by pixel manner in the RGB color system for the
acquired digital
image after converting colors obtained in device dependent RGB color system
into device
independent standard RGB color system (sRGB). The step of generating a sample
of
most frequent sRGB colors responsive to analyzing color of skin images in the
sRGB
color system for the acquired digital image may include preserving a plurality
of sRGB
color values.
[0037] In this embodiment of the invention, the sRGB color system may be
used for image analysis. Determination of other skin characteristics,
melanoma, skin
related tumors and skin related disorders require image analysis based on
other color
systems such as YIQ, YCbCr, L*a*b*, L*u*v* and HSL/HSV. The enhancement of the
current algorithm may include at least one of these color systems and
its/their correlation
with presented sRGB analysis.
Page 14 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[0038] According to the first aspect, a method for determining skin
characteristics and cosmetic features using color analysis includes a step of
modeling the
R, G and B component color distribution with Gaussian probabilistic
distribution with
estimated parameters (expected value and standard deviation) on the generated
sRGB
color sample for the acquired digital image further including approximating
colors on the
generated sRGB color samples by a Gaussian normal distribution. In accordance
with an
exemplary embodiment of the present invention the step of approximating colors
on the
generated sRGB color samples by a Gaussian normal distribution comprises
approximating colors on the generated sRGB color samples by a superposition of
a
plurality of Gaussian normal distributions.
[0039] According to the first aspect, a method for determining skin
characteristics and cosmetic features using color analysis includes a step of
generating a
phototype of the skin through a decision tree unit responsive to the estimated
distribution
model parameters colors. The phototype of the skin may be generated according
to a
corrected Fitzpatrick classification. In accordance with an exemplary
embodiment of the
present invention, the step of generating phototype of the skin according to
corrected
Fitzpatrick classification includes generating phototype of the skin according
to a skin
type scale which ranges from very fair skin to very dark skin.
[0040] According to a second aspect, a system for skin phototype
determination using photograph analysis may be disclosed. The system may
include an
image capturing device for capturing digital images of a skin. The image
capturing device
may include a digital camera unit.
[0041] According to the second aspect, the system for skin phototype
determination using photograph analysis may include an analyzer coupled to the
image
capturing device for performing a pixel by pixel analysis of a picture of a
part of a
person's skin. The analyzer may include a quantization device for generating a
look-up
table of most frequent colors appearing on the picture of the part of the
person's skin.
[0042] According to the second aspect, the system for skin phototype
determination using photograph analysis may include a sampling device coupled
to the
image capturing device for generating standard Red Green Blue (sRGB) color
samples
for the captured digital image of the skin.
Page 15 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[0043] According to the second aspect, the system for skin phototype
determination using photograph analysis may include an approximating device
coupled to
the sampling device for approximating the color distribution parameters on the
generated
sRGB color samples using the estimates of expected value and standard
deviation for the
captured digital image of the skin. The approximating device may include at
least one
Gaussian normal distribution unit.
[0044] According to the second aspect, the system for skin phototype
determination using photograph analysis may include a decision tree unit
coupled to the
approximating device for generating a phototype of the skin using Red and Blue
components of the approximated colors. The decision tree unit may include a
Fitzpatrick
scaling unit for categorizing a skin phototype in accordance with a skin type
scale which
ranges from very fair skin to very dark skin.
[0045] According to the second aspect, an exemplary embodiment of the
present invention discloses a scaled Gaussian normal distribution unit for
approximating
colors on the generated sRGB color samples using estimates of expected value
and
standard deviation for the captured digital image of the skin.
[0046] According to the second aspect of the present invention, the system
for skin phototype determination using photograph analysis may include a
subsystem for
determination of cosmetic features for a human element and a veterinary
element. The
cosmetic features may further include features pertaining to hair, nail and
skin.
[0047] In another aspect the system may include a sampling device for
generating standard Red Green Blue color samples of the captured digital image
of the
skin, the generated samples of standard Red Green Blue are in the range of
values
between 0 and 255 and they are preserved for further processing.
[0048] In another aspect the system may include an approximating device
coupled to the sampling device for approximating the color distribution
parameters on the
generated sRGB color samples in the range of values between 0 and 255 by
Gaussian
normal distribution using the estimates of expected value and standard
deviation for the
captured digital image of the skin.
[0049] In another aspect the system may further include a decision tree unit
coupled to the approximating device for generating a phototype of the skin
using standard
Page 16 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
Red and Blue components of the approximated colors, the decision tree unit
with an
algorithm equates estimates of expected values and standard deviation for the
captured
image of the skin to the Fitzpatrick notation of skin analysis for
determination of skin
phototype.
[0050] In another aspect the system may automatically adjust lighting
intensity and wavelengths and angles in order to assess various factors of the
skin.
[0051] In yet another aspect of the system skin phototype may be determined
using photograph analysis for use in cosmetics and surgical industry.
[0052] In an aspect of the invention, a skin care device may include an
electromagnetic radiation source capable of directing incident electromagnetic
radiation
to a location on the skin of a user, a radiation detector for measuring
various parameters
of radiation re-emitted from the location, and a skin condition analysis
module coupled
to the detector, the analysis module capable of generating a skin condition
assessment in
real-time, based partly on at least one of RGB analysis and diffused
reflectance analysis
of the radiation parameters. In the device, incident electromagnetic radiation
may include
radiation in at least one of the visible, near-infrared, and near-ultraviolet
spectrum. The
incident radiation may be white light. In the device, the radiation parameters
include at
least the degree of polarization of the re-emitted radiation. In the device,
the radiation
source may be a set of light emitting diodes. In the device, the skin
condition assessment
may be also partly based on analysis of a photographic image of a skin region
surrounding the location. In the device, the device may be a miniature device.
Miniature
may mean that no dimension of the detector exceeds six inches. The device may
further
include a memory module for storing the skin condition assessment. The device
may
further include a user interface. The device may further include a display
surface. The
skin assessment data of locations may be overlaid on an image of a larger skin
region and
displayed on the display surface. The device may further include an access
restriction
module used for restricting access to authorized users only. The access
restriction module
may be based on biometric access control. The device may be capable of
generating
alerts about abnormal skin conditions in real-time. The user interface may be
operated
using voice and/or eye movement commands. The device may further include a
skin care
regimen recommendation module that generates a displayable skin care regimen
Page 17 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
recommendation. The skin care regimen recommendation may be based at least
partly on
determination of a skin profile of the user and use of skin care regimen
recommendations
of persons with a similar profile. The skin care regimen recommendation module
may be
linked to a product database. The product database may include products
available in a
point-of-sale location. The availability of a specific product recommended by
the skin
care regimen recommendation module may be indicated by an audio-visual signal.
The
device may further include a skin care regimen effectiveness module that
generates a
displayable skin care regimen effectiveness report. The device may further
include a
communication module for communicating with a remote computer. The
communication
may occur wirelessly. The communication may occur over an internet. The remote
computer may be operable by a physician. The device may be wand-shaped. The
device
may be wearable by the user.
[0053] In an aspect of the invention, the device an electromagnetic radiation
source capable of directing incident electromagnetic radiation to a location
on the skin of
a user, a detector for measuring various parameters of radiation re-emitted
from the
location,a skin condition analysis module coupled to the detector, the
analysis module
capable of generating a skin condition assessment in real-time, based partly
on at least
one of RGB analysis and diffused reflectance analysis of the radiation
parameters, and a
display panel for reflecting the image of the user. In the device, the display
panel may be
touch-sensitive such that touching the location in a skin region image
displayed in the
display panel triggers display of a magnified image of the location. The skin
care device
may further include a camera. The camera may be integral with the display
panel. The
camera may be wirelessly linked to the display panel. In the device, the
display panel
may be a mirror. In the device, a stored image of the user may be used to
automatically
identify the person. The device may further include a user interface for
controlling the
skin care device. The user interface may be operated using voice and/or eye
movement
commands. The device may further include a skin care regimen recommendation
module
capable of generating a displayable skin care regimen recommendation. The skin
care
regimen recommendation may be based at least partly on determination of a skin
profile
of the user and use of skin care regimen recommendations of persons with a
similar
Page 18 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
profile. The device may further include a skin care regimen effectiveness
module
capable of generating a displayable skin care regimen effectiveness report.
[0054] In an aspect of the invention, a system and method for moving
information objects available on a website to a receptacle to communicate with
a plurality
of people in a controlled access community network may include enabling
movement of
a plurality of information objects from a predetermined website to a web based
network
responsive to a regimen of a person, a routine of a person, a purpose of use
of an
information object of the plurality of information objects and a degree of
affinity of a first
person towards a second person, initiating at least one customized action from
the actions
including a drop down movement; a drag and drop movement for populating data;
and a
pop-up movement in a Graphical User Interface (GUI) responsive to enabling
movement
of a plurality of information objects from a predetermined healthcare website,
and
enabling transportation of the plurality of information objects across a
plurality of
websites. In the system and method, the plurality of information objects may
pertain to a
questionnaire on at least one of a human skin condition, product information,
an article, a
blog posting, an image, a video, an individual message, a forum posting, and a
veterinary
skin condition. In the system and method, the plurality of information objects
pertains to
a questionnaire on human cosmetic parameters and veterinary cosmetic
parameters. The
questionnaire on human cosmetic parameters and veterinary cosmetic parameters
may
include questions on at least one of a human nail and a veterinary nail. The
questionnaire
on human cosmetic parameters and veterinary cosmetic parameters may include
questions on at least one of a human hair and a veterinary hair. In the system
and
method, the purpose of use of the information object may pertain to
controlling at least
one of cleansing, protection, repair, moisturizing, elasticity, firmness,
glow, luminosity,
anti-inflammatory properties, anti-itch properties, anti-wrinkle properties,
firming,
exfoliating, anti-redness properties, oil controlling, anti-aging properties
and shine of a
human skin. In the system and method, the degree of affinity of a first person
towards a
second person comprises at least one of a relationship of friendship between
the first
person and the second person; a genetic similarity between the first person
and the second
person; a similarity of lifestyle between the first person and the second
person; a climatic
similarity between a first residential environment and a second residential
environment;
Page 19 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
and a skin type similarity between the first person and the second person. In
the system
and method, the step of enabling transportation of the plurality of
information objects
across a plurality of websites may include a sub-step of dragging an item of
user interest
off a website of the plurality of websites in a predetermined format and
transferring
through an electronic signal to affiliates of a user accessing the website.
The affiliates of
the user may be friends and relatives of the user or associated experts. In
the system and
method, the step of enabling movement of a plurality of information objects
from a
predetermined website to a web based network may include a sub-step of
enabling drop
down menus on the Graphical User Interface (GUI) responsive to a plurality of
end user
convenience and requirement parameters. In the system and device, the
plurality of
people in a web based network includes a plurality of people in an online
friendship
network. In the system and device, the plurality of people in a web based
network
includes a plurality of people in an online social network.
[0055] In an aspect of the invention, an interface including a social
networking domain and at least one skin health assessment and recommendation
unit for
enabling users of the interface to perform a skin health assessment within the
interface
and to receive product and regimen recommendations from a recommendation
engine
based on a predetermined usage of health assessment and maintenance data may
include
a regimen tracker populated using a drag and drop facility, a rating unit for
rating a
plurality of healthcare facilities, and a product information unit for
enabling a user to
obtain product information by conducting a web based search of a plurality of
web based
drag and drop products, web based images and bar code scans. In the interface,
the
regimen tracker includes a diet tracking unit. In the interface, the plurality
of healthcare
facilities comprises at least one of skin cleansing, skin protection, skin
moisture control,
skin repair, skin elasticity, skin luminosity, skin firmness, skin wrinkles,
pore size on
skin, spots on skin , glow on skin, hair color, hair type, age and life stage
further
including marriage, pregnancy, dating and social life. In the interface, the
product
information comprises at least one of a product type, a product function, a
product
format, a product appropriateness level, a regimen information, product
articles, product
blogs, product safety, product toxicity, a product effectiveness index, a
product cost
information, and a product timeliness information. In the interface, the
interface is a
Page 20 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
multiple language and customized interface for: web based applications; mobile
phone
applications; touch screen applications; and personal digital assistant
applications. In the
interface, the interface is seamlessly coupled with a dermal imaging device
for
customized web based access, control and maintenance of spectral analysis of
image data
acquired from a degree of polarization of reflections and re-emission of
incident light
from skin structures. The degree of polarization of reflections and/or re-
emissions of
incident light from skin structures is derived from at least one of a Red
Green Blue
(RGB) color analysis of a plurality of digital images; and an analysis from
spectroscopic
data image analysis.
[0056] In an aspect of the invention, a system and method for determining a
health state may include obtaining the answers to a series of subjective
questions
regarding health conditions, obtaining an objective health assessment report
through a
dermal imaging device, and generating a combination of answers to the series
of
subjective questions and the objective health assessment report to thereby
generate a
health state output and a real skin type output. In the system and method, a
real skin type
output is generated based on biophysical properties generated by at least one
of a person
seeking skin health monitoring, a spa, and a cosmetic advisor. In the system
and method,
the objective health assessment report may include an objective skin health
assessment
report on at least one of systemic hydration, skin hydration, skin firmness,
skin wrinkles,
pore size on skin, spots on skin, glow on skin, melanocyte, melanin,
hemoglobin,
porphyrin, keratin, carotene, collagen, elastin, sebum, sebaceous gland
activity, sweat
pore, sebaceous pore, moisture level, elasticity, luminosity, firmness, fine
line, wrinkle
count, pore size, percent of open pores, skin elasticity, skin tension line,
spots, viscosity,
epidermal, dermal sebum levels, skin color, psoriasis, allergy, red area,
general skin
disorder, infection, tumor, sunburn, rash, scratch, pimple, acne, insect bite,
itch, bleeding,
injury, inflammation, photodamage, pigmentation, tone, tattoo, percent burn,
burn
classification, mole, aspect of a skin lesion, melanoma, dermally observed
disorder,
cutaneous lesion, cellulite, boil, blistering disease, congenital dermal
syndrome,
cutaneous mycoses, melasma, vascular condition, rosacea, spider vein, texture,
skin ulcer,
wound healing, post-operative tracking, melanocytic lesion, nonmelanocytic
lesion, basal
cell carcinoma, seborrhoic keratosis, sebum hair color, hair type, nail
condition, and age
Page 21 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
and life stage further including marriage, pregnancy, dating and social life.
In the system
and method, the objective health assessment report is sent to an end user
through at least
one of email, SMS, MMS, mobile phone, a graphical user interface (GUI) of an
internet
connected device, and a touch screen enabled personal digital assistant. The
system and
method may further include obtaining health assessment and maintenance data
from a
physiologically polarized light data. The step of obtaining health assessment
and
maintenance data from a physiologically polarized light data comprises
obtaining health
assessment and maintenance data from a Red Green Blue (RGB) color analysis
device,
wherein the data comprise at least one of a white light data, a blue light
data, and an ultra
violet light data. The step may further comprise obtaining at least one of the
white light
data, the blue light data, and the ultra violet light data by reading and
recording
conditions of at least one of the dermis and epidermis. Obtaining health
assessment and
maintenance data from a physiologically polarized light data comprises
obtaining data
pertaining to age, geography and demography for a person subjected to health
monitoring.
[0057] In an aspect of the invention, a web-enabled health tracking method
and system may include a camera comprising a photo guide unit for generating
notes for
each photograph captured, an interface coupled between the camera and a web-
enabled
computing system for uploading the photograph captured by the camera, a
graphical user
interface unit included in the web-enabled computing system for generating a
frequently
asked questionnaire unit further comprising a self answer guide module, a
scoring
module coupled to the frequently asked questionnaire unit, a comparison module
coupled
to the scoring module for comparing: a color parameter; a symmetry parameter;
and a
border parameter, an automation unit coupled to the graphical user interface
for enabling
a time-based synchronization of the frequently asked questionnaire unit, the
scoring
module, and the comparison module, and a learning unit coupled to the
automation unit
for activating: a user training module, an article module coupled to the user
training
module, a blogging unit coupled to the user training module and the article
module, and
a report unit including an email unit for emailing health related information.
In the
system and method, the camera comprises a tracking unit for tracking at least
one of skin
spots over time, laser treatment effectiveness, cellulite content in skin,
condition of veins
Page 22 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
and capillaries, botox treatment effectiveness, anti-aging treatment
effectiveness, anti-
acne treatment effectiveness, and a pictorial history of skin to be given to
the doctor. The
skin spots over time include at least one of blemishes, scars, rashes,
lesions, and moles.
In the system and method, the web-enabled computing system for uploading the
photograph captured by the camera further includes a walkthrough module for
walking
through features of a skin health record of a first time user of the system, a
personal skin
photo album for reviewing pictorial history of a regular user of the system,
and a product
quality menu for tracking product expiration dates. In the system and method,
the
interface for uploading the photograph further includes a reminder unit for
next photo for
a regular user of the system; and a cosmetic status unit coupled to the
reminder unit for
displaying a current usage of a cosmetic for the regular user of the system.
The current
usage comprises a usage of at least one of a moisturizer, an antiseptic, a
toner, a laser,
and a botox. The system and method may further include a photo review unit for
date
based reviewing of at least one of a condition of a predetermined body part, a
current
usage status of a cosmetic, and a recommended usage list of cosmetics. In the
system and
method, the report unit may further include a secure transmission unit for
sending a
health assessment report to a medical practitioner, an affinity unit for
discussing health
assessment data with a friend, and a printing unit for printing health
assessment data.
[0058] In an aspect of the invention, a mobile device-based health assessment
system and method may include a photograph capturing device for capturing a
skin
image of a mobile device user, a transmission unit coupled with the photograph
capturing
device for uploading the captured skin image to a network location, a global
positioning
device coupled to the photograph capturing device for determining a location
of the
photograph capturing device, and a weather estimation device coupled to the
photograph
capturing device to determine a weather condition at a location of the mobile
device user
to thereby obtain a remote diagnosis report. In the system and method, the
photograph
capturing device further comprises at least one of a skin photograph
assessment unit, a
nail photograph assessment unit, and a hair photograph assessment unit. In the
system
and method, the global positioning device comprises a location tracker for
answering user
raised questions pertaining to geographical positioning of the user. In the
system and
method, the location tracker includes a database pertaining to weather
intensive
Page 23 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
cosmetics. The system and method may further include a phone number tracker
for
enabling a mobile device user to contact health assessment and cosmetic
outlets.
[0059] In an aspect of the invention, a system and method for estimation of
skin type and skin features to create a unique spectral signature may include
convoluting
data from a first image captured in incident diffuse white light, wherein the
data relate to
reflected and/or re-emitted polarized or white light, convoluting data from a
second
image captured in incident polarized light, wherein the data relate to
reflected and/or re-
emitted polarized light, comparing extreme positions of at least two unique
convolutions
generated by convoluting data from the first image and the second image, and
determining a distance between minimum and maximum intensity positions in
convoluted red minus blue spectral plots from the at least two unique
convolutions for
generating a numerical skin type output. In the system and method, the
physiological
white light comprises three spectral intervals including a width less than 100
nanometer.
The three spectral intervals pertain to red, green, and blue (RGB) colors. The
three
spectral intervals provide a natural white light sensation to a human eye. In
the system
and method, the step of comparing extreme positions of at least two unique
convolutions
comprises comparing a component (R-B)(W-P) for the reflected and/or re-emitted
polarized light, and a component (R-B)W for the white light. The two unique
convolutions in white light and polarized light further include a White Red
component
(WR), a White Blue component (WB), a reflected and/or re-emitted Polarized
Blue
component (PB) and a reflected and/or re-emitted Polarized Red component (PR).
The
two unique convolutions are based on a numerical value difference correlating
to medical
standards. The system and method may further include a spectral convolution
scheme
wherein multiple combinations of subtraction of blue spectrum from red, in
white light
and polarized white light are determined, wherein the spectral interval is
expressed in a
wavelength scale interval of 100 nanometers to 300 nanometers.
[0060] In an aspect of the invention, a system and method for creating a
unique spectral signature of skin features may include a RGB (Red Green Blue)
color
channel spectral plot generated from digital images including single
wavelength light
matter interaction thereby generating skin type characterization output, skin
moisture
conductivity and skin elasticity in numerical and descriptive standards. In
the system and
Page 24 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
method, the RGB (Red Green Blue) color channel spectral plots generated from
digital
images include multi-wavelength light matter interaction.
[0061] In an aspect of the invention, a system and method to track and store
movement parameters of an imaging device moving over a subject area may
include the
steps of capturing an image of the subject area at a plurality of locations,
identifying a
direction of movement of the imaging device using an image processing
technique for at
least one captured frame, recognizing the direction of movement of the imaging
device
by comparing each frame with at least three distinct features captured to
thereby
triangulate a location of the imaging device, and comparing data of the
captured image
with a predetermined image database to store the image of the subject area and
to store
placement parameters of the imaging device. In the system and method, the step
of
capturing the image of the subject area at a plurality of locations comprises
a sub step of
capturing a continuous video image of the subject area. In the system and
method, the
step of capturing the image of the subject area at a plurality of locations
comprises a sub
step of capturing a frame by frame sequence of images of the subject area. In
the system
and method, the step of identifying a direction of movement of the imaging
device using
an image processing technique comprises a sub-step of a frame by frame
comparison of
the captured image to identify movement parameters of the imaging device. In
the
system and method, the step of recognizing the direction of movement of the
imaging
device by comparing each frame with at least three distinct features captured
to
triangulate a location of the imaging device comprises a sub-step of capturing
a direction
of movement of the imaging device by comparing three or more distinct
positions across
different frames.
[0062] In an aspect of the invention, an automated location tracking and data
storage method and system for an imaging device may include an image capturing
unit, a
positioning unit coupled to the image capturing unit for positioning the
imaging device
on a subject area, and an image processing unit for enabling a frame by frame
comparison
of the captured image and for enabling the imaging device to capture three or
more
distinct points to triangulate a location of the imaging device to identify a
direction of
movement of the imaging device. In the system and method, the image capturing
unit
comprises a digital camera. In the system and method, the image capturing unit
Page 25 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
comprises at least one of a mobile device and a Personal Digital Assistant
(PDA). In the
system and method, the image processing unit comprises a comparison unit for
comparing positions of three or more distinct points across different frames
to capture
direction of movement of the imaging device. The system and method may further
include a sub-system for measuring lateral motion of the image capturing unit
from a
predetermined point to a new location on the subject area.
[0063] In an aspect of the invention, a system and method for determining a
surgical excision margin may include illuminating a melanocytic lesion skin
with an
incident light source, detecting a characteristic of the light reflected
and/or re-emitted
from the melanocytic lesion, and determining a border between the melanocytic
lesion
and surrounding healthy tissue based on at least one characteristic of the
reflected and/or
re-emitted light. In the system and method, the incident light is directed at
a selected
angle alpha. In the system and method, varying alpha varies the depth of the
measurement of the layers in the melanocytic lesion. Each depth has a specific
angle
which produces a full polarized reflection. In the system and method, the
incident light is
unpolarized light. The unpolarized light is at least one of white light, light
of a single
wavelength, and light of multiple single wavelengths. In the system and
method, the
incident light is polarized light. In the system and method, the reflected
and/or re-emitted
light is at least one of polarized light and unpolarized light. In the system
and method,
the characteristic is at least one of light source, light intensity,
wavelength of light, angle
of light, electrical and magnetic properties of the light, and polarization
state of the light.
An aspect of the polarization is at least one of an orientation, an amplitude,
a phase, an
angle, a shape, a degree, and an amount. In the system and method, determining
is done
using an algorithm. The algorithm involves at least one of artificial neural
networks,
fuzzy logic, fractal and multi-fractal analysis, non-linear regression, a
genetic algorithm,
white light analysis and RGB color analysis. The system and method may further
include
filtering the reflected and/or re-emitted light to obtain light of a
wavelength defined by
the filter output. Algorithmic analysis is performed on the filtered image. In
the system
and method, determining involves creating an image of the difference between
reflected
diffusion light and reflected polarized light. In the system and method,
determining
involves comparing the aspect of the polarization of the reflected and/or re-
emitted light
Page 26 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
to a calibration signal. In the system and method, determining further
comprises
considering at least one of user input and a visual analysis.
[0064] These and other systems, methods, objects, features, and advantages of
the present invention will be apparent to those skilled in the art from the
following
detailed description of the preferred embodiment and the drawings. All
documents
mentioned herein are hereby incorporated in their entirety by reference.
BRIEF DESCRIPTION OF THE FIGURES
The invention and the following detailed description of certain embodiments
thereof may
be understood by reference to the following figures:
[0065] Fig. 1 depicts a skin care system for skin health analysis and
monitoring, and skin care assessment and recommendation.
[0066] Fig. 2 depicts a mechanism for light polarization by a skin structure.
[0067] Fig. 3 depicts a process for skin care examination.
[0068] Fig. 4A & B depict a front and back view of a dermal imaging device.
[0069] Fig. 5 depicts a skin health monitoring page of a skin care system.
[0070] Fig. 6 depicts an interactive modeling tool of a skin care system.
[0071] Fig. 7 depicts a recommendations page of a skin care system.
[0072] Fig. 8 depicts a user interface of a skin care system.
[0073] Fig. 9 depicts a welcome page of a skin care system.
[0074] Fig. 10 depicts a questionnaire page of a skin care system.
[0075] Fig. 11 depicts a skin image capture page of a skin care system.
[0076] Fig. 12 depicts a results page with bar graphs of a skin care system.
[0077] Fig. 13 depicts a results page with line graphs of a skin care system.
[0078] Fig. 14 depicts a summary screen of a skin care system.
[0079] Fig. 15 depicts an elasticity summary screen of a skin care system.
[0080] Fig. 16 depicts a summary screen of a skin care system.
[0081] Fig. 17 depicts an elasticity summary screen of a skin care system.
[0082] Fig. 18 depicts a map of a user interface for a skin care system.
[0083] Fig. 19 depicts a review page of a skin care system.
Page 27 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[0084] Fig. 20 depicts a review page of a skin care system.
[0085] Fig. 21 depicts a My Experience page of a skin care system.
[0086] Fig. 22 depicts a What Works page of a skin care system.
[0087] Fig. 23 depicts an Info For Me page of a skin care system.
[0088] Fig. 24 depicts an example of a skin care shelf portion of a user
interface of a skin care system.
[0089] Fig. 25 depicts an example of a skin care shelf portion of a user
interface of a skin care system.
[0090] Fig. 26 depicts a user interface of a skin care system.
[0091] Fig. 27 depicts a registration page of a skin care system.
[0092] Fig. 28 depicts a recommendation page of a skin care system.
[0093] Fig. 29 depicts a mobile content map for a mobile user interface of a
skin care system.
[0094] Fig. 30 depicts a How Good Is This Product message flow.
[0095] Fig. 31 depicts a What Should I Look For? message flow
[0096] Fig. 32 depicts a Suncheck message flow.
[0097] Fig. 33 depicts an Alert message flow.
[0098] Fig. 34 depicts an Options message flow.
[0099] Fig. 35 depicts an algorithm and method for analyzing materials.
[00100] Fig. 36 depicts the reflection and capture of white light and
reflected
polarized light from a specimen based on varying angles.
[00101] Figs. 37A&B depict color coordinate systems that can be used in
digital image processing.
[00102] Fig. 38 depicts a histogram of color density.
[00103] Fig. 39 depicts a normalized color channel histogram correlated to
wavelength scale.
[00104] Figs. 40A&B depicts overlaid, normalized color channel histograms.
[00105] Figs. 41A&B depicts a convolution of individual color channel
histograms.
[00106] Fig. 42 depicts the combination of the two convolutions of the two
color channel histograms.
Page 28 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00107] Fig. 43 depicts a mathematical modeling of a portion of Maxwell's
color triangle.
[00108] Figs. 44A & B depict the resulting spectral signatures for light and
dark skin.
[00109] Figs. 45A - C depict the resulting spectral signatures for pure and
alloy
metals.
[00110] Figs. 46A & B depict the resulting spectral signatures for different
types of water.
[00111] Fig. 47 depicts a block diagram of a skin care device embodiment.
[00112] Fig. 48 depicts a wand-shaped skin care device embodiment.
[00113] Fig. 49 depicts a vertical display panel including skin care device.
[00114] Fig. 50 depicts an embodiment of a wearable skin care device.
[00115] Fig. 51 depicts positive and negative intensities on a waveform as a
function of emission and absorption of specific wavelengths within skin
tissue.
[00116] Fig. 52 depicts the comparison between spectral signatures of healthy
skin and malignant skin around a reference wavelength.
[00117] Fig. 53 depicts malignant pigmented skin in white and physiologically
polarized white light.
[00118] Fig. 54 depicts the comparison of convolutions between healthy,
benign and malignant skin lesions.
[00119] Fig. 55 depicts a system for tracking and targeting an image.
[00120] Fig. 56 depicts a system for determining an excision margin.
[00121] Fig. 57 depicts a system for determining an excision margin.
[00122] Fig. 58 is a flowchart illustrating a process for RGB color analysis.
[00123] Fig. 59 is a diagram depicting a pixel view of an acquired digital
image of a sample of person's skin.
[00124] Fig. 60 is a diagram depicting a pixel view of the acquired digital
image of a sample of person's skin after quantization.
[00125] Fig. 61 is a diagram depicting a Histogram / Distribution of standard
R, G and B colors on one of the taken photographs of a patient whose skin
phototype is
classified as type III by Fitzpatrick, and their Gaussian normal approximation
/ hull.
Page 29 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00126] Fig. 62 is a diagram depicting a Histogram / Distribution of standard
R, G and B colors on one of the patient's photographs whose skin phototype is
classified
as type VI by Fitzpatrick, and their Gaussian normal approximation / hull.
[00127] Fig. 63 is a flowchart illustrating an algorithm for determining the
skin
phototype according to the estimated values of mathematical expectation for R
and B
colors in a standard RGB color system.
[00128] Fig. 64 depicts an embodiment of a friend toolbar.
[00129] Fig. 65 depicts the auto-scroll feature of the friend toolbar.
[00130] Fig. 66 depicts the drag-and-drop share functionality of the friend
toolbar.
[00131] Fig. 67 depicts the drag-and-drop share functionality of the friend
toolbar.
[00132] Fig. 68 depicts sharing skin data as a data object with friends.
[00133] Fig. 69 depicts posting skin care data as a data object on a blog or
forum where users may discuss the data.
[00134] Fig. 70 depicts sharing skin data as a data object where the data
object
becomes part of the content that a user may wish to discuss.
DETAILED DESCRIPTION
[00135] Provided herein may be methods, systems, and a device for dermal and
non-dermal imaging. Throughout this disclosure the phrase "such as" means
"such as
and without limitation". Throughout this disclosure the phrase "for example"
means "for
example and without limitation". Throughout this disclosure the phrase "in an
example"
means "in an example and without limitation". Throughout this disclosure, the
term
"product" refers to any medical, non-medical, cosmetic, skin, hair, or nail
care product.
Generally, any and all examples may be provided for the purpose of
illustration and not
limitation.
[00136] Real-time analysis of digitally captured skin-related and other
information may facilitate real-time skin condition assessment, real-time skin
regimen
recommendation, and real-time evaluation of the effectiveness of a selected
skin regimen.
Real-time analysis of digitally captured data may be performed by using a skin
care
Page 30 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
device embodying the principles of the invention disclosed herein. A skin care
device
embodying the principles of the invention may include, for example, an
electromagnetic
radiation source capable of directing incident electromagnetic radiation, a
radiation
detector for measuring various parameters of the re-emitted radiation, and a
skin
condition analysis module capable of generating a skin condition assessment in
real-
time.
[00137] The skin condition assessment may be cosmetic and/or medical in
nature. By way of example, and in no way limiting the scope of the invention,
the skin
condition assessment may include any one of an acne condition assessment, a
pore
condition assessment, a wrinkle condition assessment, a skin elasticity
assessment, a skin
oiliness assessment, a skin moisture assessment, a skin luminosity assessment,
a skin
sebum assessment, a skin redness assessment, a skin inflammation assessment, a
skin
texture assessment, a skin color assessment or any combination of the listed
assessments.
For example, the pore condition assessment can help in determining whether the
pores
are clean, open and of optimal health.
[00138] Skin-condition data may be acquired, for example, by directing
incident electromagnetic radiation to a location, such as a pin-point
location, on the skin
of a person and detecting the re-emitted radiation from the location by using
a radiation
detector. The effectiveness of generating high-quality, real-time skin
condition
assessments may be enhanced in some embodiments by using a skin condition
analysis
module that bases its analysis at least partly on diffused reflectance
spectroscopy. The
quality of real-time skin condition assessments may be further enhanced in
other
embodiments by using white light as the incident radiation and by detecting
the red-
green-blue components of the re-emitted light.
[00139] Referring to Fig. 1, a system for skin health analysis, monitoring,
and
recommendation may comprise host hardware 108, such as an imaging device 108,
for
capturing biophysical skin properties such as in a skin health test 160,
performing pre-
diagnosis 162, and performing remote monitoring 164 using a light source 127;
a user
interface 102 interfacing with the host hardware 108, an online platform 120,
or a mobile
platform 124 for capturing demographic information, additional anecdotal
information on
skin health, current skin care regimen 118, rankings and ratings 138 of
current skin care
Page 31 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
products and regimen, populating a skin care shelf 114, and accessing a skin
cycle
monitor 140, health and/or wellness information 142, games 148, a gift guide
144, a
wishlist 119, a Daily Report 134, simulation tools 132, a type determination
engine 130, a
shopping cart 113, and the like; a host system 104 for processing and
analyzing captured
information such as by employing an algorithm 150, obtaining an expert
consultation
128, data integration 152, and analysis tools/API's 154 to define a skin state
158; other
inputs 112 to the host system 104, which may be subject to ranking/ rating
feedback 138,
for providing additional granularity in identifying, monitoring, and adjusting
a skin state
158, such as a wearable monitor 182, a mobile communications device 184, a
social
network 188, product information 190, wellness information 192, a plug-in (web
capture)
194, a barcode scan 198, conventional information/ questionnaire answers 101,
a
query/search 103, third part experts 105, third party hardware 109, third part
service
providers 111, and the like; and data storage 110 for storing data from the
host hardware
108, host system 104, user interface 102, and other inputs 112, such as
hardware 168,
removable memory 170, a wireless communication device 174, a computer 178, a
practitioner record 180 such as a dermatologist, general physician,
aesthetician, spa
employee, salon employee, cosmetic salesperson, and the like, a personalized
manufacturing record 172, and the like. While dermal embodiments are
contemplated
throughout this disclosure, except where context prohibits such embodiments
should be
understood to encompass non-dermal embodiments, such as and without limitation
any
hair, nail, agricultural, veterinary, internal, biological and non-biological
embodiments.
[00140] An imaging device 108 may be used to capture images of skin
structures to obtain biophysical skin properties such as in a skin health test
160, a pre-
diagnosis 162, remote monitoring 164, and the like. The imaging device 108 may
also be
adapted to capture images of non-dermal structures, such as hair, nails,
teeth, eyes,
internal organs and structures, and the like. The imaging device 108 may use
an internal
or external light source 127 to provide a specific sequence of irradiation
using
unpolarized light, such as diffusion light, white light, monochromatic light,
light of
multiple single wavelengths, and the like, then polarized light in order to
obtain data on
skin structures. In embodiments, the incident light may be polarized or
unpolarized and
the reflected or re-emitted light may be polarized or unpolarized. The
polarized light
Page 32 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
may result from the reflection on the skin and is not polarized from the light
source. The
capture and storage of the reflections enables the imaging and analysis of
skin lesions, as
well as all types of skin diseases, skin problems, and cosmetic concerns and
indications.
Analysis of polarized reflections may enable obtaining thermal, electrical,
and magnetic
properties of the imaged skin area. The images may be transmitted to an
analysis facility
154, analyst, practitioner and the like, which may also include assessment
with patient
questionnaires, to determine a final analysis of skin health. The device 108
may also
employ specific targeted wavelengths, such as in the red, green, and blue
areas, to
identify key features, based on spectroscopic and quantitative analysis of
skin lesions.
The device 108 may be used with diffused reflectance techniques, as well as
with color
imaging analysis based on indirect results from spectroscopic techniques (DR,
SF, etc).
In embodiment, the device 108 may be adapted to emit polarized light. The
device 108
may be adapted to emit more than one type of light and may be able to switch
among or
combine various light sources 127. The skin health analysis may be compared
with a
previous user skin health analysis, other users' skin health analysis, other
users'
experience data, and ingredient, product, and regimen characteristics to
provide a
recommendation for and track the effectiveness of a product or regimen 108.
[00141] Referring now to Fig. 2, in an embodiment, the imaging device 108
may comprise an illumination source 127 to direct unpolarized light, diffusion
light,
white light, monochromatic light, light of multiple single wavelengths,
polarized light,
and the like, upon the skin at an angle alpha, a sensor for detecting
reflected or re-emitted
light from a skin structure, and an image storage device for storing and
transmitting the
captured images. A skin structure may be at least one of a cell, a molecule, a
group of
cells, a group of molecules, an epidermis and sublayers, a basement membrane,
a dermis,
a subcutis, a gland, a stratum, a follicle, a pore, a vascular component, and
the like
resident within the skin. In an embodiment, the light source may be white
light for
generating reflected or re-emitted light and diffuse emission, such as
polarized light, to
measure the electrical and magnetic components of the skin. White light may be
emitted
as a combination of wavelengths of light across the spectrum of visible light.
Incident
unpolarized light may be directed at its target at a defined angle `alpha'
from vertical. As
the value of alpha changes, such as and without limitation over a range of 0
to 90 degrees
Page 33 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
from vertical, incident unpolarized light may interact with different
structural elements of
the skin since varying the angle of incidence affects the depth of
penetration. The angle
alpha may be changed by changing the position of the light source, either
manually,
through a remote control, through a user interface 102, and the like. The
relationship
between depth of penetration and alpha may be defined by the formula depth =
f(alpha).
For each skin structure which may correspond to a particular known depth
within the
skin, there may be a specific angle of incidence which produces a full
polarized
reflection. By analyzing the reflected or re-emitted light and/or diffuse
emission, either
polarized and/or diffusion, information on the underlying skin structures
responsible for
the reflection and/or re-emission may be obtained. The diffuse emission occurs
because
there is scattering and absorption that occurs from light bouncing around in
the
substructures. The polarization of the light may be due to classical / quantum
effects of
skin structures interacting water. That is, skin structures possess enough of
a magnetic
and electric field to be able to alter the polarization of light as it strikes
the structures and
to affect the wavelength of light as it strikes the structures. An aspect of
the polarization
of the reflected or re-emitted light, such as an orientation, an amplitude, a
phase, an
angle, a shape, a degree, an amount, and the like, may correlate with various
measures
associated with the particular skin structures targeted, and ultimately, a
skin state 158.
For example, a lesion present in a particular skin structure may cause the
diffusion of a
portion of the reflected or re-emitted light resulting in reflected or re-
emitted light that is
partially polarized and partially diffused. For example, collagen structures
are one
indicator of a biological difference between a benign and a malignant
melanocytic skin
lesion. The collagenous differences may affect the polarization state of
reflected or re-
emitted light, and the resultant images may indicate locations of tumor center
and tumor
periphery. Such images may aid a practitioner in visualizing excision margins,
as will be
further described herein. Because melanocytes are located at the lower part of
the
epidermis, the appropriate wavelength may be selected for this depth as well
as for the
chromophores within the various types of nevi.
[00142] If incident light is polarized, only the electrical properties of skin
will
be apparent but unpolarized incident light may reveal both the electrical and
magnetic
properties of skin. While using polarized light may generate improved
induction of
Page 34 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
optical activity, the data sets generated may be of less value as compared to
the data sets
captured using incident unpolarized light, such as white light, a
monochromatic light,
light of multiple single wavelengths, and the like. By measuring the effects
between
1 OE-34 and 1 OE-30 Js, one can make measurements at the border area of
quantum and
classical physics effects on the skin and as a difference of action of
electrical and
magnetic forces of valence electrons of skin's biomolecules.
[00143] In an embodiment, the wavelength and/or intensity of the incident
light
may be modified in order to measure the presence of specific molecules, such
as
collagen, elastin, cadherin, hemoglobin, and the like. Certain molecules
possess the
property of endogenous fluorescence. For example, if incident light is limited
to a
particular wavelength, such as 325 nm, collagen may be detected at an emission
wavelength of 400 nm and 405 nm. Table 1 lists certain illustrative examples
of
excitation and emission maxima of biological molecules that exhibit endogenous
fluorescence, such as amino acids, structural proteins, enzymes and coenzymes,
vitamins
and vitamin derivates, lipids, porphyrins, and the like. To detect the
presence of specific
molecules in the skin, a user may shine a light of a specified wavelength,
such as and
without limitation those shown in the excitation maxima column, onto the skin
and
collect reflected or re-emitted light to identify the presence of specific
emission
wavelengths in the reflections. It may be understood by one knowledgeable in
the art that
many different single wavelengths and combinations of wavelengths of light may
be used
to illuminate the skin.
Endogenous fluorescence Excitation Emission
maxima (nm) maxima
(nm)
Amino acids
Tryptophan 280 350
Tyrosine 275 300
Phenylalanine 260 280
Structural proteins
Collagen 325 400, 405
Elastin 290, 325 340, 400
Enzymes and coenzymes
FAD, flavins 450 535
NADH 290,351 440,460
NADPH 336 464
Page 35 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
Endogenous fluorescence Excitation Emission
maxima (nm) maxima
(nm)
Vitamins
Vitamin A 327 510
Vitamin K 335 480
Vitamin D 390 480
Vitamin B6 compounds
Pyridoxine 332,340 400
Pyridoxamine 335 400
Pyridoxal 330 385
Pyridoxic acid 315 425
Pyridoxal 5o-phosphate 330 400
Vitamin B12 275 305
Lipids
Phospholipids 436 540, 560
Lipofuscin 340-395 540, 430-460
Ceroid 340-395 430-460, 540
Porphyrins 400-450 630, 690
FAD, flavin adenine dinucleotide; NADH, reduced nicotinamide adenine
dinucleotide; AND(P)H, reduced
nicotinamide adenine dinucleotide phosphate.
[00144] In an embodiment, light may be emitted at any wavelength, such as
across the range from 280 nm to 3800 nm. Incident light may be blue, yellow,
orange,
red, or some other light.
[00145] Continuing to refer to Fig. 1, in an embodiment, the light source may
be integral to the device 108 or provided from an associated source. The light
source
may be a light-emitting or laser diode (LED) of any wavelength, such as and
without
limitation 280, 340, 360, 385, 405, 395, 400, or 480 nm incident excitation
wavelengths,
as well as infrared and near-infrared. Wavelengths in the ultraviolet and
infrared ranges
may also be emitted by the device 108. The light source may be diffusion
light, white
light, monochromatic light, light of multiple single wavelengths,
incandescent,
electroluminescent, fluorescent, halogen, ultraviolet, polarized light,
collimated light,
light provided by a wireless communications device, light provided by a fiber
optic cable,
and the like. In an embodiment, the light source may comprise a diffuser to
provide
diffuse incident light.
Page 36 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00146] In an embodiment, a sensor for detecting reflected or re-emitted light
from the skin may be embodied in optics resident in a CCD camera, CMOS-based
imaging system, digital camera, webcam, camera embedded in a communications
device
such as a cell phone or iPhone, PDA (Personal Digital Assistant), a watch or
other
wearable device for continuous monitoring of the skin as in a sports-type
indication, a
third party device, a scanner, and the like. The sensor may be adapted to
absorb any
wavelength of light, such as near IR or visible wavelengths. The sensor may be
adapted
to automatically filter out particular wavelengths. The sensor may be adapted
to image
any size area, such as a small portion of the skin, the full face, a complete
cutaneous
examination, and the like. The sensor may be adapted to operate without any
intervening
fluids between the device 108 and the area of concern, or may be used with an
oil-like
application or other reflective media to the area of concern. The sensor may
be adapted
to detect reflected or re-emitted light, from any distance from the area or
when in contact
with the area of concern, which may be used for subsequent visual and/or
algorithmic
analysis. The images generated from this reflected or re-emitted light may be
considered
both visual as well as spectroscopically resolved images or electromagnetic
skin maps.
The sensor may have an internal calibration scale that enables measuring the
size of the
region being imaged as well as the distance from the imaged area. In an
embodiment, a
lens may focus the reflected or re-emitted light from the detection optics
onto a visible-
NIR sensitive CCD, CMOS, or other sensory device. In an embodiment, the sensor
may
be adapted to acquire images at a high frame rate. In an embodiment, the
device may
possess a high magnification lens.
[00147] In an embodiment, the device 108 may store captured images for
analysis and/or transmittal to an analysis facility 154. The analysis facility
154 may be a
practitioner, an automated analysis tool, a practitioner employing analysis
tools, and the
like. Data storage 110 may occur manually when image capture is initiated, may
occur
automatically upon contact with the skin, may be remotely controlled, and the
like. Data
may be stored in an internal device memory 168 or may be stored externally in
memory
media 170 such as USB memory, an external hard drive, a mass storage device,
and the
like. The device may be able to connect externally, either through a wired
connection or
wirelessly, to a computer, such as a laptop, kiosk, desktop computer, central
server, and
Page 37 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the like. For example, the connection may be a direct USB connection. When the
device
108 is connected to the computer, captured data may be downloaded or
transmitted either
automatically or upon manual initiation from the device 108 to the computer.
For
example, the device 108 may have a cradle in connection with a computer. When
the
device 108 is placed in the cradle, data may be transmitted or downloaded from
the
device 108. Additionally, the device 108 may receive software updates when
connected
to the computer, such as through the cradle. In embodiments, the device 108
may have
no internal storage and may only be able to transmit or store data externally
through a
persistent hard-wired or wireless connection. Data transmittal and storage may
be a fully
automated process or may be manually operated. Data may be transmitted over a
wireless network connection, a cellular connection, a wired connection, a
Bluetooth
connection, and the like. Data transmittal from the device 108 may enable
remote
assessment techniques. In an embodiment, non-image data may also be stored
and/or
transmitted by the device 108 as described herein, such as voice responses,
text
responses, video data, and the like. The device 108 may have an internal
microphone to
record audio, a video camera to record video, a keyboard input to record text
responses,
and the like. In an embodiment, the device 108 may use externally available
audio and
video.
[00148] In an embodiment, data storage may be in a skin health record 121.
The skin health record 121 may be an object or database or repository for an
individual
that contains information on key medical, non-medical, and cosmetic
indications related
to a user's skin. This may comprise images, graphics, icons, written history,
personal
demographic information, levels of cosmetic conditions such as moisture,
elasticity,
firmness, texture, color level, or non-medical conditions such as
inflammation, and the
like. A user may self-populate the record 121 with data from any device 108,
109 or
input 112. The record 121 may contain a history of skin concerns, comments, a
user
blog, and the like. In an embodiment, the skin health record 121 may auto-
populate upon
acquisition of an image. For example, when a user submits their first image
for analysis,
a record 121 may be automatically created and populated with information,
which may be
edited, derived from the image and its analysis.
Page 38 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00149] In an embodiment, data storage 110 may occur in a practitioner record
180. A practitioner record 180 may be a repository of key health
characteristics
including background demographic data, personal information, information on
diet, skin
health record 121 and the like. It may have embedded images, links to other
image data
files, tracking effectiveness of personal skin products, medical products, and
OTC
products and the like and their historical impact on key parameters. It may
also capture
community data or data of selected individuals who may be similar to the
patient or user
and may include rankings and comments and the like
[00150] In an embodiment, data storage 110 may be in a personalized
manufacturing record 172. Based on the skin health measurement 160, product
ingredients to obtain a desired effect to make the skin healthy may be
selected. This
ingredient selection may be achieved by analyzing and tracking the change of
various
skin health parameters through the application of various products and
ingredients
through using the device 108 and tracking the change of the skin health over
time through
a personalized manufacturing record 172. Once the selected product ingredients
are
identified, they may be mixed to create a product best suited for the
individual's skin
characteristics and/or desired goals (such as improved moisturization). Thus a
personalized product may be developed for the user. Additionally, this same
process
could be used for creation of specific customized skin products and
ingredients for
medical and non-medical purposes and conditions.
[00151] In an embodiment, the form of the data captured may be compatible
with any standard image processing and manipulation software and techniques,
word
processing software, slideshow presentation, spreadsheet applications, and the
like. For
example, the captured data may be in any suitable image format, such as jpeg,
tiff, pict,
png, bmp, gif, pdf, and the like. In an embodiment, multiple images may be
captured as a
movie or a movie may be constructed from combining multiple images.
[00152] In an embodiment, the device 108 may be powered by any suitable
source, such as an electric power plug, a battery, solar power, USB power, and
the like.
A user may initiate power to the device 108 in order to begin acquiring
images.
Acquisition may commence automatically, may commence when the device 108 is
placed
Page 39 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
against the skin, may commence when a trigger, such as a button, is actuated
by a user,
and the like.
[00153] The device 108 may have a display for viewing the area to be imaged.
For example, a user may use the display with positioning tools to obtain exact
images
over time, such as a series of images taken over different days. The display
may be
integral to the device 108 or may be a separate display. For example, the
device 108 may
be connected to a monitor, such as that of a computer, using a wired
connection or a
wireless connection. In an embodiment, a user interface 102 to the device 108
may
display a real time view of the imaging.
[00154] The positioning tools may enable tracking and targeting. Referring to
Fig. 55, a method of tracking and targeting is depicted. The positioning tools
may be
used to track and store movement parameters of the imaging device 108 moving
over a
subject area. First, the device may capture an image of the subject area at a
plurality of
locations. Then, the device 108 may identify a direction of movement of the
imaging
device 108 using an image processing technique for at least one captured
frame. The
image processing technique may recognize the direction of movement of the
imaging
device by comparing each frame with at least three distinct features captured
to thereby
triangulate a location of the imaging device, as shown in Fig. 55. The data of
the
captured image may be compared with a predetermined image database to store
the
image of the subject area and to store placement parameters of the imaging
device 108.
If no entry exists in the database, a new entry may be made. The step of
capturing the
image of the subject area at a plurality of locations may include a sub-step
of capturing a
continuous video image of the subject area. The step of capturing the image of
the
subject area at a plurality of locations may include a sub-step of capturing a
frame by
frame sequence of images of the subject area. The step of identifying a
direction of
movement of the imaging device using an image processing technique may include
a sub-
step of a frame by frame comparison of the captured image to identify movement
parameters of the imaging device. The step of recognizing the direction of
movement of
the imaging device by comparing each frame with at least three distinct
features captured
to triangulate a location of the imaging device may include a sub-step of
capturing a
direction of movement of the imaging device by comparing three or more
distinct
Page 40 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
positions across different frames. The positioning tools may be an automated
location
tracking and data storage system for the imaging device 108, including an
image
capturing unit, a positioning unit coupled to the image capturing unit for
positioning the
imaging device on a subject area, and an image processing unit for enabling a
frame by
frame comparison of the captured image and for enabling the imaging device to
capture
three or more distinct points to triangulate a location of the imaging device
to identify a
direction of movement of the imaging device. The image capturing unit may
include a
digital camera. The image capturing unit may include at least one of a mobile
device and
a Personal Digital Assistant (PDA). The image processing unit may include a
comparison unit for comparing positions of three or more distinct points
across different
frames to capture direction of movement of the imaging device. The automated
location
tracking and data storage system may further include a sub-system for
measuring lateral
motion of the image capturing unit from a predetermined point to a new
location on the
subject area.
[00155] In an embodiment, the device 108 may have security features in order
to protect the privacy of user data. For example, the device 108 may have a
unique
MAC-ID with encryption technology.
[00156] In an embodiment, the device 108 may be associated with peripherals
or other functional attachments. For example, the device 108 may be associated
with a
blood pressure monitor or sensor, a heart rate monitor or sensor, and the
like. For
example, the device 108 may be used to perform a pre-diagnosis 162 of a skin
lesion
while also monitoring other endpoints such as blood pressure, heart rate, and
the like in
order to assess other aspects of health in addition to skin health.
[00157] In an embodiment, the device 108 may be sized to permit a user to
operate the device 108 in a handheld fashion. The device 108 may sized for
portability.
The device 108 may adapted for single-handed operation. For example, the
device may
be embodied as in Fig. 4 A & B, but it may have multiple other embodiments in
any
shape and/or size, such as a mirror, a large device adapted to image a large
area, a PDA, a
scanner, a mobile communication device, and the like. In Fig. 4 A, the
illumination
source is visible as a ring of LED's around a central detection area. In both
images, the
size, handheld nature, and portability are clearly demonstrated. The ease of
operation
Page 41 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
enables even an inexperienced user, such as a home user connected to a laptop,
to employ
the device 108. The device 108 may be a self-contained unit and not part of a
larger
camera system. In an embodiment, the device 108 may be designed for one handed
ergonomic holding. In an embodiment, the device 108 may be used with or
without
application of reflective media. In an embodiment, the device 108 may be used
to
capture images at a distance, close-up, in direct contact, and the like. For
example,
software loaded on a computer interfaced with the device 108 may prompt for
near
distance and far distance image capture.
[00158] In an embodiment, the device 108 may also be a standalone, non-hand-
held version, which may be used to take images or particular body components
or
materials.
[00159] In some embodiments of the skin care device, the device may be a
miniature one, enabling portability and hand-held use. Some embodiments of the
skin
care device may be in the form of a hand-held and portable wand that can be
conveniently moved across a skin region to be examined. Some other embodiments
of
the skin care device may be so miniaturized that no dimension of the skin care
device
exceeds six inches. Such skin care devices may be embedded in wearable
accessories, for
example, bracelets, necklaces, ear-rings, and the like. Some embodiments of
the skin
care device may have a convenient user interface and/ or a display surface. In
some
embodiments of the skin care device, the device may be coupled to or embedded
in a
vertical display panel, for example but not limited to, a mirror, an LCD
screen, a plasma
screen, and the like.
[00160] Referring to Fig. 47, an exemplary skin care device 4700 embodying
the principles of the invention is shown in a block diagram. The skin care
device 4700
may include an electromagnetic radiation source 4702, a radiation detector
4704, and a
skin condition analysis module 4708.
[00161] The electromagnetic radiation source 4702 may be capable of directing
incident electromagnetic radiation to one or more locations on the skin of a
person. For
example, and not by way of limitation, the radiation source 4702 may be a set
of light
emitting diodes (LEDs). In certain embodiments, the incident radiation emitted
by the
radiation source 4702 may include radiation in the visible, near-infrared
(NIR) and near-
Page 42 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
ultraviolet (NUV) spectrum. In certain other embodiments, the incident
radiation may
include white light.
[00162] As depicted in Fig. 47, the electromagnetic radiation source 4702 may
be coupled to the radiation detector 4704. The radiation detector 4704 may be
capable of
detecting the radiation re-emitted from the location and measuring various
radiation
parameters of the re-emitted radiation. As shown in the Fig. 47, the radiation
detector
4704 may be coupled to the skin condition analysis module 4708. A variety of
radiation
parameters may be detected by the radiation detector, including, for example
but not
limited to, degree of polarization, intensity of the radiation at different
wave-lengths, and
the like. The electromagnetic radiation sources, radiation detectors, and the
skin
condition analysis module have been previously described herein.
[00163] The skin condition analysis module 4708 may be capable of analyzing
the radiation parameters of the reflected radiation and other information to
generate a
skin condition assessment. The skin condition analysis module 4708 may be
adapted to
generate the skin condition assessment in real-time. In some embodiments, the
radiation
detector 4704 measures diffused reflectance. In some other embodiments, the
incident
radiation may be white light and the radiation detector 4704 may measure the
red, green,
and blue components of the re-emitted light.
[00164] In certain embodiments, the skin condition assessment may also be
partly based on analysis of a photographic image of the skin location.
[00165] As used in the specification and the appended claims, the term
"diffused reflectance" may refer to radiation, sometimes loosely referred to
as light,
scattered in many directions from target samples. Diffused reflectance is the
complement
to specular, or mirror-like, reflection. If a surface is completely non-
specular, the
reflected or re-emitted light will be evenly spread over the hemisphere
surrounding the
surface. Diffused reflectance stems from tiny irregularities on surfaces of
targets and is
the reflection of incident light from uneven or granular surfaces of targets
such that
incident light strikes the targets and is scattered over wide angles.
[00166] Some embodiments of the skin care device may have a memory
module for storing the skin condition assessments and other data, such as with
timestamps. Some embodiments of the skin care device may have a communication
Page 43 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
module for communicating the skin condition assessments and other data with
timestamps to a remote computer. The communication of data may occur, for
example,
over a wire, wirelessly, using an internet, and the like. The skin condition
assessments
and other data may also be accessed in remote locations via mobile devices
and/or
computers. Such remote access may be particularly convenient for service
providers,
such as for example, dermatologists.
[00167] Some embodiments of the skin care device may have a user interface
to enable a user to interact with the skin care device. The user interface may
enable a
user to give instructions to the device, for example, to analyze the available
information
to generate a real-time skin condition assessment of a skin location or a
larger skin
region. In some other embodiments, the user interface may be voice-operated
providing
the facility to give commands to the skin care device through speech commands.
Other
examples of user interfaces that may be used in the skin care device are
graphical user
interface (GUI), web-based user interface (W UI), command line interface,
touch
interface, and any combination of the above.
[00168] In certain embodiments, the user interface may also provide alerts to
a
user if any abnormal skin condition, such as for example, a clogged pore, is
detected. The
alerts may be in the form of a light signal, a beep, an email alert, an SMS
alert, and the
like. There may be other methods, such as a small electric tingle, a mark, a
sound, and a
light, a heat emitting signal, and the like, to alert users about skin
conditions requiring
user attention.
[00169] Some embodiments of the skin care device may have also have a
display surface either for a more convenient and intuitive user interface
and/or for
viewing an image of a skin region and/or for viewing some useful skin-related
information, for example, a skin condition assessment report, a skin regimen
recommendation report, and/or a skin regimen effectiveness report. In some
embodiments, the display surface and/or the user interface may be touch-
sensitive to
enable touch-control of the device.
[00170] In some embodiments, the skin condition assessment data of locations
may be overlaid on an image of a larger skin region displayed on the display
surface,
providing a useful picture of the health of the entire skin region in a single
view.
Page 44 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00171] Some embodiments of the skin care device may also have an access
restriction module restricting access to patient data to authorized users
only. The access
restriction module may be based on a user name and password feature and/or
biometric
access control, for example, fingerprint recognition, facial recognition,
retina recognition,
and the like.
[00172] In some embodiments, the skin condition analysis module 4708 may
have access to user information like age, gender, ethnic group, and the like,
and such
information may be used to build a user profile and used in analysis of the
skin condition.
[00173] The skin care device 4700 may be used in a user's home, a user's
bathroom, a cosmetic store, a provider's office, a mobile location, and the
like. The skin
care device 4700 may be used at any time of the day, such as before going to
bed, before
or after using a cleanser on the skin, and the like.
[00174] The skin care device 4700 may have a skin care regimen
recommendation module 4710 capable of generating a displayable skin care
regimen
recommendation. The skin care regimen recommendation may include information
not
only about the most appropriate skin-care products, but also information about
the best
way of applying the product, the timing, amount, and frequency of application,
and the
like. The skin care regimen recommendation module 4710 may be linked to the
skin
condition analysis module 4708 so that the skin care regimen recommendation is
personalized to the skin condition of each person. The skin care regimen
recommendation may be generated in real-time based on skin condition
assessments
generated by the skin condition analysis module 4708, product information, and
other
relevant information analyzed using algorithms, as described herein. In some
embodiments, the skin care regimen recommendations generated by the skin care
regimen recommendation module 4710 may be displayed to the user in real-time,
for
example, on a display surface attached with the skin care device 4700.
[00175] In some embodiments, it may be possible to print the skin care
regimen recommendations generated by the skin care regimen recommendation
module
4710.
[00176] In some embodiments, the skin care regimen recommendations
generated by the skin care regimen recommendation module 4710 are based at
least
Page 45 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
partly on determination of a skin profile, or skin state 158, of the user and
use of skin care
regimen recommendations of persons with a similar profile.
[00177] In some other embodiments, the skin care regimen recommendation
module 4710 is coupled to a skin-care product database 190. If the products
recommended by the skin care regimen recommendation module 4710 are available
in
the product database 190, the user may be informed and given an option to
purchase the
product immediately. In some embodiments, the user may operate the skin care
device
4700 in a point-of-sale location, for example, a retail store, and the
availability of a
product recommended by the skin care regimen recommendation module 4710 may be
indicated by an audio-visual signal, such as for example by lighting up the
shelf in which
the product is located.
[00178] A user practicing a specific skin care regimen, for example, use of a
skin-care product in a prescribed manner, may be interested in tracking the
effectiveness
of the skin care regimen over a period of time. The skin care device 4700 may
have a
skin care regimen effectiveness module 4712. The skin care regimen
effectiveness
module 4712 may be coupled with the skin condition analysis module 4708. The
skin
condition of the user may be tracked at different points of time using the
skin care device
4700 and may be displayed to the user on a display surface. The device could
also help
track changes by various activities - exercise, food, smoking, work, and the
like.
[00179] Fig. 48 shows an embodiment of a skin care device 4700 in which the
skin care device is wand-shaped. For example, a user may switch on the wand-
shaped
device 4800 and move the device over her face. The wand-shaped device may have
a
grip 4802, a radiation detector 4808, an indicator 4804 that may provide an
indication
such as with light, warmth, sound, and the like, an LED light 4810, and a
power source
4812.
[00180] The wand-shaped device 4800 is functionally similar to the skin care
device 4700 described earlier. The wand-shaped device 4800 may comprise an
electromagnetic radiation source, a radiation detector, and a skin condition
analysis
module. The wand-shaped device 4800 may be miniature, hand-held, and portable.
[00181] In some embodiments of the wand-shaped device, the electromagnetic
radiation source may be one or more LEDs. Each of the LEDs may have unique
Page 46 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
predetermined frequencies. In some embodiments, the one or more LEDs may be
arranged in a line to form a light strip.
[00182] In some embodiments, the wand-shaped device 4800 may be powered
via a USB coupled to an external power source or through built-in batteries,
or other
similar power source.
[00183] As the wand is moved over the skin, light is emitted from the
radiation
source 4702. Then, the radiation detector 4704 detects re-emitted light and
sends
information back to the skin condition analysis module 4708. The module 4708
employs
an algorithm for skin condition analysis.
[00184] Fig. 49 shows another embodiment of a vertical panel-including skin
care device 4900, in which the skin care device comprises an electromagnetic
radiation
source 4702, a radiation detector 4704, a skin condition analysis module 4708,
a user
interface 4714, and a vertical display panel 4902.
[00185] The vertical display panel 4902 may have the user interface 4714 on
the sides of the vertical display panel 4902. In some embodiments, the display
panel may
be touch-sensitive and in such cases, the vertical panel itself may be part of
the user
interface. An image of a skin region may be displayed in the display panel. A
user may
touch a location on an image and this may trigger display of a magnified image
either on
the display panel or on another screen. A menu bar may show up in the user
interface
4714, and the user may be able to view various reports, for example, a skin
condition
assessment report, a skin regimen recommendation report, a skin regimen
effectiveness
tracking report, and the like.
[00186] The user interface 4714 may enable a user to give instructions to the
device, for example, to analyze the available information to generate a real-
time skin
condition assessment of a skin location or a larger skin region. In some other
embodiments, the user interface may be voice-operated providing the facility
to give
commands to the skin care device 4900 through normal speech commands. Other
examples of user interfaces that may be used in the skin care device 4900 are
graphical
user interface (GUI), web-based user interface (W UI), command line interface,
touch
interface, and any combination of the above.
Page 47 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00187] The basic functioning of the vertical panel-including skin care device
4900 is similar in many respects to the skin care device 4700. The
electromagnetic
radiation source 4702 is capable of directing incident electromagnetic
radiation to one or
more locations on the skin of a person. For example, and not by way of
limitation, the
radiation source 4702 may be a set of light emitting diodes (LEDs). In certain
embodiments, the incident radiation emitted by the radiation source 4702 may
include
radiation in the visible, near-infrared (NIR) and near-ultraviolet (NUV)
spectrum. In
certain other embodiments, the incident radiation may include white light.
[00188] As depicted in Fig. 49, the electromagnetic radiation source 4702 may
be coupled to the radiation detector 4704. A variety of radiation parameters
may be
detected by the radiation detector 4704, including, for example but not
limited to, degree
of polarization, intensity of the radiation at different wave-lengths, and the
like.
[00189] In certain embodiments of the vertical panel-including skin care
device, the skin condition assessment may also be partly based on analysis of
a
photographic image of the skin location.
[00190] Some embodiments of the vertical panel-including skin care device
may have a memory module for storing the skin condition assessments and other
data,
such as with timestamps.
[00191] Some embodiments of the vertical panel-including skin care device
may have a communication module for communicating the skin condition
assessments
and other data with timestamps to a remote computer. The communication of data
may
occur, for example but not limited to, over a wire, wirelessly, using an
internet, and the
like. The skin condition assessments and other data may also be accessed in
remote
locations via mobile devices and/or computers. Such remote access may be
particularly
convenient for service providers, such as for example, dermatologists.
[00192] In certain embodiments, the user interface 4714 may also provide
alerts to a user if any abnormal skin condition (for example, a clogged pore)
is detected.
The alerts may be in the form of a light signal, a beep, an email alert, an
SMS alert, etc.
There may be other methods e.g. a small electric tingle, a mark, a sound, and
a light, a
heat emitting signal, etc. to alert users about skin conditions requiring user
attention.
Page 48 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00193] In some embodiments, the skin condition assessment data of locations
may be overlaid on an image of a larger skin region displayed on the vertical
display
panel 4902, providing a useful picture of the health of the entire skin region
in a single
view.
[00194] Some embodiments of the vertical panel-including skin care device
may also have an access restriction module restricting access to private
information to
authorized users only. The access restriction module may be based on a user
name and
password feature and/or biometric access control, for example, fingerprint
recognition,
facial recognition, retina recognition, etc.
[00195] In some embodiments, the skin condition analysis module 4708 may
have access to user information like age, gender, ethnic group, etc., and such
information
may be used to build a user profile and used in analysis of the skin
condition.
[00196] The vertical panel-including skin care device 4900 may be used in a
consumer's home, a consumer's bathroom, a cosmetic store, a provider's office
and/or a
mobile location. The vertical panel-including skin care device 4900 may be
used at any
time of the day, such as before going to bed, before or after using a cleanser
on the skin.
[00197] In some embodiments of the vertical panel-including skin care device,
the device may include or be coupled with a skin care regimen recommendation
module
capable of generating a displayable skin care regimen recommendation.
[00198] In some other embodiments of the vertical panel-including skin care
device, the device may include or be coupled with a skin care regimen
effectiveness
module capable of generating a displayable skin care regimen effectiveness
report.
[00199] In some embodiments of the vertical panel-including skin care device,
the vertical display panel is a mirror.
[00200] In some embodiments of the vertical panel-including skin care device,
the vertical display panel is an LCD panel or a plasma screen.
[00201] In some embodiments of the skin care device, the device also includes
or is coupled with a camera for taking photographic images of a skin region.
[00202] In certain embodiments of the skin care device, the camera is
integrally attached to the display surface or display panel. In certain other
embodiments,
Page 49 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the camera is either wired to the display surface or display panel. In other
embodiments,
the camera is wirelessly coupled to the display surface or display panel.
[00203] In certain embodiments of the vertical panel-including skin care
device, the user interface 4714 may have one or more buttons (not shown
explicitly) for
doing a skin scan and/or analysis. The buttons may be of different types, for
example
push buttons, hard wired buttons, or a combination of both. The user may touch
a button
on the display panel for doing a skin scan, while she may touch another button
for
directing the machine to do a skin analysis.
[00204] Fig. 50 shows an embodiment of a wearable skin care device 5000, in
which the device is in the form of a wearable device. The wearable device can
be worn
by a user in the form of necklace, ear-rings, bracelets, a patch, or as a
sensor attached to a
strap, and the like. Such wearable devices can be persistent, personalized
skin care
monitors.
[00205] The wearable skincare device 5000 is functionally similar to the skin
care device 4700 described earlier. Similar to the skin care device 4700, the
wearable
skincare device 5000 comprises an electromagnetic radiation source, a
radiation detector,
and a skin condition analysis module. Preferably, the wearable skincare device
5000 is
miniature, hand-held, and portable, and no dimension of the device exceeds six
inches.
[00206] In some embodiments of the wearable skincare device, the
electromagnetic radiation source may be one or more LEDs. Each of the LEDs may
have
unique predetermined frequencies. In some embodiments, the one or more LEDs
may be
arranged in a line to form a light strip.
[00207] In some embodiments, the wearable skincare device 5000 may be
powered via a USB coupled to an external power source or through built-in
batteries,
motion power, solar power, or other similar power source
[00208] Embodiments of the wearable skincare device may also have sensors
for measuring various body and environmental parameters. Examples of body
parameters
that could be measured by the wearable skincare device are body temperature,
hemoglobin antioxidant level, etc. Examples of environmental parameters that
could be
measured by the wearable skincare device are air cleanliness, humidity,
temperature, UV
index, external air quality, smoke index, etc
Page 50 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00209] In an embodiment, the device 108 may be adapted for use as a
component of a minimally invasive medical device associated with laparoscopy,
cytoscopy, ureteroscopy, arthroscopy, endoscopy, dermoscopy, gynecology,
urology,
dentistry, natural orifice insertion analysis such as through ears, mouth,
anus, nose, and
external breast cancer analysis through the skin, and the like. For example,
the system
may be able to process the data and to appear on a video monitor or other
display in a
surgical suite or other medical setting. A medical professional may be able to
select a
viewing mode, such as still image capture or video capture, and may be able to
manually
adjust the parameters of the light source, sensor and display to assist in
observation,
identification, and monitoring with the device 108. In an embodiment, the
system may
be pre-programmed with various protocols for the various types of medical
procedures
and tissues types that a medical professional may encounter such that the
system may
automatically handle the device 108 based on the medical professional's
indication of the
type of procedure and tissue being examined.
[00210] For example, the device 108 may be used as part of a system and
method for distinguishing between healthy and suspect tissue in real or near-
real time on
a patient. The imaging device 108 allows a surgeon or other practitioner to
precisely
determine the border area around a surgical intervention for primary cutaneous
melanoma, skin cancers, and other skin diseases that require excision around
the skin.
Generally, the surgical excision of suspect tissue, such as cutaneous
melanoma, may be
determined either by a surgeon's experience or through a Breslow scale and
punch biopsy
that determines the thickness of a melanoma and hence generally agreed-to
border areas.
The device 108 allows an automatic determination of the excision margin for
primary
cutaneous melanoma based on the optical characteristics of the surrounding
skin. By
precisely defining where there is healthy tissue and where there is suspect
tissue, a
surgeon could leave a larger amount of healthy tissue around a site, decrease
recurrence
and decrease micrometastasis in surrounding skin while enabling minimal
surgical
morbidity and improved cosmetic appearance. The device 108 and associated
algorithms
150 and analysis techniques, such as the convolution technique and RGB color
analysis
discussed later herein, embodied in software, may be employed to image a
particular site,
and determine border area, suspect tissue, either before surgery, in pre-
surgery, or during
Page 51 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
surgery. The software could also show post surgical analysis of affected skin
tissue.
Using the device 108 allows more precise determination of the border area
instead of
relying on subjective experience or fixed tables as noted in medical journals
and other
published works. The advantage of this method is better isolated suspect
tissue and
retaining a greater degree of healthier tissue. Referring now to Fig. 56, a
melanocytic
lesion is displayed. The visible melanoma 5602 or suspect tissue is surrounded
by
normal looking skin, but which may contain unhealthy /diseased tissue that
must be
excised 5604 (pseudo-normal skin 5604). The device 108 may be able to
visualize the
border between healthy and non healthy tissue 5608, thereby allowing the
surgeon to
spare healthy tissue 5610 that should remain intact. The device 108 may
perform an
estimation and provide an outlined area 5612 indicating where the surgeon
should cut the
tissue. In Fig. 57, an embodiment of a user interface for visualizing a
melanocytic lesion
is displayed along with access to tools for analyzing an image of the lesion
5702,
manually selecting a border 5704, automatically selecting a border 5708,
drawing a
border area 5710, and the like.
[00211] In an embodiment, the device 108 may enable a skin health test 160.
The imaging device 108 may be used to perform a skin health test 160 to learn
the
characteristics of the skin and to obtain a diagnosis. The hardware device may
capture an
image and enable analysis of the image. The imaging components within the
device 108
may enable measuring various skin health characteristics like color, age,
damage,
collagen, elastin, pores and types, keratin, and the like. The skin health
test 160 may be
performed in the home, in a spa, clinic, hospital, from a mobile phone at any
location,
and the like. The skin health test 160 may be used in conjunction with
specific
background information through questionnaires, image upload, genetic testing,
DNA
samples, and lifestyle habits to determine a skin state 158. The test 160
would respond
with specific information related to the biophysical health of the skin, a
portion of which
would be physical and genetic disposition to certain medical or non-medical or
cosmetic
problems or conditions.
[00212] In an embodiment, the device 108 may enable a pre-diagnosis 162.
This is a system of pre-diagnosis where a practitioner (such as the user, a
dermatologist,
medical practitioner, aesthetician, and the like) may receive or request from
a user to take
Page 52 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
an image and/questionnaire of a skin concern or the like and receive a pre-
diagnosis
based on algorithmic analysis of pre-existing conditions. The user may submit
a
questionnaire and image with a pre-diagnosis of conditions prior to going to
see a
practitioner and allow a follow-up. Images captured by the device may be
submitted to
obtain a preliminary diagnosis to enable effectively referring the case to the
best
practitioner. The pre-diagnosis 162 may be performed by software algorithms on
the
images, manual analysis, a combination thereof, and the like. The pre-
diagnosis 162 may
include the preliminary assessment as well as indicate the time required and
the steps
required for the final diagnosis or assessment. This pre-diagnosis 162 feature
may enable
effective scheduling of the practitioner. The pre-diagnosis 162 could also
help screen for
particular skin issues as well as identify users with certain issues.
[00213] In an embodiment, the device 108 may enable remote monitoring 164.
The user may use the device in the privacy of their home, work, or any other
location to
perform remote monitoring 164 and submit images to track progress of their
skin's health
or medical conditions. A practitioner may be able to remotely guide changes in
treatment
or guide on prevention factors. Remote diagnosis may greatly increase
efficiency of
progress monitoring since users will not have to make a physician trip to the
provider,
and the provider could conveniently select a time during the day to observe
the patients
change. The monitored data may be viewed as a recording or in real time.
[00214] In an aspect of the invention, the imaging device 108 may illuminate
an area of concern at a known angle of incidence with unpolarized light. To
obtain a
spectral diagram based on the magnetic properties of the area only, the
reflected polarized
light, which possesses the electrical properties of the area of concern, may
be subtracted
from any reflected diffusion light, which possesses electromagnetic properties
of the area
of concern. The distribution of pixels in the image corresponding to the
diffusion light
and reflected polarized light may be determined and indicated by any
conventional
means. For a known image sensor, a one-to-one mapping of pixel image
distribution
between the diffusion light image, corresponding to an electromagnetic signal,
and
reflected polarized light, corresponding to an electrical signal image, may be
made with a
distribution of the intensity of the spectroscopic data for the same area. A
magnetic
gradient image of the area may be made by equipment such as an AFM-MMR (Atomic
Page 53 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
Force Microscopy in Magnetic Mode Regime) and from the one-to-one
correspondence,
a skin state 158 may be based on the gradient image, diffusion light image,
and reflected
polarized light image.
[00215] In an embodiment, the device 108 may be an imaging device 108 for
performing digital spectroscopic imaging of the skin. Incident unpolarized
light may be
delivered, either vertically or on an angle alpha from vertical, from an
unpolarized light
source associated with the device 108, such as a white light, diffuse light,
monochromatic
light, light of multiple single wavelengths, and the like, to a target skin
structure. White
light, which possesses both electrical and magnetic properties, when incident
onto a skin
structure at a particular angle interacts with the structure's components and
leads to the
reflected or re-emitted light having a polarized light component. In
embodiments, the
incident light may be polarized. Unpolarized light reflected by skin
structures may
become polarized, at least in part. The reflected or re-emitted light, either
polarized or
diffusion light, may be captured by the device 108. Such multispectral skin
imaging may
be used to develop an electromagnetic skin topography. By measuring aspects of
the
polarization of the reflected or re-emitted light such as an orientation, an
amplitude, a
phase, an angle, a shape, a degree, and an amount, and the wavelength of the
reflected or
re-emitted light, the biophysical properties of skin structures may be
obtained. A skin
state 158 may be determined from the aggregate biophysical data obtained from
one or
more skin structures as well as a visual analysis of the captured images and
any
additional data obtained from the user anecdotally. For example, the skin
state 158 may
encompass data on moisture, wrinkles, pores, elasticity, luminosity, and any
of a number
of measures, as described herein. By varying alpha, the angle of incident
white light, the
depth of penetration of the light to skin structures may be varied. Each depth
within the
skin corresponds to different skin structures. For each skin structure or
depth, there may
be a specific angle which produces a full polarized reflection. For example, a
certain
angle of incidence may be used to obtain data for skin structures within the
epidermis,
however, the angle of incidence may need to be changed in order to obtain data
on skin
structures within the subcutis which resides at a different depth within the
skin. The
angle of incidence may be modified to penetrate the skin anywhere from a few
microns
up to a few centimeters, thus enabling the capture of reflections from other
non-dermal
Page 54 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
structures. For example, the device 108 may be used as a non-invasive imaging
tool,
such as to image tumors, breast cancer, melanoma, and the like. In an
embodiment, the
area to be imaged may be any biological tissue that may have normal or
pathologic
variations in its structure, such as variations in the tissue's birefringent
properties. For
example, scars, keloids, hypertrophic scars, and stria all have organizations
of collagen
fibers that are different from normal skin. Since collagen is a primary
determinant of
cutaneous wound repair, it may be of interest to monitor changes in collagen
structure
and concentration. For example, the stage of healing may be determined by the
size of
collagen bundles which may increase as healing progresses, by the organization
of
collagen structures at the molecular or small-fibril level which may increase
as healing
progresses, by the return or increase of birefringence, and the like. Since
collagen
structures are polarization-sensitive, changes that occur in the structures
may be
monitored using a polarization-based technique during scar formation, the
healing
process, and treatment of scars, as has been and will be further described
herein.
[00216] Being able to measure the electrical and magnetic properties of
various
skin structures may enable the differentiation between healthy and non-healthy
skin
structures. Normal or healthy skin structures exhibit a unique conformation
that differs
from the conformation exhibited by equivalent structures when unhealthy or
abnormal.
These conformational changes can be detected by differences in an aspect of
the light
reflected off of skin, re-emitted light, or amount of absorption in the skin,
such as an
aspect of the polarization of the reflected or re-emitted light. The aspect of
polarization
may be the wavelength of the light, an orientation, an amplitude, a phase, an
angle, a
shape, a degree, an amount of polarization of the light, and the like.
According to
Maxwell's equations, light can be described as comprising an electric field
and a
magnetic field which can be described as two vectors, E and B, which behave as
waves.
The vectors are perpendicular to the propagation direction of the light, and
they are
orthogonal to each other. Furthermore, given the electric field E, B can be
determined
via Maxwell's equations, and vice versa. Thus, by measuring the electrical
component of
the light reflected, re-emitted, or absorbed by the skin structures, the
magnetic component
or the degree of polarization/polarization state may be determined.
Alternatively, the
light may spread to other wavelengths that can be measured. By comparing those
Page 55 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
electrical and magnetic readings from the polarized component of reflected or
re-emitted
light and non-polarized white light to that of normal or healthy skin
structures incident
with light at the same or similar angles, changes may be detected in the skin
structure and
its molecular or structural conformation. Based on the amount or other aspect
of both
electrical and magnetic determination, specific defects such as cancer, skin
diseases,
cosmetic indications and the like, may be detected, since each range of
measurements
may correspond to a particular defective conformation. If any other molecules,
cell, or
structure are now incident with the same type of light at the same angle, the
strength of
certain wavelengths of the reflected component may enable the measurement of
the
intensity of the difference in conformation states of the measured component.
The
polarization state of the reflected or re-emitted light may be described by a
number of
parameters. The polarization state may be described in terms of the
polarization ellipse,
specifically its orientation and elongation. Parameters which may be used to
describe the
polarization state may include the azimuth angle (w) which is the angle
between the
major semi-axis of the ellipse and the x-axis, the ellipticity (E) which is
the ratio of the
two semi-axes, the ellipticity angle which is the arctangent of the
ellipticity, the
eccentricity, the amplitude and phase of oscillations in two components of the
electric
field vector in the plane of polarization, and the like. For example, an
ellipticity of zero
corresponds to linear polarization and an ellipticity of 1 corresponds to
circular
polarization. The polarization of the reflected or re-emitted light may be at
least one of
elliptical, linear, circular, left-circular, right-circular and any potential
combinations
thereof.
[00217] In an embodiment, determining a skin state 158 may comprise
processing and analyzing 154 the reflected or re-emitted light to obtain
images for visual
and spectroscopic analysis. Analysis 154 may be facilitated by examining the
wavelength and other characteristics of the reflected or re-emitted light. For
example, if
the incident light is white light, the reflected or re-emitted light may be
filtered to
examine a collection of wavelengths or a single wavelength and, ultimately, a
specific
skin structure fluorescence. In another example, monochromatic or semi-
monochromatic
light, such as provided by an LED may be used to excite targeted fluorophores
and
chromophores. In this example, fluorescence of deeper layers may be extracted.
The
Page 56 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
reflected or re-emitted light in this example may also be filtered to isolate
a specific
fluorescence. In another example, varying the wavelength of the illuminating
light may
enable detection of biophysical properties from various depths within the
skin. In
addition, certain chromophores, such as the various forms of hemoglobin found
in blood,
have specific absorption bands; thus processing of data created with different
color light
may yield information about chromophore distribution that may be polarization-
sensitive.
The wavelength dependence may be obtained in several ways: 1) illuminate
sequentially
with light of a single wavelength or multiple single wavelengths and collect
each
resultant image separately; or 2) illuminate with white light and examine the
reflected or
re-emitted light for individual wavelengths or a collection of individual
wavelengths
either during detection or during processing. Algorithms 150 may be used to
obtain
information from data obtained by either method by processing and analyzing
one or
more wavelengths of light to form a spectroscopic, polarization-based image.
In an
embodiment, the combination of both techniques may enable the elimination of
the
reflection from the surface of the skin.
[00218] In an embodiment, filtering may be employed to filter out a range of
wavelengths, such as those belonging to the ultraviolet, infrared, near
infrared, visible,
and the like. The filter may be a digital or an analog filter. For example,
captured
images may be processed by software that may be able to employ digital filter
techniques
to process the images for analysis. For example, using software, any digital
filter
parameter may be selected such as a particular cutoff wavelength, a set of
single
wavelengths, a sampling interval, and the like. For example and without
limitation, a
digital filter may be used to isolate reflections of 405, 458, 488, 532, 580,
and 633 nm
wavelengths. In another example, an analog filter may be employed to filter
the images
as they are captured, such as a filter that is integral to the optics of the
device 108, or as
they are stored, transmitted, manipulated, processed, and the like, such as
with an
external analog filter. Filtering the images may result in obtaining images of
underlying
structures and/or a specific pattern of polarization. Filtering the images may
result in the
separation of the electrical and magnetic components of the reflected or re-
emitted light.
Filtered images may be subjected to algorithmic analysis. Filtering may
eliminate
reflections due to skin surface reflections by isolating specific wavelengths
of light. For
Page 57 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
example, sebaceous glands may appear as bright spots in an image when only a
certain
wavelength of light is isolated for analysis, while isolation of a different
wavelength of
light enables the visualization of all the pores in the imaged area. Thus, the
fluorescence
from deeper layers may be isolated. Image processing may be used to count and
measure
changes in the sebaceous glands and pores, including count, size, activity of
gland,
quantity of sebum/other materials inside the sebaceous gland, quantity of
sebum/other
materials inside the pore, age of the contents within the gland, age of
contents within the
pore, amount of inflammatory processes surrounding the gland, and the like.
Multiple
images from different image sources may be combined for the analysis. The
analysis
results in function, diagnosis, prognosis of skin health, such as disposition
to acne,
oilyness, shine, viscosity, and the like. The analysis may be combined with
color image
processing (RGB analysis, for example) to determine other skin
characteristics.
[00219] In an aspect of the invention, a host system 104 may comprise
algorithms 150, data integration 152, analysis tools/ API's 154, a skin state
158, an expert
consult 128, and the like. The skin state 158 may be a data object or
characterization of
skin based on tests 160, pre-diagnoses 162, and monitoring 164 performed by a
device
108, user input, expert consult 128, other inputs 112, analysis 154,
algorithms 150, and
the like. The skin state 158 along with all of the underlying data and user
information
may be stored in a skin health record 121. In an embodiment, the host system
104 may
comprise server architecture. The host system may be technology agnostic. The
host
system 104 may comprise one or more cloud computing, service-oriented
architecture,
distributed objects, and the like.
[00220] In an embodiment, expert consult 128 may provide analysis,
recommendations, assessment advice, and the like. The skin image data
collected as well
as the pre-diagnosis, in addition with any other allied data such as
physician's diagnosis,
insurance, blood analysis, and the like may be referred to an expert either by
the user or a
practitioner, or by other users to obtain an analysis, recommendation or
assessment
advice. Experts could be located in geographically distant locations, and may
have very
different skills. For example, the skin image data and analysis may be shared
at the
request of another user with an herbal specialist in India, or the user may
request the
image data to be shared with an aging expert in France to learn of best suited
skin care
Page 58 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
treatment from their experience. The expert's consultation analysis may be
maintained
on the host system 104 as part of the skin history record 121 and may be
accessed by the
user at their convenience, or shared with other users.
[00221] In an embodiment, the system 104 may be a home-based, in clinical or
medical settings, at spas and salons, at a cosmetics counter and in cosmetics
sales, and the
like to perform skin analysis discretely and accurately in a low cost, rapid,
and secure
fashion. In embodiments, the device 108 may integrate with a user interface
102, online
platform 129, mobile platform 124 and the like to perform analysis 154, skin
state 158
record keeping, obtain referrals/analysis from a remote practitioner or
algorithm 150, and
the like. The home-based system 104 may allow a practitioner, who may be any
qualified or unqualified person to give advice, to analyze cosmetic or non-
cosmetic
conditions that may be captured by an imaging device 108 or third party device
109 and
give advice and recommendations on products, regimen, diet, lifestyle and the
like based
on inputs from questionnaires, uploaded images, and the like. The system may
consist of
a starter website that may be customizable for a personal business where the
practitioner
could organize clients' cosmetic skin health, track their regimens, recommend
products,
be their online advisor, and the like. This would leverage the analysis and
device
platform to allow a practitioner to analyze comments, images, questions,
and/or concerns
and the like and give advice, consultation on lifestyle improvement and
tracking. A spa/
salon based system may enable personalized skin assets. For example, the spa
may own
the device, the device may capture images to feed a large scale display
adapted to present
a skin condition, and then a practitioner may be able to simulate the effect
of treatment.
Users may compare a skin state 158 with peers or other spa goers and generate
recommendations based on what worked for them or what they bought. Desired
improvements may be correlated to ingredients and most effective products /
regimens
118 for the users' skin. The regimen 118 may be a feature that enables users
to learn
what product sequence would work best for their skin, based on a hardware-led
personalized skin care assessment 122 and / or type determination 130 for the
skin and
product experience sharing via ranking and rating 138 and / or comments
regarding
product effectiveness and experience (e.g. smell, taste , feel, texture,
color, etc.)
collection. The regimen 118 may be a dynamic recommendation based on users'
Page 59 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
collective inputs as well as experts' inputs on products that would best suit
the user's
individual needs.
[00222] The spa / salon based system 104 may generate product/ service
recommendations based on a skin state 158, offer one-click shopping based on
recommendations and enable SKU tracking, offer wellness packages such as
through a
contractual relationship, provide the ability to port regimen from spa to spa,
from home to
spa, and the like, enable optimization of regimens/ advising such as helping
practitioners
tailor the length of a procedure, enable development of targeted therapies,
enable clear,
visual communication to clients, generate effectiveness of products/services
reports, and
the like. Reports may be based on or comprise correlation with other users,
feedback on
regimen 118, modifications of a regimen 118, skin cycle monitoring, and the
like. A
medical practitioner based system, such as a dermatologist, general physician,
metabolist,
and the like, may enable pre-diagnosis, may link to the practitioner's
scheduling system,
may enable pre-pricing of services, may enable follow-up tracking, and the
like. A
cosmetic sales or retail based system 104 may enable integration with
inventory of
product enabling clearing of inventory. A handheld/portable device 108 may be
used at a
makeup counter, in a drugstore, at a home or trade makeup show/party, and the
like.
Users may purchase peripherals/accessories for the device, such as a holster,
charger, and
the like. Users may pay-per-scan or may have a subscription scanning service
and the
like. The system 104 may be based in health clubs, gyms, resorts, and the
like. A
cosmetics manufacturing / testing based system may enable skin state-based
product
design, targeting skin care samples to particular consumers, and the like. The
system 104
may be veterinarian based to monitor veterinary dermal- and non-dermal
concerns. The
system 104 may be based in a hospital, ER, military setting, and the like to
enable rapid
assessment of medical conditions, triaging urgent skin care, and the like. The
system 104
may be agriculturally based to enable application to fruits, vegetables, and
other such
agricultural products. The system 104 may be used in a battlefield scenario or
in an
austere environment, such as in space flight, air flight, underwater,
submarine, and the
like, to enable wound management, battlefield diagnosis and triage, and the
like. The
system 104 may be research based to enable comparing any materials and their
specific
Page 60 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
composition. Based on using the reading of the electrical property of the
light, a user
may be able to determine a similarity or difference between imaged material.
[00223] In an embodiment, determining a skin state 158 may comprise
employing an analysis 154. In an embodiment, the acquired data may be analyzed
by a
practitioner, such as a physician, dermatologist, spa employee, clinical trial
practitioner,
aesthetician, cosmetologist, nutritionist, cosmetic salesperson, and the like.
The
practitioner may analyze the data upon acquisition, visually, with the
assistance of an
algorithm 150, expert consult 128, database 115, and the like. In an
embodiment, the
practitioner may be remote from the location of data acquisition. In an
embodiment, an
algorithm 150 may be used to process and analyze 154 the reflected or re-
emitted light to
obtain spectroscopically resolved images, either automatically or under the
control of a
user, practitioner, and the like. For example, to obtain a spectroscopic image
of the
magnetic properties of the area only, an algorithm 150 may be used to generate
an image
of an area of concern using the difference between the reflected polarized
light, which
possesses the electrical properties of the area, and the reflected diffusion
light, which
possesses the electromagnetic properties of the area of concern. Algorithms
150 may be
rules-based software and processes to 1) analyze imaging evidence to obtain
skin health,
2) correlate skin health with ingredients, medicaments, and/or products that
may be best
suited for the determined skin health, 3) correlate skin health with peers in
a skin health
community, and 4) recommend and design personalized products based on skin
health
and/or other like users usage experience, 5) observe measurable changes in
skin health,
and the like. Algorithms 150 may be automated. Algorithms 150 may be used to
analyze
154 medical concerns, such as degree of suspicion of cancer, rash analysis,
and the like.
Algorithms 150 may be used to analyze 154 non-medical concerns, such as the
effectiveness of a medical, non-medical, or cosmetic regimen 118, a pimple
avoidance
regimen 118, a sun-protection effectiveness, an itch prevention cream, and the
like.
Algorithms 150 may be useful for correlating desired improvements with
ingredients and
most effective products for improving or maintaining the user's skin health.
The
algorithm 150 may utilize a calibration scale to determine the skin structures
imaged
based on the angle of incidence, wavelength and intensity of the light source,
an aspect of
the reflected or re-emitted light, filter parameters, and the like. Algorithms
150 may be
Page 61 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
useful for determining a dermascopic effect, a luminescence effect, a
spectroscopic
effect, and the like. For all algorithms 150, there may be an input, an
output, and
functional parameters to modulate the algorithm 150. In an embodiment,
analysis 154
may comprise examining at least one of: physical data and/or an image of the
material
using diffusion white light; physical data and/or an image of material using
light of a
single wavelength or multiple single wavelengths; physical data and/or an
image of the
material using polarized, reflected or re-emitted light of a certain angle;
physical data
and/or an image of the material generated using the difference between
diffusion white
light and polarized reflected or re-emitted light of a certain angle; physical
data and/or an
image of the material generated using the difference between light of a single
or multiple
wavelengths and polarized, reflected or re-emitted light of a certain angle;
and the like.
Algorithms 150 may be used with data and images generated by the device 108 or
third
party hardware 109. Algorithms 150 may be used with data and mages captured
using
any image capture device or technique, employing any kind of incident light,
such as
unpolarized light, polarized light, monochromatic light, diffuse light, white
light, multiple
single wavelength light, and the like. In embodiments, any captured data or
image may
be subjected to algorithmic analysis, as described herein.
[00224] In an embodiment, the algorithm 150 may be based on artificial neural
networks, non-linear regression, or fuzzy logic. For example, the algorithm
150 may be
used in skin lesion diagnosis based on a probabilistic framework for
classification. Two
kinds of data may be inputs to the neural network or to non-linear regression:
numerical
data such as intensity, size, numbers, and the like, and descriptive data such
as white,
gray, dark, and the like. Fuzzy logic may directly encode structured
descriptive data in a
numerical framework. Based on associative memories, learning algorithms 150,
and
adaptive control system behavior, neural and fuzzy machine intelligence may
enable
correspondence between input data taken from collected images and a
biophysical skin
state 15 8.
[00225] In an embodiment, the algorithm 150 may be based on fractal and
multi-fractal analysis of images based on biophysical and spatio-temporal
data. Both
digital image data and spectroscopic data of skin may be analyzed using
Hausdorff
dimensions (fractal property) and Kolmogorov's entropy (K-entropy). Then,
Page 62 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
spectroscopic data may be divided into spatio-temporal cells and analyzed as
multi-
fractal objects, yielding information about a level of functional disharmony
of skin
structures (epidermal and dermal). Structural data of these two analyses can
be correlated
to determinate a one-to-one correspondence between them. Once fractal
correlations
between digital image data and spectroscopic data of skin are established, it
may be
possible to obtain information about a functional state of skin structures
through multi-
fractal analysis of digital image data.
[00226] In an embodiment, an algorithm 150 may be for the analysis 154 of
data integrity. For example, an algorithm 150 may be able to determine if the
image has
been captured in high enough detail to render subsequent analyses reliable.
[00227] In an embodiment, an algorithm 150 may be useful for the analysis of
skin characteristics, obtaining the biophysical properties of the skin, and
determining a
skin state 158. The skin state 158 may capture a combination of underlying
skin structure
with time-based variance. Some variation may be predictable but some may be
based on
a transient condition like infection, sunburn, hormonal imbalance, and the
like. The
algorithm 150 may be able to measure aspects such as the structure, form,
concentration,
number, size, state, stage, and the like of melanocytes/ melanin, hemoglobin,
porphyrin,
keratin, carotene, collagen, elastin, sebum, sebaceous gland activity, pores
(sweat and
sebaceous), wrinkles, moisture, elasticity, luminosity, all forms of the
aforementioned,
such as derivatives, salts, complexes, and the like. The algorithm 150 may be
used to
make a quantitative assessment of clinical, medical, non-medical, and cosmetic
indications, such as moisture level, firmness, fine lines, wrinkle count and
stage, pore
size, percent of open pores, skin elasticity, skin tension lines, spots, skin
color, psoriasis,
allergies, red areas, general skin disorders and infections, or other skin
related concerns
for the user such as tumors, sunburns, rashes, scratches, pimples, acne,
insect bites,
itches, bleeding, injury, inflammation, photodamage, pigmentation, tone,
tattoos, percent
burn/ burn classification, moles (naevi, nevus), aspects of skin lesions
(structure, color,
dimensions/asymmetry), melanoma, dermally observed disorders and cutaneous
lesions,
cellulite, boils, blistering diseases, management of congenital dermal
syndromes, (sub)-
cutaneous mycoses, melasma, vascular conditions, rosacea, spider veins,
texture, skin
ulcers, wound healing, post-operative tracking, melanocytic lesions, non-
melanocytic
Page 63 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
lesions, basal cell carcinoma, seborrhoic keratosis, sebum (oiliness), nail-
and/or hair-
related concerns, and the like. The algorithm 150 may also be useful for the
analysis of
and obtaining the physical properties and composition of hair, nails,
biological
substances, gaseous substances, food, wine, water, liquid, metal, non-metals,
plastics,
polymers, and the like. Either manually or as determined by an algorithm 150,
a targeted
wavelength or wavelengths may be employed for specific endpoint measurements.
[00228] Either a specific wavelength or multiple wavelengths may be chosen
for the incident light or a specific wavelength or wavelengths may be isolated
by
filtering, as described herein. An algorithm 150 may determine the presence,
absence,
structure, form, and the like of particular skin structures based on the
properties of the
reflected or re-emitted light. For example, an algorithm 150 may detect which
axes/
angle the light is polarized on and compare this to signature emission spectra
of
individual proteins/ underlying skin structures. Each skin structure may have
a unique
signature pattern based on the electrical and magnetic contributions of
molecule(s)
present in the skin structure. The algorithms 150 may identify, analyze and
separate the
electrical and magnetic components of the unique polarization signal, as
described herein.
The signals may correlate with the aggregate conformation state of molecules
in the skin
structure. By comparing this signal to a standard calibration signal, aspects
of the
underlying skin structures may be determined. The standard calibration signal
may be
provided by a catalog of skin structures/ molecules and their specific
wavelength of
observation. The catalog may be developed by the technique described herein or
any
other spectroscopic technique. For example, to determine moisture levels in
the skin, an
algorithm 150 may determine a ratio of the reflected polarized light and
reflected
diffusion light and correlate the ratio with a moisture level. Ideally, close
to 100%
polarized light may be generated from reflections, however if a portion of the
reflected or
re-emitted light is diffusion light, such as 95% polarized, 5% diffusion, the
amount of
diffused light may be correlated with a level of moisture. Incident
unpolarized light may
interact with a skin structure and lead to varying amounts of polarization of
the reflected
or refracted light. This polarized reflected or refracted light strength may
be measured.
This polarization may be as much as 100 percent, however, the reflected
polarized
strength may even be less than 100% in some cases. The incident angle and the
imaged
Page 64 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
material would help determine the maximum strength possible for the
polarization of the
reflected or re-emitted light. It should be understood that there may be a
maximum
amount of polarization with a maximum of 100% for a particular incident angle,
but any
amount of polarization ranging from 0 to 100% polarized may be expected from
the light
reflected by any skin structure. The underlying cause for the differences in
reflection
may be due to the ratio of the captured and free water in the skin. To
determine
elasticity, an algorithm 150 may determine the concentration of elastin per
area of
concern. To determine luminosity, an algorithm 150 may combine moisture levels
and
skin color into a single, objective assessment. Objective measures may be
correlated
with an expert grading scale or other external measure. To determine firmness/
tightness,
an algorithm 150 may combine an assessment of collagen and elastin
concentrations in an
area of concern along with the activity of sebaceous glands (as measured by
number of
glands, percent open/closed, level of clog/ fill). The algorithm 150 may be
able to
overlay varying wavelengths and intensities and spectroscopic techniques, such
as
reflectance, excitation/emission, and the like. The algorithm 150 may be able
to process
and analyze 154 images collected by the device 108 or any other imaging device
using
unpolarized light, polarized light, or a combination thereof. The algorithm
150 may be
able to process and analyze 154 many different types of images, such as
thermoelectromagnetic (TEM) images or electromagnetic (EM) images, images
collected
with incident polarized light, traditional dermoscopy images,
spectroscopically resolved
images, conventional images, harmonized light images, and the like. The
algorithm 150
may be able to calculate a variance measurement of skin state 158 over time.
Determining a skin state 158 may also include, in addition to the processing
and analysis
of images of the skin for various measures and endpoints as described herein,
a visual
analysis of the images, user entered information, and third party information,
such as
lifestyle, smoking history, exercise habits, diet, allergies, and the like.
For example, a
user may enter anecdotal information, such as medication they may be taking,
recent
overexposure to sun, stage in a menstrual cycle, and the like.
[00229] Referring to Fig. 35, in an embodiment, an algorithm 150 comprise
spectral convolution of digital images taken with: 1) "angled white light", or
white light
incident on an angle sufficient to produce a polarized reflection; and 2) "non-
angled
Page 65 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
white light", or white light incident on an angle that produces substantially
no polarized
reflections. While the foregoing discussion will focus on skin as the primary
specimen, it
should be understood that any specimen, such as material characterized by
covalence
effects, ionic effects, and hydrogen bond effects, including skin, hair,
biological
materials, foodstuffs, liquid, wine, metallic materials, non-metallic
materials, and the like
may be specimens for the algorithm 150. Briefly, a digital image of a specimen
is
captured with non-angled light 3502 and angled light 3504, blue and red color
channel
histograms are generated for each image 3508, 3510 and are normalized to the
relative
intensity of the light, and the color channel histograms are correlated to a
wavelength
scale 3512, 2514. The spectral convolution proceeds in two steps. The first
step involves
subtracting, for each of the red and blue color channels, the color channel
histogram for
angled light from the color channel histogram for non-angled light 3518. Two
composite
histograms are generated, the blue color channel composite histogram and the
red color
channel composite histogram. The second step of the spectral convolution
involves
subtracting the blue channel composite histogram from the red channel
composite
histogram 3520. Continuing to refer to Fig. 35 throughout the discussion of
Figs. 36
through 43, the various steps of the algorithm will now be described in
greater detail.
[00230] Referring now to Fig. 36, a specimen 3604, which may be any suitable
material for imaging as described previously, may be illuminated with non-
angled white
light 3608 and angled white light 3610. As described previously herein,
varying the
angle of incidence affects the depth of penetration of the light to various
skin structures.
For each skin structure which may correspond to a particular known depth
within the
skin, there may be an angle of incidence which produces a polarized
reflection. By
analyzing the reflected or re-emitted light, either polarized 3614 and/or
diffusion 3612,
captured by an imaging device 3602, information on the underlying skin
structures
responsible for the reflection maybe obtained. The term "angled white light"
3610 refers
to incident white light that is directed towards the specimen at an angle
sufficient to
produce a polarized reflection. The term "non-angled white light" refers to
incident
white light that is not directed at a specific angle towards the specimen and
is diffuse. In
this case, the non-angled white light may produce reflected white light,
polarized light, or
a combination thereof. In an embodiment, reflected polarized light generated
by non-
Page 66 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
angled white light may be of a different characteristic than polarized light
generated by
angled white light.
[00231] Referring now to Fig. 37, Maxwell's color triangle, in Fig. 37B, may
facilitate an understanding of the nature of white light. Maxwell's color
triangle depicts
the complete visible color spectrum, with reference to specific wavelengths.
In order to
establish a mathematical coordinate system for the RGB color space, a
simplified version
is used with straight lines, shown in Fig. 37A. Each of the vertices of the
outer triangle
corresponds to an ideal color, either ideal green, red, or blue going
clockwise from the
top. Along the sides of a Maxwell triangle mixing of two of the three color
components
occurs with every possible proportion. As one travels from the side towards
the center,
the third primary color becomes increasingly important. Near the center at the
"equal
energy" point, E, a true white is seen, with radial axes extending to each of
the three
vertices. Mixing of the full intensity of red, green, and blue gives this true
white. Thus,
every point on the triangle is a result of a mixture of at least one of red,
green, and blue,
including the point representing white light. For example, the solid circle
3702
represents a point in color that is between pure/dark blue and pure white.
Similarly, the
dashed circle 3704 represents a point in color that is between pure/dark red
and pure
white. Using digital photos of white paper, the coordinate system may be
validated, as
represented by the internal triangle 3708. The internal triangle 3708
validates the system
when the sides are parallel to the limits of the color space lines of the
original coordinate
system. If they are not parallel, then the coordinate system is not valid.
[00232] Referring now to Fig. 38, an RGB histogram for each color channel is
generated for each of the images. An RGB digital image has three color
channels: red,
green, and blue. Each of these channels may be examined and analyzed
separately. A
blue color channel histogram is generated for the image taken with non-angled
white
light and another blue color channel histogram is generated for the image
taken with
angled white light. Similarly, a red color channel histogram is generated for
the image
taken with non-angled white light and another red color channel histogram is
generated
for the image taken with angled white light. For example, an automated system
may be
used to generate the histograms for each color channel, as shown in Fig. 38.
By simply
specifying which channel 3804 a user may wish to examine, a histogram 3802 may
be
Page 67 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
generated for that channel. The histogram may be normalized to the relative
intensity of
the light. Normalizing the histograms to the intensity of incident light is
important to be
able to process the histograms generated from different images. Referring now
to Fig.
39, the RGB color channel histograms are then correlated to a specific
wavelength scale
to generate RGB color channel spectral plots.
[00233] Referring now to Figs. 40A and B, the data from the pair of images are
then combined mathematically in two steps. In the first step, the blue color
channel
spectral plot generated from the image taken with angled white light 4004 is
subtracted
from the blue color channel spectral plot generated from the image taken with
non-angled
white light 4002 to generate a blue color channel composite spectral plot. The
two
spectral plots 4002, 4004 are shown first overlaid in Fig. 40A and then
subtracted in Fig.
41A. Similarly, the red color channel spectral plot generated from the image
taken with
angled white light 4008 is subtracted from the red color channel spectral plot
generated
from the image taken with non-angled white light 4010 to generate a red color
channel
composite spectral plot. The two spectral plots 4008, 4010 are shown first
overlaid in
Fig. 40B and then subtracted in Fig. 41B. Subtraction maybe facilitated by
aligning the
spectral plots by wavelength and mathematically subtracted the normalized
intensities at
each wavelength. For example, if the intensity is 0.005 at 470 nm for the blue
channel
spectral plot from angled white light and the intensity at the same wavelength
of the blue
channel spectral plot from non-angled white light is 0.003, the resultant
spectral plot
would comprise an intensity of -0.002 at 470 nm. The specific intensities and
wavelengths in the spectral plots reflect the specific properties of the
underlying material
and the angle at which the material was exposed to light.
[00234] Referring now to Fig. 42, the two color channel composite, normalized
spectral plots are then combined to create a unique spectral signature of the
specimen.
The normalized, composite blue channel spectral plot is subtracted from the
normalized,
composite red channel spectral plot. The scale is determined as a difference
in
wavelengths between the red and blue color images, starting from the darkest
point in
both colors. This scale is based on the mathematical coordinate system for
Maxwell's
color triangle. For example, and referring to Fig. 43, the lower part of
Maxwell's color
triangle is shown plotted out in Fig. 43B, with arrows indicating the
correspondence in
Page 68 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the plot with the position on the color triangle shown in Fig. 43A. Position 1
in the plot
corresponds to ideal blue in Maxwell's color triangle, position 2 corresponds
to true
white, and position 3 corresponds to ideal red. Points 1 and 3 are aligned
when
convoluting the composite spectral plots to obtain the spectral signature,
hence the unit
scale on the convoluted histogram is a difference of wavelength (e.g. 500-
400nm to 700-
400nm).
[00235] The spectral signature obtained may be analyzed for a number of
characteristics, such as number of peaks and troughs, amplitude and shape of
peaks and
intermediate structures and patterns, and the like. Various mathematical,
visual, and
algorithm processing techniques may be used to process and analyze the
spectral
signatures. The spectral signatures obtained for various specimens may be
unique, for
example, the spectral signature in Fig. 44A is for light skin while the
spectral signature in
Fig. 44B is for dark skin.
[00236] In an embodiment, the algorithm may be used for identifying metal
composition, purity, strength, and the like. For example, the spectral
signature may be
used to distinguish between metals. The spectral signature in Fig. 45A is for
a pure
metal, aluminum, while the spectral signature in Fig. 45B is for an alloy of
metals,
PbMnTe. The spectral signature may also be used to distinguish between similar
substances with different compositions. For example, the spectral signatures
in Fig. 45B
and Fig. 45C are both for the PbMnTe alloy but the alloy of Fig, 45B is of a
different
composition as compared to the one in Fig. 45A.
[00237] In an embodiment, the algorithm 150 may be used to analyze water
quality, composition, purity, and the like. For example, the spectral
signature for filtered
water is shown in Fig. 46A in comparison with the spectral signature for
highly purified
water, shown in Fig. 46B.
[00238] The spectral signature may further be enhanced by subtracting the
spectral contribution attributable to the source light from the reflected
light spectrum in
order to normalize the spectral signature to specific skin conditions. For
example the
spectral signatures in Figures 51 through 54 may be normalized by subtracting
the source
spectral signature from the reflected light spectral signature. By subtracting
the source
spectral signature, the resulting spectral waveform is normalized to only the
changes in
Page 69 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the skin from the interaction with incident light. In this way, specific type
of incident
light may be used which may be more amenable to detecting certain structures,
compositions, or conditions. In some embodiments, a spectral signature for the
subtraction of RGB histograms for angled light from non-angled light may be
calculated
and used to subtract from the final spectral signature for the material.
[00239] Other convolutions may be possible, such as for a yellow color
channel or some other color channel. Additionally, pre-determined convolutions
may
also be possible.
[00240] Referring now to Figure 51, positive intensities 5101 represent a net
reflection or emission at specific wavelengths based on material
characteristics while
negative intensities 5102 represent a net absorption from the source light's
spectral
signature. Negative intensity 5102 indicates no absorption of source light at
specific
wavelengths based on material characteristics. The source may be selected for
use in
examining specific biophysical or material criteria in order to produce a
specific
waveform for analysis.
[00241] Referring now to Figure 52, it is possible to determine changes in
skin
state 158 using spectral characteristics of specifically selected light
sources based on
specific biophysical criteria. Figure 52 shows a comparison of PB(S-O)
signatures
showing an example for differences between benign/healthy expected tissues and
diseased tissue. Changes, such as in the 462nm-485nm range in Fig. 52, such as
absorption or emission within the spectral diagram may correspond to
additional changes
in tissue processes, tissue activity, or presence of other molecules that
indicate a changed
state of skin. By measuring these changes, it is possible to determine healthy
and
diseased or disturbed states of the skin. The characterization of healthy
tissue based on
emission and or absorption may be determined at a specific reference
wavelength 5209
that is based on the source light selection. For example, the spectral
signature of healthy
skin 5201 using a specific source light shows little or no absorption or
emission in the
spectral range 5205. The spectral diagram shows normal spectral
characteristics 5206
right of the reference wavelength at line 5203. Additionally, characteristics
in the area
5207 to the left of the reference wavelength at the line 5204 indicate
diseased
characteristics due to re-emission or emission 5211, while the area 5208 to
the right of
Page 70 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the line 5204 indicates absorption 5210. The area 5207 corresponding to
wavelengths
462nm-485nm shows additional activity due to additional changes in tissue
processes,
activity, or presence of other molecules that indicated a changed state of
skin. The size
and shape of peaks, troughs, curves, frequency, spacing, specific sections of
wavelength
differences, and the like may also correspond to concentrations of molecules,
stages of
disease progression, skin characteristics, and the like.
[00242] In an embodiment, the algorithm 150 may only use reflected polarized
light due to increased selectivity for specific biophysical or material
characteristics. For
example and referring to Fig 53, the reflected polarized and/or emitted
polarized light
spectral signature 5302 may be much more sensitive to certain biophysical
characteristics
than simple white light convolution 5301. Figure 53 depicts the spectral
signatures for
malignant melanocytic lesions. The spectral diagram showing emission 5305 in
the
polarized 5302 spectral signature is much taller than the spectral diagram
showing
emission 5303 in the non-polarized 5301 spectral signature. Similarly, the
spectral
diagram showing absorption 5306 in the polarized 5302 spectral signature is
much deeper
than the spectral diagram showing emission 5304 in the non-polarized 5301
spectral
signature.
[00243] In an embodiment, the algorithm 150 may be used to analyze healthy
and non-healthy or malignant skin. For example, the spectral signatures for
healthy, non-
pigmented skin 5401 and 5402, healthy pigmented skin 5403 and 5404, and
malignant
pigmented skin 5405 and 5406 are shown in Fig. 54. Both polarized (bottom) and
white
light (top) spectral signature convolutions are shown for purposes of
comparison. The
spectral signature of normal, healthy skin 5401 and 5402 shows very little
absorption or
emission relative to the source light spectrum around referent wavelength
485nm.
Similarly, the healthy, benign pigmented skin lesion 5403 and 5404 shows very
little
absorption or emission to the left or right of the reference wavelength 485nm.
The
malignant tissue, however, clearly shows absorption and emission effects
around the
referent wavelengths with higher amplitudes and shifting of the spectral
diagram peaks
and valleys.
[00244] In embodiments, these spectroscopic techniques may be useful for a
variety of analytical tests where the test substrate comprises a light-
sensitive component.
Page 71 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00245] In an embodiment, elements of the waveform may be tagged and
tracked over time in order to track changes in the characteristics of the
material or
specimen, such as peaks, troughs, curves, frequency, spacing, specific
sections of
wavelength differences, and the like.
[00246] In an embodiment, the algorithm 150 may be incorporated for
automated measurement as part of an integrated device that conducts surface
analysis,
such as a skin imaging device or metal testing device. In an embodiment, the
algorithm
150 may be part of a remote analysis system whereby a surface imaging device
may
capture images and send them to a processing center where the algorithmic
computations
may be made.
[00247] In an embodiment, the algorithm 150 may be used for the analysis of
hair in order to determine the health of hair follicles, composition, and the
like.
[00248] In an embodiment, the algorithm 150 may be used for the counterfeit
analysis of money. For example, a unique signature may be created for each
series of
appointment and/or issue.
[00249] In an embodiment, the algorithm 150 may be useful for the analysis of
anti-perspirant effectiveness. In certain cases, axillary odor may be an
indication of
sickness or some other medical condition, such as lymphoma, apocrine gland
sweating,
hyperhidrosis, hydradenitis suppurativa, or other sweat related medical
problems. The
algorithm 150 may be useful in determining a scale of deodorant effectiveness
based on
an individual's specific sweat gland activity and type. The algorithm 150 may
enable
measuring the activity of sweat glands located in the axilla, feet, palms, and
the like. The
algorithmic analysis may enable the classification of sweat glands and may
enable the
suggestion of appropriate products/ingredients for treatment. The algorithm
150 may be
able to determine the effectiveness of an anti-perspirant based on the impact
on sweat
gland activity.
[00250] In an embodiment, the algorithm 150 may be useful for determining a
veterinary condition, such as Mad Cow disease. For example, imaging the tongue
of a
cow or any mucosal or dermal area where the disease may manifest may allow for
the
detection of a disease state using the algorithm 150. White light imaging, as
described
Page 72 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
herein, in combination with UV imaging may facilitate detection of a Mad Cow
disease
state.
[00251] In an embodiment, the algorithm 150 may be useful for monitoring
post-operative cosmetic concerns, such as stretch mark progression and
diminishment,
and the like.
[00252] In an embodiment, the algorithm 150 may be useful for predicting and
monitoring secretion from the mammary glands of lactating women. If milk
production
is predicted to be low based on the algorithmic analysis, suggestions may be
made to
increase milk production.
[00253] In an embodiment, an algorithm 150 for determining a skin state 158
may facilitate measuring, tracking, and monitoring a skin state 158 as well as
the
effectiveness of a regimen 118, topical and/or systemic therapies, avoidance
routines,
diet, and the like. For example, the skin state 158 may be measured at
intervals and
current measurements may be compared to previous measurements to determine
skin
health changes. As will be further described herein, the results from the
algorithm 150
may feed into a recommendation engine to provide feedback and modifications to
aspects
of the regimen 118.
[00254] In an embodiment, an algorithm 150 for determining a skin state 158
may enable a diagnosis. The diagnosis may be an early diagnosis by
distinguishing
between critical and non-critical indications. For example, the algorithm 150
may be
able to distinguish between a minor sunburn and a third degree sunburn
requiring medical
attention. Use of the device 108 to capture images enables a user to readily
transmit the
images to any practitioner for remote assessment, to track progression of a
skin condition,
rapidly compare images to previous images, other user images or third party
images, such
as images in a dermascopic database 115, and the like, and to make an
immediate
assessment with no need for historical knowledge, and the like. Historical
data and the
results of modeling tools 132 may be compared to the images to assist in
analysis, either
by an algorithm 150, a practitioner, or a practitioner employing an algorithm.
Also, in
addition to images, user input in the form of audio, video, or text anecdotes
describing the
issue, such as a level of pain, a sensation of heat, an itchiness, and the
like, may be useful
in analyzing the images to determine a diagnosis. The algorithm 150 may enable
non-
Page 73 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
linear regression, such as principal component analysis (PCA), which may be a
biomedical analysis used in conjunction with spectrometric analysis for
analyzing
medical and health conditions. The algorithm 150 may enable a simple pattern
analysis
for diagnosis. The algorithm 150 may be able to determine the thermo- and
electroconductivity conditions of skin lesions. In an embodiment, the
algorithm 150 may
be able to diagnose a melanocytic lesion by examining the images for the
relationship of
changes in collagen and porphyrin, as a change in collagen but not porphyrin
may
indicate a change from a normal lesion to a dysplastic lesion. The skin state
158 may be
compared with a table of indicators for various types of lesions. In an
embodiment, the
algorithm 150 may be able to diagnose UV damage. UV damage may be difficult to
assess from a conventional superficial view as UV damage may be present even
in
wrinkle-free skin. However, UV damage may be assessed by examining skin
structures
for an increase in melanin production; global distribution, damage and count
of
superficial blood vessels; change in hemoglobin count: changes in the
thickness of the
epidermis; changes in the quantity and global distribution of collagen, and
the like. In an
embodiment, diagnosis may not require processing the border of the lesion, as
it may not
be a key factor in final analysis of the skin lesion. In an embodiment, the
algorithm 150
may be able to diagnose oral cancer.
[00255] In an embodiment, an algorithm 150 for determining a skin state 158
may enable cosmetics manufacturing validation or cutaneous clinical trials.
For example,
a skin state 158 may be determined prior to medical, non-medical, skin care
product or
cosmetics application and a time lapse series of images may be acquired to
track the
medical, non-medical, skin care product, and cosmetics effectiveness.
[00256] In an embodiment, there may be methods for storing, handling,
integrating, and analyzing a skin state 158. The skin state 158 may be stored
in the
device 108 itself, on a PC, in a central server, a salon record, an e-medicine
record, a
medical repository, a cosmetic clinical studies database 115, a mobile device,
and the
like. The device 108 may communicate with a user interface 102, an online
platform
120, a mobile platform 124, and the like to upload, deliver, share, and/or
port images,
analysis 154, skin states 158, data, track history, user profiles, and the
like, as will be
further described herein. For example, a user may use a device 108 embodied in
a mobile
Page 74 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
device to capture an image of the skin and upload it to a mobile platform 124
for analysis
154 to determine a skin state 158. In response, the user may receive a
personalized
regimen 118 for sun protection given the user's skin state 158. Other factors
that may be
used to determine the regimen 118 may be the current UV Index, time of day,
location,
kind of sun protection product the user prefers, and the like. In the same
example, the
user may have already obtained a skin state 158 determination and they need
not upload a
new image but simply request a regimen 118 recommendation from the mobile
platform
124 given the already determined and stored skin state 158. Once a skin state
158 is
determined, it may be accessible by and/ or integrated with any element of the
user
interface 102, online platform 120, mobile platform 124 and the like. A user
may choose
to share the skin state 158 as part of a practitioner record 180.
[00257] In an embodiment, an algorithm 150 for determining a skin state 158
may enable an analysis of differences and similarities among peers. The
algorithm 150
may determine peers of a user who may be most like them in terms of skin state
158 or
other criteria such as gender, age, ethnicity, behaviors such as smoking,
working
outdoors, and the like, diet, regimen 118, and any other identifying factors.
The
algorithm 150 may be able to interface with an online platform 120, third
party database
115, or third party service provider 111 to access skin states 158 and
demographic
information for comparison. For example, a user may wish to know what other
women
in their mid-30's of the same skin color are using for foundation. By
employing the
algorithm 150, a user may be able to determine their own skin color, identify
peers
according to the search criteria, and view details on their peers' regimen 118
or the
results of the specific search query 103. The algorithm 150 may enable grading
of the
skin relative to a peer group. Using the algorithm 150, a user's skin state
158 may be
compared to a previously defined skin state 158 in order to monitor the skin
state 158
over time. A user's skin state 158 may also be compared to the skin state 158
of other
individuals or groups of individuals to identify peers whose skin state 158 is
closest to the
user. Once a peer, such as a similar individual or group, is identified, the
system may
display the skin care products and/or skin care regimen that is effective for
the peer.
Similarly, any comparison among users may be made by the system, such as a
comparison of at least one of age, gender, location, climate, skin color,
ethnicity, and the
Page 75 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
like, to identify a peer. In an embodiment, as the device 108 captures data
from users and
determines skin states 158, the information may be fed back into the algorithm
150 to
further enhance the peer identification and product recommendation process.
[00258] In an embodiment, an algorithm 150 for determining a skin state 158
may enable prediction/simulation tools 132. Having determined a skin state
158, an
algorithm 150 may be able to simulate progression of aging, simulate skin care
treatment
effects and skin care and cosmetic regimens 118, simulate progression of a
skin
condition, and the like. Referring to Fig. 6, a user may use a user interface
102 to access
the simulation tools 132. In the example, the image of an entire face may be
used but it
should be understood that simulation tools 132 may be used to generate
simulations for
any size area of concern. After selecting or capturing a starting image, a
user may
indicate the kind of simulation they would like to perform. For example, the
user may
like to perform a simulation of aging only, or a simulation of aging and
treatment effects.
The simulation tool 132 may return data on overall appearance, wrinkle count,
elasticity,
luminosity, moisture, product usage simulation, and the like. For example, the
output
may also include a split image with the original face on one half and a new
simulated
output on the other half.
[00259] In an embodiment, an algorithm 150 for determining a skin state 158
may enable skin cycle monitoring 140. By monitoring skin at determined
intervals, skin
conditions with a cyclical nature may be monitored, predicted, pre-empted and
the like.
For example, skin conditions associated with a season, weather, pollen count,
hormone
level, environmental condition and the like may be identified and monitored by
a skin
cycle monitor 140.
[00260] In an embodiment, an algorithm 150 may be used to generate
searchable and/or indexable tags to associate with images and may take
advantage of
image tagging. Images may be tagged with information relating to the content
of the
image, such as information relating to a skin state, a skin condition, a
gender, an
ethnicity, an age, a regimen, a treatment, and the like. The information may
be gathered
by algorithmic analysis, user input, visual inspection of the image, and the
like. An
algorithm 150 may be used to perform a search 103 using the information
associated with
the image as a search term. In embodiments, the information may be stored
separately
Page 76 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
from the image, such as an entry in a user profile, or may be stored in
association with an
image. In an embodiment, a search 103 may be performed against information or
images
from other users' or a third party database 115 to identify similarities or
differences in
images or information. For example, a user may use information to search for
peers with
a similar skin condition in order to determine what to expect as the condition
progresses.
In another embodiment, the search 103 or query for advice or recommendation
from
experts may be performed against product information 190, wellness information
192,
skin care regimens 118, third party experts 105, and the like. For example, a
user may
use information to search for product information 190 indicating an
effectiveness of a
product for the user's skin condition. In an embodiment, the search 103 may be
performed to determine an availability of a product, an inventory of a
product, a price of
a product, and the like. For example, a user may use the information to search
a store
catalog for a specific product that may be effective for the user. In the
example, the user
may be pale skinned and be interested in identifying an inventory of a self-
tanning
product formulated specifically for pale skin. In an embodiment, the image
itself may be
used as a search query 103. For example, the image itself may be used to
search a
database 115 of skin images. In an embodiment, images and information entered
into the
system 104 may be leveraged to develop new algorithms 150 for enhanced
diagnosis.
For example, algorithms 150 may be developed for non-skin specific diseases
with
dermal manifestations, such as rheumatoid arthritis.
[00261] In an embodiment, an algorithm 150 may be useful for analyzing
product characteristics. For example, an algorithm 150 may be able to take
product
ingredients and match the product up with a projected effectiveness on a
particular skin
state 15 8.
[00262] In an embodiment, an algorithm 150 may use RGB color analysis.
The algorithm may employ standard RGB analysis and correlation with skin
structures in
determining skin phototype. The calculation of parameters for determining skin
phototype is fast and the skin phototype can be found in a very short period
of time using
a simple skin and cosmetic parameters classification routine.
[00263] Exemplary embodiments of the present invention are directed to a
method and system for determining skin characteristics and cosmetic features.
The
Page 77 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
method and system provide a minimal error and speed efficient skin analysis.
The present
technique describes a method and a system for determining a skin phototype of
acquired
digital image in a Red Green Blue (RGB) color system.
[00264] In an exemplary embodiment of the present invention, a method for
determining skin characteristics and cosmetic features using color analysis
includes a step
of analyzing color of skin images in a pixel by pixel manner in a Red Green
Blue (RGB)
color system for an acquired digital image. The colors obtained in device
dependent RGB
color system are then converted into device independent standard RGB color
system
(sRGB) which will be used in subsequent color analysis. The step of analyzing
color of
skin images in a pixel by pixel manner in a sRGB color system for an acquired
digital
image comprises analyzing a picture of a part of a person's skin by generating
a table of
most frequent colors appearing in the picture.
[00265] In this embodiment of the invention, the sRGB color system has been
used for image analysis. Determination of other skin characteristics (e.g.
elasticity,
melanin, oil concentration etc.), melanoma, skin related tumors and skin
related disorders
may require image analysis based on various color systems such as YIQ, YCbCr,
L*a*b*, L*u*v* and HSL/HSV. The enhancement of the current algorithm 150 may
include at least one of these color systems and its/their correlation with
presented sRGB
analysis. This will most likely lead to in-depth refinement and overall
accuracy of the
current results as well as further embodiments of the present invention. Apart
for the
human skin related issues, this method of image analysis is also applicable to
any content
whether it be animals, products, plants or any other material whose surface
needs to be
analyzed by a digital image.
[00266] A method for determining skin characteristics and cosmetic features
using color analysis includes a step of generating a sample of most frequent
sRGB colors
responsive to analyzing color of skin images in a pixel by pixel manner in the
RGB color
system for the acquired digital image after converting colors obtained in a
device-
dependent RGB color system into a device-independent standard RGB color system
(sRGB). The step of generating a sample of most frequent sRGB colors
responsive to
analyzing color of skin images in the sRGB color system for the acquired
digital image
comprises preserving a plurality of sRGB color values.
Page 78 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00267] A method for determining skin characteristics and cosmetic features
using color analysis includes a step of modeling the standard R, G and B
component
color distribution with Gaussian probabilistic distribution with estimated
parameters
(expected value and standard deviation) on the generated sRGB color sample for
the
acquired digital image further including approximating colors on the generated
sRGB
color samples by a Gaussian normal distribution. In accordance with an
exemplary
embodiment of the present invention the step of approximating colors on the
generated
sRGB color samples by a Gaussian normal distribution comprises approximating
colors
on the generated sRGB color samples by a superposition of a plurality of
Gaussian
normal distributions.
[00268] A method for determining skin characteristics and cosmetic features
using color analysis includes a step of generating a phototype of the skin
through a
decision tree unit responsive to the estimated distribution model parameters
colors. The
phototype of the skin is generated according to a corrected Fitzpatrick
classification, or
any other applicable color classifier. In accordance with an exemplary
embodiment of the
present invention, the step of generating a phototype of the skin according to
corrected
Fitzpatrick classification includes generating a phototype of the skin
according to a skin
type scale which ranges from very fair skin to very dark skin.
[00269] According to an exemplary embodiment of the present invention, the
system for skin phototype determination using photograph analysis includes a
subsystem
for determination of cosmetic features for a human element and a veterinary
element. The
cosmetic features further include features pertaining to hair, nail and skin.
[00270] According to an exemplary embodiment of the present invention, the
image of the skin sample of a person's body can be captured by any digital
camera. The
acquired digital image sample of the person's skin may be analyzed in a pixel
by pixel
manner in the RGB color system. After the conversion of colors from a device-
dependent
RGB color system into a device-independent standard RGB color system (sRGB), a
table
of most frequent sRGB colors which appear on the image may be generated.
According
to an example, the generated table may consist of 256 most frequent colors
which appear
on the image of the person's skin. The color samples obtained from the image
may be
approximated by a Gaussian normal distribution (or a (scaled) superposition of
few
Page 79 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
Gaussian normal distributions). Therefore the estimates of expected value
(using
weighted mean) and standard deviation (using unbiased (n-1) method as the
precise
expected value is unknown / estimated) for each of the acquired digital images
may be
evaluated. The phototype of the skin may be determined through a decision tree
with the
estimated expected value and standard deviation. Fitzpatrick classification
may be used
for categorizing a skin phototype in accordance with a skin type scale which
ranges from
very fair skin to very dark skin.
[00271] Referring to Fig. 58 is a flowchart 5800 illustrating a process for
determining a skin phototype of an acquired digital image of a part of a
person's skin is
shown. The process starts at block 5810 where in an image of a part of a
person's skin is
captured. The image capturing device may be a digital camera or the like.
Processing
flow continues to logical block 5820 wherein analysis of acquired digital
image is done in
a pixel by pixel manner in a RGB color system. After converting all colors
from the
device-dependent RGB color system into a device-independent standard RGB color
system (sRGB), a table of most frequent colors which appear on the acquired
digital
image may be generated using a quantization technique at block 5830. In
accordance with
an example embodiment of the invention, at block 5840 a plurality of sRGB
color
values/samples generated between a range of values 0 and 255 may be preserved
for
further analysis. This range of values has been proven to be more convenient
for skin
type determination than the one between 0 and 1. The transformation from one
to another
can be done simply by dividing the values with 255 and vice versa. In the next
stage 5850
and 5860 approximations of colors on the samples are done by Gaussian normal
distribution, at block 5860 the estimates expected value and standard
deviation are
evaluated. Finally at block 5870, the phototype of skin of the acquired
digital image is
determined according to corrected Fitzpatrick classification using decision
tree.
[00272] According to an exemplary embodiment of the present invention, the
decision tree may be an algorithm wherein the estimated expected value and
standard
deviation are equated to the values of Fitzpatrick classification/notation
values in
determining the phototype of the skin. The effectiveness of this approach may
be seen in
research regarding parametric skin distribution modeling for skin segmentation
/
detection.
Page 80 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00273] Referring to Fig. 59, a diagram depicting a pixel view of an acquired
digital image of a sample of person's skin is shown. The image of a sample of
a person's
skin is captured under white emitting light. The image may be captured by any
digital
camera and the like under white emitting light. An analyzer coupled to the
image
capturing device may analyze the acquired digital image in a pixel by pixel
manner in the
RGB color system. The analysis of the acquired digital image in a pixel by
pixel manner
in the sRGB (after RGB to sRGB color system conversion) is not only limited
for
determining skin phototype but also may be useful for other purposes like
classification
of other skin characteristics (e.g. elasticity, melanin, oil concentration
etc.), melanomas
and other skin tumors/disorders and the like.
[00274] Digital images captured from a sample of person's skin are usually
given in the RGB color system. The present technique employing an algorithm
150 for
determining skin phototype in one aspect is dependent on this color system,
although
device independent due to conversion to sRGB color system. The calibration of
the image
capturing device like digital camera or the like should be taken into
consideration
carefully, so that the eventual color offset could be corrected. The color
offset correction
in the present technique can be implemented from any known techniques in the
previous
art and color offset correction can also be implemented in software used in
the present
technique in determining skin phototype.
[00275] Referring to Fig. 60, a diagram depicting a pixel view of the acquired
digital image of a part of person's skin after quantization is shown. The
image of the
sample of the person's skin is captured under the white emitting light. The
image may be
captured by any digital camera and the like under white emitting light. The
analyzer
coupled to the image capturing device analyzes the acquired digital image in a
pixel by
pixel manner in the RGB color system. The analysis of acquired digital image
in a pixel
by pixel manner in the sRGB (after RGB to sRGB color system conversion) is not
only
limited for determining skin phototype but also may be useful for other
purposes like
classification of other skin characteristics (e.g. elasticity, melanin, oil
concentration etc.),
melanomas and other skin tumors/disorders and the like. Color quantization or
color
image quantization is a process that reduces the number of distinct colors
used in an
image, usually with the intention that the new image should be as visually
similar as
Page 81 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
possible to the original image. Color quantization is critical for displaying
images with
many colors on devices that can only display a limited number of colors,
usually due to
memory limitations, and enables efficient compression of certain types of
images.
[00276] An image quantization technique may be applied to the captured
image. A table of 256 most frequent colors which appear on the acquired
digital image of
the part of person's skin may be generated using a sampling device coupled to
the
analyzer. The acquired color samples from a digital image may be preserved in
the
sRGB color system. In accordance with an example embodiment of the present
invention,
the generated color samples may be preserved in their range of values between
0 and 255
in the sRGB color system. This range of values has been proven to be more
convenient
for skin type determination than the interval ranging between 0 and 1.
[00277] Accordingly colors on the samples may be approximated by Gaussian
normal distribution (or a (scaled) superposition of few Gaussian normal
distributions)
through an approximating device coupled to the sampling device. Further the
estimates of
expected value (using weighted mean) and standard deviation (using unbiased (n-
1)
method as the precise expected value is unknown / estimated) for each of the
acquired
digital image may be calculated with approximating device coupled to the
sampling
device.
[00278] Usage of an algorithm 150 of the present technique is depicted in Fig.
61 and Fig. 62 and the algorithm 150 for RGB color analysis is depicted in
Fig. 63.
[00279] Referring to Fig. 61, a diagram depicting a Histogram / Distribution
of
standard R, G and B colors on one of the taken photographs of a patient whose
skin
phototype is classified as type III by Fitzpatrick, and their Gaussian normal
approximation / hull is shown. The relevant estimates are R (expected value
of red)
=171.1304 and B (expected value of blue) =135.3047 , for example. The
estimates are
compared with the decision tree described below for determining skin
phototype. The
phototype of skin is determined according to corrected Fitzpatrick
classification.
Fitzpatrick Skin Typing Test questionnaire (skin type scale) which ranges from
very fair
(skin type I) to very dark (skin type VI) is often used to determine skin
phototype.
[00280] Dermatologists use the Fitzpatrick Classification Scale to classify a
person's complexion and tolerance to sunlight. In accordance with an exemplary
Page 82 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
embodiment of the present invention, the Fitzpatrick scale classifies skin
types from Ito
VI.
[00281] Type I - Very white or freckled skin, always bums with sun exposure
(very fair; often in people with red or blond hair and blue eyes)
[00282] Type II - White skin, usually bums with sun exposure (fair; often in
people with red or blond hair and blue, green, or hazel eyes)
[00283] Type III - White or olive skin tone, sometime bums with sun
exposure (fair; seen in people with any hair or eye color)
[00284] Type IV - Brown skin, rarely bums with sun exposure (common in
people of Mediterranean descent)
[00285] Type V - Dark brown skin, very rarely bums with sun exposure
(common in people of Middle-Eastern descent)
[00286] Type VI - Black skin, never bums with sun exposure
[00287] The images of skin are captured under white emitting light with an
image capturing device, such as a digital camera, video camera or the like. An
analyzer
analyzes the captured image pixel by pixel of a part/sample of a person's
skin. A
sampling device coupled to the analyzer generates a table of 256 most
frequently
occurring colors in the captured image. The acquired color samples from
digital image
are preserved in the sRGB color system. The generated color samples are
preserved in
their range of values between 0 and 255 in the sRGB color system. An
approximating
device coupled to the sampling device may calculate the estimates of expected
value
(using weighted mean) and standard deviation (using unbiased (n-1) method as
the
precise expected value is unknown / estimated) for each of the acquired
digital images. A
decision tree coupled to the approximating device determines the skin
phototype. This
imaging turns out that expected values of R and B may be sufficient for
determining skin
phototype according to following decision tree. An exemplary embodiment of the
present
invention illustrates below.
Page 83 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
1, (uR <_MR)A(ue MBU)
2, (MRI <_ uR < MRU) A (Ma' <_ P,3 < MeU )
3, (MRI <_ PR < MRU) A (Ma' <_ ue < MeU )
4, (MRI <_ PR MRU) A (Me" <_ u8 <_ MeU)
5, (MRI < PR < MRU) A (Me' < P,3 < MeU )
Phototype = 6, (MRI <_ PR) A (Pe MB'U )
1/2, (MR2'I < ,U, < MR2'u) A (Ma 2'I < ,u8 < M82'u )
2/3, (MR 3"C 1RMRSu)A(Me3"<_!~8Mesu)
3/4, (MR4,I <- PR MR4u)A(M84I </u8 ~M84u)
4/5, (MR5,I <uR MR5u)A(M85I </u8 ~M85u)
5/6, (MR6,I <- PR MR6u)A(M86I </u8 ~M86u)
Further examination, all other cases
Mn'LOr' n=1,2,3,4,5,6,1/2,2/3,3/4,4/5,5/6
[00288] The values R,e have been
determined from the images analyzed by using the programmed neural network.
[00289] Fig. 62 is a diagram depicting a Histogram / Distribution of R, G and
B colors on one of the patient's photographs whose skin phototype is
classified as type
VI by Fitzpatrick, and their Gaussian normal approximation / hull. Here the
relevant
estimates are R (expected value of red) =189.7173 and B (expected value of
blue)
=103.537, in accordance with an example embodiment of the present invention.
The
estimates are compared with the decision tree mentioned above for determining
the
phototype of the skin.
[00290] Referring to Fig. 63, a flowchart 6300 illustrating an algorithm 150
for determining the skin phototype according to the estimated values of
mathematical
expectation for standard R and B color in sRGB color system is shown. The flow
chart
describes the algorithm 150 developed in accordance with the present technique
wherein
the photograph of a part of person's skin is captured with a digital camera or
the like
under white emitting light at logical block 6310. At logical block 6320 the
captured
digital image is analyzed in a pixel by pixel manner in the RGB color system.
A
quantization technique is employed for analyzing the captured image in a pixel
by pixel
Page 84 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
manner in the sRGB color system at logical block 6330. The color samples
obtained from
the image can be approximated by a Gaussian normal distribution (or a (scaled)
superposition of few Gaussian normal distributions). Therefore the estimates
of expected
value (using weighted mean) and standard deviation unbiased (n-1) method (as
the
precise expected value is unknown / estimated) for each of the acquired
digital images
may be evaluated. Now at logical block 6330 the phototype of the skin is
determined
according to the decision tree.
[00291] As will be appreciated by a person skilled in the art, the various
implementations of the present technique provides a variety of advantages.
Firstly, the
present technique determines skin phototype using regular low-cost digital
photography
equipment under standard environmental conditions. Secondly, the analysis
performed on
the captured digital image may be useful in recommendation of cosmetic product
and
medical or surgical purposes. Thirdly, the picture quantization algorithm and
calculation
of estimates expected value and standard deviation are fast, this makes it
easier to
determine skin phototype in a short span of time using a simple routine.
Fourthly, the
analysis performed may be useful for classification of other skin
characteristics (e.g.
elasticity, melanin, oil concentration etc.), melanomas, skin tumors or
disorders and the
like.
[00292] In an embodiment, new algorithm 150 development by practitioners,
users, service providers 111, and the like may be enabled by a software
development kit
that anyone could use to develop new algorithms 150 and APIs 154 for the
device 108.
[00293] Referring now to Fig. 3, in an embodiment, a process for collecting
images, performing skin analysis, communicating findings and scheduling follow
up, if
required may commence with image capture by a user using a device 108. The
user may
also answer questions or provide additional details regarding a user-entered
imaging,
cosmetic regimen, area of concern, or the like. Using the user interface 102,
the data may
be communicated to an analyst 304 or a computer for analysis 154 by any
communication
method, such as over a network, the Internet, wirelessly, and the like. In
certain
embodiments, as the data are collected or communicated, a payment system 302
may be
accessed by the user. In the example shown, an insurance company may access
the data,
however, payment may be effected or requested by any interested entity such as
a one-
Page 85 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
time payment by the user, a subscription by the user, a third party service
provider 111, a
platform 120,124, a practitioner, and the like. The entered data may be
analyzed by the
analyst, by software in real-time, by analysts assisted by software
assistance, and the like.
An initial analysis may be to determine data integrity. In instances where the
data do not
pass the integrity test, it may be communicated back to the user. The
analyst's
assessment may be assisted by software that uses an algorithm to determine
type of
condition and/or recommended care/treatment. Historical analysis and data, and
modeling tools may be used to assist the analyst's assessment. Relevant
parties
(company personnel, payment providers, physicians, medical personnel, users,
amongst
others) may receive the analysis and/or user specific details for follow up or
other actions
that may be required. The analysis 154 may be stored 308 by the system and/or
submitted to a practitioner for approval 310. In embodiments, storage 308 may
require
practitioner approval 310. A test of the severity 312 may determine the
selection of an
appropriate method of communication with the user. If the result of the test
312 is
positive, the user may be notified immediately by a preferred communication
method,
such as telephone, instant message, and the like. If the result of the test
312 is negative,
the user may similarly be notified, however, the notification may take a less
urgent route,
such as by email or postal mail. In any event, the software tool may recommend
an
appropriate communication method and media, based on the assessment and may
populate preset templates with the information/message to be communicated. In
addition, notification by any means may also include a notification of
practitioner
availability. The analysis 154 may trigger a practitioner availability /
scheduling tool.
For example, prior to transmitting the results on severity 312 to the user, a
practitioner
availability may be assessed and transmitted simultaneously. The user may
access
availability and scheduling tools in order to obtain and confirm an
appointment time.
[00294] In an embodiment, a user interface 102 for a skin analysis system 104
may be used to interface with the device 108, store images, deploy algorithms
150, track
a skin state 158 by keeping track of images from any number of areas of
concern, the
interval between image capture, a projected next image capture date,
communicate
findings to a practitioner, interact with simulation tools 132, skin type
determination tools
130, a skin cycle monitor 140, practitioner availability / scheduling tools,
and the like.
Page 86 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00295] In embodiments, the user interface 102 may be operable as an
application running on a device 108, a computer, server, kiosk, or the like,
on an online
platform 120, on a mobile platform 124, and the like. Any and all aspects of
the user
interface 102 described herein may be applicable to the user interface 102
running in any
environment.
[00296] In an embodiment, the user interface 102 for the device, as will be
further described herein, may be integral with the device 108, such as
embodied in the
keypad of a communications device or a series of buttons, switches, keys and
the like
disposed on the device 108, or may be external to the device 108, such as
software
running on a computer, on the Internet, on an intranet, on a mobile
communications
device, on an online platform 102, on a mobile platform 124, and the like. The
user
interface 102 may be used to modify a setting of the device 108, such as the
magnification, light source, light intensity, wavelength of light, angle of
light, electrical
and magnetic properties of the light, positioning of sensor, duration of image
capture,
image size, data storage, data transmittal, and the like.
[00297] Referring now to Fig. 5, the user interface 102 may organize and index
images captured by date, area of concern, skin state, and the like. For
example and
without limitation, as seen in the Fig. 5, four images captured from the same
area of
concern are indexed by their number within the series. In an embodiment, the
user
interface 102 may show in real time the field of view on the skin being imaged
as well as
populate the user interface 102 with the images once taken or once submitted
by the user.
The user interface 102 may keep track of the first image, latest image, next
image, and
the like. The user interface 102 may allow users to shuffle through image s
and use the
images as a basis for simulation 132, as described herein. The user interface
102 may be
used to set a reminder for next image capture. The user interface 102 may be
used to
create a report of the images and skin state 158. The user interface 102 may
be used to
transmit the report to a practitioner. In an embodiment, the user interface
102 may be
used to launch a skin type test. In an embodiment, the user interface 102 may
depict a
form of a body. As a user interacts with the depiction of the body, such as
with an
indicating device, the portions of the body that have been imaged may be
linked with the
images such that the images may pop-up or be otherwise accessed. The user
interface
Page 87 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
102 may be adapted to collect data from the user in response to prompts. The
user
interface 102 may employ an algorithm 150 to check the integrity of the
captured images.
The user interface 102 may guide the user in capturing images and providing
user input
in association with the images.
[00298] In an embodiment, the user interface 102 may interface with host
hardware 108 or third party hardware 109. Hardware 108, 109 may comprise an
imaging
device that may connect with a computer, online platform 120, mobile platform
124, and
the like via the user interface 102 and enable users to capture an image that
enables
measure various skin health, condition and type parameters. The hardware
device
108,109 may be a standalone device or connect via or be embodied in a
computing device
of either medical or non-medical use. The user interface 102 may guide the
connection
process for the hardware device 108, 109. The device 108, 109 may store
images, reports
and recommendations generated and maintain a repository of the image, all as
part of a
skin health record 121. It may enable a systematic storing of the skin health
record 121.
Third party hardware 109 may comprise devices such as moisture sensors,
cosmetic
analysis machines, dermascopes, cameras, x-ray machines, MRIs, medical record
providers and software, web cameras, communication devices, and the like.
Third party
hardware 109 may connect to the system 104 seamlessly to enable the user to
gain a
better analysis, and share such sets of data with other experts or users.
[00299] In an embodiment, the user interface 102 may enable type
determination 130. Characteristics may be captured to determine the skin
characteristics
and the skin state 158 of the users' skin. Broad genetic parameters, such as
ethnicity, skin
color, location factors, environmental factors (such as pollen count, weather,
etc.) , and
lifestyle factors may be collected in addition to image and skin health data
to determine
the users' skin state 158. This skin state 158 may be correlated with product
experience
ranking and ratings 138 to enable providing a recommendation for most
effective
products.
[00300] The user interface 102 may display a regimen 118. The regimen 118
may be a feature that enables users to learn what products and product usage
pattern
would work best for their skin based on a hardware- or community-led
personalized skin
care assessment 160 and / or type determination 130 and product experience
sharing via
Page 88 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
ranking and rating 138 and / or comments regarding product effectiveness and
experience
(such as smell, taste, feel, texture, color, and the like). The regimen 118
may be a
dynamic recommendation based on users' collective inputs as well as experts'
inputs on
products that would best suit the user's individual needs.
[00301] In an embodiment, the user interface 102 may enable simulation tools
132. Users may be able to upload an image and model various skin parameters
(such as
moisture level in skin, collagen level, age, and the like.) and observe
changes in the
image. Additionally, users may be able to model the impact of various products
and
regimens 118 (skin care, cosmetic, medical, nail care, hair care, and the
like) on the
image. Simulation tools 132 may enable users to view changes on the entire
image or
split half of the image to show a comparison of modeled change with current
image. The
user's images could also be automatically or manually optimized for the best
look and the
products or regimen 118 to obtain that look may be provided. Simulation tools
132 may
also enable consumers to model the skin characteristics or state 158 of other
selected
users or non-users, such as celebrities, luminaries, average users, and the
like.
[00302] In an embodiment, the user interface 102 may enable a daily report
134. The daily report 134 may be a report that provides the user information
largely
customized and most relevant to the user based on their skin state 158. The
daily report
134 may list skin care regimen 118 to be followed based on the environmental
and
lifestyle factors relevant to the user, may indicate new product information
190, show the
current skin care shelf 114 and rankings 138 or change in rankings 138,
feedback from
users or experts 105 on products most relevant to the user, and the like. The
daily report
134 may include information about clinical trials and upcoming results, new
product
releases and status, events, various factors affecting the skin such as the
day's weather
forecast, UV index, temperature, pollen count, and the like, and other data to
provide
value to the user. The daily report 134 may report on whether a product is
nearing its
shelf life or may require replenishment based on a recommended usage protocol.
The
daily report 134 may be provided to the user by the user interface 102, paper,
email,
SMS, RSS, video or any other communication media.
[00303] In an embodiment, the user interface 102 may enable a wishlist 134.
The wishlist 134 may be a function that a user could select and add products
to a part of
Page 89 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
the skin care shelf 114 using drag and drop functionality or other selection
mechanism as
they surf the web or otherwise access product information 190. They could
share this
function with other users, friends and/ or family so that other people could
see the wish
list 134. Other users could then select the products off the wish list 134 and
purchase and
send the product to the user.
[00304] In an embodiment, the user interface 102 may enable ranking and
rating 138. Ranking and rating 138 may be performed for various product
characteristics
as well as on the various raters and rankers. Product experience may be
collected from
users in simple ranking and rating 138 format as well as textual comment data
to be
stored in a database. This ranking and rating 138 may be real time, and may be
synthesized to show what is most relevant to the user based on like users or
peers, such as
users with any of the following characteristics: same age, same sex, same skin
type, same
ethnicity, geography, moisture levels, and the like. These ranking and ratings
138 may be
dynamic ranking and ratings 138. The users may be shown either the total
number of
rankers / raters and/or the weighted percent score ranking or rating 138. The
ranking and
rating 138 may comprise any of the following characteristics: perceived
effectiveness,
smell, touch, feel, texture, ability to absorb product, stains left by
product, ease of use,
and the like. Users may also be able to upload their images and obtain
effectiveness/look
ranking and rating 138 for different product recommendations from other users
or experts
105. For example and without limitation, a user may upload data and/or images
and
request rating and feedback on better products from an herbal expert in India,
aging
expert in Japan, and the like. Users providing ranking and rating 138 for
various
products may themselves be rated by other users. This may enable selection of
the most
effective and unbiased users and help identify potential experts 105. A small
select group
of highly ranked users may be offered exclusive writing / publishing and
ranking / rating
privileges.
[00305] In an embodiment, the user interface 102 may enable a skin cycle
monitor 140. The skin cycle monitor 140 may indicate when the last image was
collected
and countdown to the next scan based on a time interval, such as the time
required to
replenish the skin or any other interval. Currently, it is believed that the
skin replenishes
Page 90 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
itself every 28 days. The skin cycle monitor 140 may take into consideration
age,
environmental changes, and other factors to indicate the upcoming scan
schedule.
[00306] In an embodiment, the user interface 102 may enable wellness/ health
142. The user interface 102 may collect lifestyle data and also provide
lifestyle (such as
sleep, rest, exercise, and the like) and health (such as vitamins, food,
products usage, and
the like) recommendations based on the users particular skin state 158 and
characteristics.
The wellness and health module 142 may enable the user to obtain a
personalized best fit
health and wellness schedule and regimen 118.
[00307] In an embodiment, the user interface 102 may enable games 148.
Users may be able to play games 148 that may enable users to model various
products,
try different hairstyles, model different hairstyles and clothes, and the
like. Users may
interact with other users or the computer to make the product selection a fun
process.
This process could also be used to collect information on user preferences and
looks.
[00308] In an embodiment, the user interface 102 may enable a gift guide 144.
Based on the user's skin state 158, personalized gift advice may be provided
to others in
the user's network.
[00309] In an embodiment, the user interface 102 may be embodied in touch
screen user navigation. A touch screen system may be employed to enable the
user to
obtain a visual look and navigate to various parts of the user interface 102,
such as
navigate to the simulation tools 132, change picture orientation, drag and
drop, and the
like. Touch screen navigation may be particularly helpful as the hardware
device 108 is
connected to a computing platform. The user interface 102 may also enable
collecting
and coordinating information from other devices 109 and/or assessments, such
as a
dermascope, blood report, biopsy report, and the like to provide additional
information
for the skin record 121.
[00310] In an embodiment, the user interface 102 may enable a purchase/
sample portal. The user interface 102 may include a purchase/sample portal
that may
enable the user to select products and complete a purchase or request a sample
to be
delivered to a pre-entered address. The portal may be available in various
social
networking platforms 188 as well as over various computing platforms, such as
an online
Page 91 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
platform 120, mobile platform 124, computer, laptop, mobile phones, and other
mobile
devices, medical-use devices, and the like.
[00311] In an embodiment, the user interface 102 may enable scheduling and
data sharing functionality. A user may be able to schedule online a meeting
with a
particular expert or practitioner and, if willing, then share a skin state 158
or specific
parts of the skin record 121 and history in part or its entirety with the
expert or
practitioner. Ranked experts and practitioners, availability, and other
criteria to aid the
selection and scheduling process may be indicated to the user. Experts may
also be able
to share particular sets of data amongst themselves, such as among
practitioners,
physician to another physician, physician to spa, spa to spa, and the like.
[00312] Other inputs 112, such as devices, features and data, may be used to
augment the data submitted by the user or as the primary data to obtain a
personalized
assessment regarding the users' beauty, cosmetic, or medical concerns related
to skin,
hair, nails, and the like. For example, certain devices may be available
commercially off
the shelf, purchased, proprietary, and the like.
[00313] In an embodiment, a wearable monitor 182 may be an input 112 to the
system 104 and user interface 102. Wearable skin health monitors 182 may
enable real
time tracking of changes in the environment and the skins health. These
devices could be
worn directly on the body, or integrated into clothing, apparel and / or
accessories carried
by the user. An example would be a user having a device that monitors the UV
level, and
provides a warning if the sun protection level accorded by a product used by
the user falls
below a set target level. These wearable monitors 182 may have independent
user
interfaces 102 or can be programmed for personalized parameters using other
input
devices. Wearable monitors 182 may also capture various physical parameters
like heart
rate, blood pressure, exercise rate, water consumption, fat counter, calorie
meter, and the
like. The monitors 182 may be able to assess hydration levels.
[00314] In an embodiment, a social network 188 may be an input 112 to the
system 104 and user interface 102. The beauty social network 188 may be a
collection of
users interested in knowing and sharing information on beauty or medical
concerns in a
personal, private, and social interactive setting. The intent may be to create
a beauty
social network 188 where users invite and link to other users to discuss such
concerns;
Page 92 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
obtain information 190, 192; perform ranking, rating, and review of products,
regimens,
experts, practitioners, other rankers/raters, and the like; complete
purchases; access a
wishlist 119; access a gift guide 144; play a game 148; review their daily
report 134; and
the like, all the while sharing experiences with other users in their network.
[00315] In an embodiment, product information 190 may be an input 112 to the
system 104 and user interface 102. A database of product information 190 may
comprise
product, name, claims, manufacturer information, ranking and ratings 138,
packaging
information, images, usage parameters, product development history or
forecast, special
handling, upcoming changes, safety information, effectiveness information,
smell, taste,
color, texture, price, geography of manufacturing, brand information, consumer
feedback
and experiences, and other such parameters that may be obtained and/ or
maintained to
assist in the selection of the best product suited to the users' individual
preferences or
conditions to obtain the best beauty or medical outcome for their skin, hair,
nails, and the
like. Additionally, similar information on service oriented products such as
massages,
facials, hair toning, and the like may also be captured as well as information
on
procedures such as liposuction, Botox treatments, laser hair removal and other
beauty,
cosmetic and/ or medical procedures related to helping the user look good,
improve or
maintain a skin state 158, and the like. Manufacturers may register product
information
190, contribute information on procedures, products in the pipeline, products
in clinical
trials, and the like. Users may rank and rate 138 products. A database update
utility may
update the database with new product information 190, store inventory, and the
like.
[00316] In an embodiment, wellness information 192 may be an input 112 to
the system 104 and user interface 102. Health and wellness information 192 may
be
captured, such as the impact of various products, primarily but not limited to
non-
prescription medications, supplements and other consumables that assist and
maintain
health and wellness (such as vitamins, protein shakes, supplements, and the
like).
Additionally, information on lifestyle recommendations (such as sleep, rest,
diet and
exercise recommendations for particular age groups/ ethnicities, etc.) may be
collected
and correlated with user preferences and characteristics to enable and provide
a holistic
health, wellness, and beauty/ cosmetic optimal personalized solution and
service.
Page 93 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00317] In an embodiment, a plug-in web capture 194 may be an input 112 to
the system 104 and user interface 102. A software component-plug in for
internet web
browsers and basket or repository may recognize graphic objects on any browsed
web
page and allow the user to select, and drag-and-drop the graphic object onto a
basket or
repository onto a page of the web browser, such as a page comprising the skin
care shelf
114. The graphic objects would be recognized through a standard reference
table that
would be accessed remotely or reside on the user's PC as part of the plug-in
module 194,
or as part of a resident software program on the computing platform. Graphic
objects may
include images for commercial products, such as skin care products or creams,
or other
objects that are part of any web e-commerce site. Once recognized, the plug-in
194 may
highlight the picture, notifying the user that is it recognized, or provide
additional
information or reference. The plug-in 194 may also recognize brand names,
trade names,
generic pharmaceutical names, trademarks, and the like.
[00318] In an embodiment, barcode scan 198 may be an input 112 to the
system 104 and user interface 102. Bar code information on various products
may be
captured to assist tracking, identification, price determination and
correlation with other
product information 190 for identifying similar substitute products, or other
allied
product information, usage recommendation, other user experience, pricing and
delivery
information, amongst other relevant sets of data. The bar code scanner 198
could be part
of the hand held user device 108, a standalone system, a manual entry
mechanism, and
the like.
[00319] In an embodiment, conventional information/ questionnaires 101 may
be an input 112 to the system 104 and user interface 102. Information 101 on
the users
and products may be captured via dynamic and static questions. Information
such as age,
sex, location, personal lifestyle traits, smoking habits, sleep patterns, skin
dryness /
oiliness and moisture levels, product likes and dislikes, experiences with
other products
along parameters such as smell, taste, absorption, staining propensity, and
the like may be
captured in a fun manner using questions and answers, games and other
interactive tools
interspersed at various points of the users' interaction with the service
product, system
104, or user interface 102. Information 101 may be captured directly form the
user or via
an intermediary, and augmented automatically via computer data population, as
an output
Page 94 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
of an algorithm 150 or by experts based on their assessment. Information 101
may be
obtained by quizzes, badge- and widget-based forms, on-the-fly, through
adaptive,
investigative questioning, and the like. Information 101 may be obtained
through
questionnaires, such as How often do you go shopping?, When do you shop for
cosmetics?, Where do you typically go? Why that spot?, Who do you shop with?
Why?,
What do you ask your friends when asking for advice?, Where do you go for new
products/ information about cosmetics?, When do you have to go to a dept
store, vs.
buying online?, When would you want to know something immediately from your
friends?, What do you ask from your friends?, How do you choose a mobile
phone?,
What do you care about menus on a cell phone?, When do you get a new cell
phone?, and
the like.
[00320] In an embodiment, third party experts 105 may be an input 112 to the
system 104 and user interface 102. The system 104 may connect various experts
such as
practitioners, physicians, medical experts, aestheticians, schedulers, product
ingredient
experts, cosmetologists, herbal, ayurvedic and homeopathic experts, health and
wellness
experts, media experts, photograph enhancement experts, and the like with
users and one
another. Users may be able to direct questions to such experts 105 who may be
located at
different places geographically over the system to obtain personalized advice.
The
experts 105 may be provided with users' data and characteristics collected and
a record of
the experts assessment may be retained in the record 121. The recommendation
provided
by the expert may be offered to the user for purchase / sample request, and
the like.
Experts may also be able to flag certain cases or sets of data for discussion
or referrals
within the expert community or with users.
[00321] In an embodiment, third party hardware 109 may be an input 112 to
the system 104 and user interface 102. The system may connect with various
third party
hardware 109, such as existing imaging solutions, camera devices, computers,
lighting
systems, sports devices such as pedometers, and the like.
[00322] In an embodiment, third party service providers 111 may be an input
112 to the system 104 and user interface 102. Third party service providers
111 may be
integrated into the system 104 to enable users to make the best personalized
product or
service selection for their hair, skin, nails, and the like for medical or
cosmetic / beauty
Page 95 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
needs, and the like. Third party service providers 111 may include hospitals,
physicians,
spas, salons, aestheticians, beauticians, cosmetic counters, drug stores,
cosmetics sales
representatives and websites, ranking and rating services, product information
databases,
testing laboratories, magazines and information providers, insurance
companies, social
networking sites, health and wellness services, photograph enhancement
services, and the
like. For example, based on a skin concern, the scheduling system for a
physician may
be integrated and scheduling options offered online to users, while also
connecting with
insurance providers to confirm coverage with the user. In addition, pre-
assessments on
the condition, availability of historical medical and/or cosmetic products
prescribed either
over the counter or by medical prescription, and / or recommended services may
be
captured to make the selection process for the user convenient and easy.
[00323] Referring to Fig. 7, a system for providing recommendations for skin
care based on a skin state 158, a skin care goal, and environmental factors
affecting the
skin may comprise obtaining a skin state 158 of an individual, categorizing
the individual
by skin state 158, and recommending products and regimens that may be
effective in
achieving a skin care goal. The system may be computer-based, Internet based,
network
based, and the like. The system may be a community-led provision of skin
services. In
an embodiment, the recommendation may be made on the basis of identifying
other users
with similar skin states and identifying a product or regimen that is
effective for them. In
an embodiment, the recommendation may be made on the basis of product
information
190, wellness information 192, a third party database 115, an expert 105, a
service
provider 111, and the like. As seen in Fig.7, a user may acquire an initial
image and
perform an analysis for a specific endpoint, such as moisture in this case.
The system
may automatically recommend certain products based on the moisture level that
may be
effective given the moisture level, a skin state 158, and the like.
Additionally, the system
may perform a projection of skin state 158 based on various skin care regimens
118, such
as maximum care, normal care, or poor care. In an embodiment, the images may
be
captured using the device 108 or third party hardware 109. Images may be
captured
using any image capture device or technique, employing any kind of incident
light, such
as unpolarized light, polarized light, monochromatic light, diffuse light,
white light,
Page 96 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
multiple single wavelength light, and the like. Any captured image may be used
to obtain
a skin state 15 8.
[00324] An embodiment of a skin care recommendation page of a skin care
system may include a report of products the user is currently using, user
input to obtain a
skin state 158, a recommendation request, and the like. The report on the
products the
user is currently using may include ranking or ratings 138. For example, when
a user
accesses the user interface 102, they may access an adaptive questionnaire to
determine
their experience with their current regimen 118, current products or therapies
used, or any
products or regimens 118 used in the past. For example, the user may be asked
to
respond to questions such as How effective is it?, How is its fragrance?, How
does it
absorb?, Does it cause breakouts?, How does it feel?, Do you think this
product is of
good value?, and the like. Of course, rankings and ratings need not be
prompted by
questions but may simply be anecdotal, deployed in a non-question format,
deployed in a
drop down menu, and the like. To obtain a skin state 158, the user may enter
data
relating to aspects such as gender, age, ethnicity, location, skin color,
environmental
factors, and the like. In embodiments, analysis 154 of images obtained from
the device
108 or third party hardware 109 may also be used to determine a skin state
158. Based
on the skin state 158, either derived from user input, analysis of images, or
a combination
thereof, users may be able to determine products and regimens 118 that may
work best
for their skin state 158 by connecting to a database containing wellness 192,
regimen 118,
expert 105, service provider 111, and product information 190, wherein the
information
may comprise product ingredients, product claims, product indications, product
pairing,
product usage protocol, product ratings and rankings 138, and the like. By
including
rankings and ratings 138, community-led recommendations may be made for skin
related
products adjusted for age, skin color, location, ethnicity, environmental
factors, and the
like. In an embodiment, the user may perform a recommendation request which
may
involve selecting a skin goal, such as moisturize, protect, cleanse, tone,
beautify, anti-
aging, wrinkle protection, skin tightening, deep cleanse, pore diminishing,
treat rosacea,
exfoliate, lighten skin, tan, sun protect, self-tan, treat acne, avoid
pimples, improve
luminosity, skin rejuvenation, treat spots, treat Crow's feet, hair removal,
scar treatment,
and the like. In embodiments, a skin goal may be automatically selected by the
system
Page 97 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
104. Automatic selection may be based on an aspect of the skin state 158. For
example,
if analysis 154 reveals that the skin is severely dry, the system may
recommend
moisturizing products for severely dry skin, or the system may recommend
ingredients to
look for in a product. The user may be able to purchase products directly from
the
recommendations page, such as by placing the product in an electronic shopping
cart 113,
or may be directed to another site for purchase. In an embodiment, the user
may add the
product to a wishlist 119 for future purchasing. In an embodiment, the user
may add the
product to a skin care shelf 114, which may be an interface to or depiction of
a regimen
118 that enables users to organize their products and regimen 118 in a logical
fashion
based on the user's specific skin characteristics 130, by usage scenario (e.g.
Morning,
afternoon, night, etc.), intent (e.g. work, fun, etc.). The beauty shelf 114
may have
multiple screens for recommendations by various bodies (e.g. Physicians,
dermatologists,
aestheticians, spa specialists, overall users, experts, people most like you,
etc.). The
beauty shelf 114 may be a personalized arrangement of products. Users may drag
and
drop products (or select to add) as they are surfing the web and discover new
products as
well as having auto-populated recommendations. The functionality may include a
program that will highlight products of interest while surfing the web. The
beauty shelf
114 may be an application that can also sit independently on social networking
sites and
other personal pages and or toolbars. The beauty shelf 114 may also indicate
purchase
date and purchase history, product expiration alerts and other usage updates.
A purchase
made off the website will automatically add to the user's beauty shelf 114,
while manual
entries for offline purchases may also be possible.
[00325] . In an embodiment, the user may be able to obtain samples of
recommended or non-recommended products directly from the recommendations
page.
The shopping cart 113 may be a functionality that integrates with the skin
care shelf 114.
Users may be able to use the personalized recommendations and select products
either for
purchase, or for sample delivery. The user may be prompted for personal
information
such as address, shipping method, credit card number and the like, and that
information
may be retained by the shopping cart 113. The shopping cart 113 may be an
independent
program, in similar fashion to the skin care shelf 114, that may reside in a
toolbar, as part
of a user interface 102 or as a program on a webpage, so that products could
be
Page 98 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
highlighted and dragged into the shopping cart 113 for later purchase.
Dragging the
product into the cart 113 may also initiate queries across the database and
across various
websites for best price, location and availability of product, consumer
experience,
rankings and ratings and the like.
[00326] Referring to Fig. 9, a product rating page of a skin care system is
depicted. To obtain recommendations, users may be asked to respond to their
medical,
non-medical, cosmetic and skin care product experiences, thereby scaling data
collection
inexpensively. For example, a user may identify a product and provide an
effectiveness
assessment, rankings and ratings 138 for the product, anecdotal information,
usage
information, and the like. This information may be stored in a wellness 192,
regimen
118, and product information 190 database in order to refine future
recommendations. In
an embodiment, user responses to product experiences may be shared with
friends and/or
other users automatically or upon request.
[00327] Referring to Fig. 10, a user interface 102 home page 1000 of a skin
care system 104 is depicted. The user may be prompted to input demographic
information such as name, gender, age, occupation, ID, address, telephone
number, email
address, payment information, new related users, and the like, which may be
stored in a
user profile or as part of a skin record 121. The home page may show a skin
record 121,
or a listing of areas imaged, date imaged, and status of analysis. Once a task
is complete
in the skin history/record 121, an icon may be displayed near the Status. The
user may be
able to launch a new Skin Health Test from the home page 1000 or submit a new
skin
concern. The user may be able to forward the analysis 154 to an interested
party; Ask an
Expert a question regarding an aspect of the skin, skin history/record 121,
image analysis,
and the like; view payment information and history; and the like.
[00328] Referring to Fig. 11, a welcome page 1100 of a skin health test is
depicted. The welcome page may provide information on the skin health test,
what
endpoints will be tested for, such as elasticity, wrinkles/ fine lines, sun
damage, glow /
luminosity, and the like. Using the analysis of the skin health test, the
system may
provide a personalized assessment of the user's skin regimen 118. The user may
initiate
the skin health test from the welcome page 1100.
Page 99 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00329] Referring to Fig. 12, a questionnaire page 1200 of a skin care system
is
depicted. The questionnaire may capture relevant skin history that may be
useful for
subsequent image analysis. The questions may be asked in multiple choice
fashion or as
open-ended questions. For example, a question may be `Where do you use your
product?' with responses including face, hands, neck, legs, torso, and the
like. Another
question may be `Why are you using your product?' with responses including to
protect,
repair, moisturize, and any other skin care goal. Another question may be,
`Why are/will
you be using your product?' with responses including reduce wrinkles / fine
lines,
increase shine / luminosity, increase softness / elasticity, and any other
skin care goal.
Other questions may include, `How long have you been using your product?',
`How often
do you apply your product?', `When do you apply your product?', and the like,
with
responses including stated intervals of time. Other information gathered may
be how the
user prefers notification, where products were purchased, if the user employs
a seasonal
usage of products, and the like. From the questionnaire page 1200, the user
may launch
the skin health test.
[00330] Referring to Fig. 13, a skin image capture page 1300 of a skin care
system is depicted. In the example, the user interface 102 may access a device
108 in
order to capture images, however, it should be understood that other devices
109 may be
conveniently used in the system. The page 1300 may show a real time view of
the area
being imaged. The user may be able to employ positioning tools to be able to
take an
exact image of an area previously imaged. Once an image has been captured and
submitted, an algorithm 150 may verify the integrity of the image. Once an
image
suitable for analysis has been captured, the user may proceed to an analysis
page 1400.
[00331] Referring to Fig. 14, a results page of a skin care system with bar
graphs is depicted. Algorithms 150 may be used to analyze the image and
provide
measurements of wrinkles, elasticity, luminosity, firmness, tightness, and the
like, as
described previously herein. In an embodiment, the measurements may be
quantitative
measurements. The first analysis may be considered a baseline for purposes of
tracking.
For each measure, the user may be compared against the baseline for their age,
skin state,
gender, ethnicity, or any other category. For example, the graph depicts the
reading for
the user in the first bar on each graph and the average baseline for people of
the same age
Page 100 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
in the second bar. It is apparent from visual inspection that the user is
better than
average, in this case. These results may be color-coded for ease of
interpretation. The
results page 1400 may include a description of each measure. The user may be
able to
request More Information for each of the measures, such as why a certain
condition is
caused and hints and tips on how to improve a skin condition. The user may be
given
instructions on when to re-scan the area, which products to use, which regimen
118 to
employ, and the like. Desired improvements may be correlated to ingredients
and most
effective products for the user's skin may be recommended. The user may access
and/or
edit a skin record 121, which may contain information about the user, images,
a
chronology of images, information derived from the images, recommendations,
products,
regimen 118, and the like. The user may access a report facility to obtain a
report.
[00332] Referring to Fig. 15, a results page of a skin care system with trend
analysis is depicted. A method for tracking the effectiveness of a skin care
product or
regimen may comprise obtaining a baseline skin health assessment; recommending
a
monitoring interval based on at least one of the skin care goal, product, and
regimen;
obtaining a second skin health assessment; comparing the second assessment to
the
baseline assessment to determine progress towards a skin care goal; and,
optionally,
optimizing the regimen 118 or product in order to improve a skin health
assessment.
When a subsequent image is acquired and submitted to the system 104, a trend
analysis
may be performed. Subsequent images may be used to track effectiveness of
products
and/or regimens 118 and, ultimately, advise the user on and optimize their
skin regimen
118, product and/or condition. The trend analysis 1502 may be useful for
determining an
intermediate skin state 158 during a regimen 118. The trend analysis 1502 may
show a
baseline reading, an average reading for healthy skin for someone of the
user's age, and
individual measurements for each type of skin condition. Progress may be shown
over
time. A time series of images, such as over a twenty-eight day skin cycle,
over a
treatment timeframe, seasonally, periodically over a year and the like may be
captured in
order to track progress of a skin state 158. The data may be presented in a
pictorial view
with data on the picture, graphical view, trend view, numerical view, text
view, and the
like. Progress may be sorted by the concerns / skin care goals that the user
may have
indicated at the beginning of the test. The user may be told when to take the
next image,
Page 101 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
how much longer to continue with a regimen 118, how to modify the regimen 118,
be
reassured about the effectiveness of a product or regimen 118, receive useful
tips, and the
like. The user may view and/or edit a skin record 121. The user may be able to
view past
images and perform a simulation 132 of future progress. The user may access a
report
facility to obtain a report.
[00333] Referring to Fig. 16, a summary screen of a skin care system is
depicted. An overall analysis for a time interval may be shown, current
measurements,
progress towards reaching a skin care goal, a product assessment, a regimen
118
assessment, advice on continuing, modifying, or terminating a regimen 118 or
product
usage, and the like. The user may view a step-by-step analysis or obtain a
full report. At
an interval, such as at the end of a suggested regimen 118, a report may
include
information on how the user's skin state 158 changed over time, if the user's
skin is
healthier than when they started the regimen 118, if the product or regimen
118 met their
initial goals, feedback on regimen 118/ product effectiveness, and the like.
Given the
current skin state 158, a new product or regimen 118 may be recommended. For
example, the system may recommend specific ingredients to look for in order to
increase
a user's luminosity given a current skin state 158. Reports may be on-screen,
printed,
custom, and the like. Reports may be shared with a practitioner for ongoing
treatment
and consultation.
[00334] Referring to Fig. 17, an elasticity summary page 1700 of a skin care
system is depicted. A step-by-step analysis of each indicator may be
performed. For
example, a step-by-step analysis of the elasticity measurement is shown in
Fig. 17. The
summary page 1700 may depict all of the data captured over an interval, such
as in a bar
graph, for each indicator on separate summary pages 1700. It should be
understood that
while Fig. 17 depicts an elasticity summary page, the summary page may
summarize data
related to any and all concerns. Progress towards meeting a skin care goal may
be
indicated by the data and its analysis or from user input. An assessment of a
user's
product or regimen 118 in meeting the skin care goal may be made. Products or
regimens 118 that may enable meeting future needs may be indicated. The system
may
also indicate products used or regimens 118 employed by other users in meeting
the
stated skin care goal.
Page 102 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00335] In an embodiment, the data acquired at a single time point or over a
time interval may be shared with other users of the skin care system,
practitioners, and
the like. In an embodiment, the data may be shared as a data object with users
of an
online platform 120 or mobile platform 124 of the skin care system, posted to
blogs, e-
mailed to third parties, and the like. In some embodiments, the data may be a
drag-and-
droppable data object. For example, the wrinkle trend analysis 1502 shown in
Fig. 15
may be shared with friends as in Fig. 68, posted on a blog or forum where
users may
discuss the data as in Fig. 69, become part of the content that a user may
wish to discuss
as in Fig. 70, and the like.
[00336] In embodiments, a system for providing recommendations for skin
care based on a skin state 158, a skin care goal, and environmental factors
affecting the
skin may comprise interaction with tools and algorithms 150 on an online
platform 120, a
mobile platform 124, a social networking interface, and the like to receive
product and
regimen recommendations and track product and regimen 118 effectiveness. The
system
may be a communication platform, online 120 or mobile 124, that connects
geographically separate consumers, manufacturers, product information,
experts, service
providers and others related to or allied to the beauty and medical field to
provide
personalized assessment regarding the consumers skin, hair, or nails queries
and
concerns. The user interface 102 may reside on an online platform 120, mobile
platform
124, or social networking interface. In some embodiments, a skin care
assessment may
be provided by algorithms 150 operating on an online platform 120 without the
use of
images or data from a device 108, that is, a user need not have data from a
device 108 to
participate in the online platform 120. The online platform 120 may be a
standalone skin
health assessment and skin care recommendation tool. However, in embodiments,
image
data may also be used by the online platform 120 to provide skin health
assessments and
skin care recommendations. A user interface 102 may interface with the online
platform
120. For example, a user may access an online platform 120 of the system for
skin health
analysis, monitoring, and recommendation to: monitor skin health, download,
process,
analyze, track, and store data from an imaging device 108 or other device 109
or monitor
182, receive product and /or regimen recommendations from an analysis/ API 154
or
from peers, compare skin state 158 and regimen 118 with peers, receive product
Page 103 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
information 190, purchase products; add recommendations to a skin care shelf
114;
organize a skin care shelf 114 by regimen 118, rankings, expiration date,
cost, skin care
goal, time of day, frequency, friends, and the like; view community ratings,
rankings and
comments on products/ regimen in a skin care shelf 114; rank/rate products;
leave
comments on products, regimens, peers products and/or regimens; and the like,
receive
new product alerts or product recalls, receive a daily report 134, interact
with a social
network 188, and the like. The user interface 102 may enable users to
conveniently take
and submit images, enter data, track history, obtain recommendations and
analysis and
perform a purchase regarding their skin, hair, and/or nail's beauty/cosmetic
or medical
concern. The user interface 102 may reside on an online platform 120 and guide
the user
while also serving as a data repository to maintain a skin record 121 and
history tracking
tool, and may help the user organize information relevant to their condition
in a logical
fashion.
[00337] In an embodiment, the user interface may comprise a skin care shelf
114. The skin care shelf 114 may be a structure that enables users to organize
their
products and regimen 118 in a logical fashion based on users' specific skin
characteristics
130 / skin state 158 by usage scenario (such as morning, afternoon, night, and
the like),
intent (such as work, fun, etc.), skin care goal (such as moisture, glow,
protect, and the
like), and the like. The skin care shelf 114 may have multiple "pages" for
recommendations by various entities (such as practitioner, physicians,
dermatologists,
aestheticians, spa specialists, overall users, experts, people most like you,
and the like).
The skin care shelf 114 may be a personalized arrangement of products, regimen
118,
and/or information 190, 192. Users may drag and drop products (or select to
add) as they
are surfing the web and discover new products as well as having auto populated
recommendations. The functionality may include a facility that may highlight
products of
interest while surfing the web. For example, a plug-in 194 may be used to
allow a user to
capture information from any location on the Internet. For example, a user may
access a
web page for a makeover article in a beauty magazine and wish to include the
products
from the makeover in their skin care shelf 114 and/or shopping cart. 113. The
user may
click on the product name and drag it over to at least one of the skin care
shelf 114 and
shopping cart 113 to obtain additional product information 190, include in
their regimen
Page 104 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
118, purchase, request samples, and the like. The skin care shelf 114 may an
application
that may also sit independently on social networking sites 188 and other
personal pages
and or toolbars. The skin care shelf 114 may also indicate purchase date and
purchase
history, product expiration alerts and other usage updates. In an embodiment,
a purchase
made off a website may automatically add to the users' shelf 114, while manual
entries
for offline purchases may also be possible.
[00338] In an embodiment, the user interface 102 may interface with a mobile
platform 124. The user interface 102 may support plug and play with various
mobile
devices 184 such as mobile phones, laptops, digital cameras, medical-use
devices, and the
like. For example, the mobile phone may have an attachment or an integrated
feature that
may enable a user to take an image of the skin and input/ capture data and
have it connect
via the web, wirelessly or via cable, to the user interface 102 and enable
seamless
connectivity and data transfer. The mobile device could be used to take images
and data
at various locations for obtaining various information from the community
(such as at the
beach to measure effectiveness of sun screen, an image of a specific location,
a product
image or a bar code image to get product feedback, best price, nearest
physical selling
location, coupons, and the like). Users may also be able to share data / ask
questions
regarding products instantaneously to other users. The mobile device could
have an
internal lens system that may be internally charges or an independently
attached lens
system that would enable using the battery power and light source of the
device to take an
image and use the in-built communication method for submitting the image.
[00339] Referring to Fig. 18, the user interface for the online platform 120
may
be depicted as a map. The home page may have a different theme or feel
depending on
the user profile, the user preference, or any other criteria. For example, it
may be fun,
serious, clinical, and the like. From the user interface, a user may review
products,
contribute anecdotes, report, review reports, review blogs by product, skin
type, and the
like, visit their beauty shelf 114, and the like. Information may be accessed
freely, with
registration, or only partially freely and partly with registration. All
products and pages
may link through the beauty shelf 114.
[00340] For example, Fig. 19 depicts a review page of the user interface of a
skin care system. The menu across the top of the user interface may enable a
user to
Page 105 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
access Reviews, Experience, Recommendation, Info For Me, Checkout, and the
like. The
user interface may depict a portion of the user profile, such as the age,
gender, location,
skin type, skin color, skin goal, picture, and the like for the user. The user
interface may
also depict what products or regimen 118 the user may be using and any
associated
review, rating, or comments of the product. Other users accessing a user
profile may
make comments on the regimen 118 or products in use, give the products or
regimen 118
a rating, recommend a different product or regimen 118, and the like. The user
interface
may present tools to aid a user in selecting a product or regimen 118. For
example, the
tools may be in the form a questionnaire or wizard guising the user to
describe their skin.
The user may provide age, gender, skin type (oiliness, sensitivity), skin
color, goal,
current brand or product, current regimen 118 and the like. In some
embodiments, the
skin type and/ or color may be detected automatically if the user interface is
interfaced
with an imaging device 108. The user may also access their beauty shelf 114
from the
user interface.
[00341] Referring to Fig. 20, a review page of a user interface of a skin care
system is depicted. The review page is shown in a different layout than the
compact view
depicted in Fig. 19.
[00342] Referring to Fig. 21, an experience page of a user interface of a skin
care system is depicted. The experience page allows users to provide a
detailed report of
experience with a product or regimen 118. For example, the user may note the
effectiveness of a product or regimen 118, such as by answering questions. For
example,
the questions may be "How effective is it?", "How does it feel?", "How is its
fragrance?",
"How does it absorb?", "Does it cause breakouts?", and the like. The
experience page
may also allow a user to update a user profile with age, gender, nickname,
location, a
photo, skin type, skin color, goal, and the like. The user may be able to
query other users
for their experience or make a general inquiry by submitting a request to an
email, MMS,
SMS, phone number, mobile device, social network, and the like.
[00343] Referring to Fig. 22, a recommendation page of a user interface of a
skin care system is depicted. Given the goal, various products or regimens 118
that may
be effective in meeting the goal may be shown on the recommendation page. The
brand
and product or regimen 118 may be shown along with a rating from the community
of
Page 106 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
users, comments from users, the ability to indicate of the user believes the
product may
better than the current product or regimen 118 in use, and the like. If the
user believes
the product or regimen 118 may be better than what they are currently using,
the product
or regimen 118 may be stored for future consideration on the beauty shelf 114.
[00344] Referring to Fig. 23, an Info For Me page of a user interface of a
skin
care system is shown. A People Like Me algorithm 150 may be used to sort the
community of users of the skin care system. Given the aspects of the user
profile, the
algorithm 150 may determine which other users are most similar along all
criteria, along
custom-selected criteria, along a combination of skin color and skin type, and
the like.
Once the algorithm 150 has determined a subset of the community of users who
are most
like the user, the user can view data for the community. For example, the user
can find
out which products work best for the subset generally, for a specific issue,
for a specific
time of day, for a specific season, and the like. The Info for Me page may
also depict the
weather for the location given in the user profile and a UV rating and any
specific tips
given the location / weather/ environment. The Info for Me page may also alert
the users
of new products being launched. The user may sort the products according to
effectiveness.
[00345] Referring to Fig. 24, an example of a beauty shelf 114 portion of a
user
interface of a skin care system is shown. Products or regimens 118 used by the
user may
be categorized by time of day use, specific effectiveness, cost, expiration,
and the like.
Each item may be clicked on to pop-up additional details about the product or
regimen
118, such as effectiveness, ingredients, suggested use, expiration date, a
link to purchase
more, a link to blog about the product or regimen 118, a link to write a
review or read
reviews, a link to the manufacturer's site, a link to an in-store coupon, and
the like. Fig.
25 depicts another example of a beauty shelf 114 portion of a user interface
of a skin care
system. Fig. 26 depicts an alternate view of the beauty shelf 114 of the user
interface of a
skin care system. In this example, friends have the ability to comment on the
products or
regimen 118 and suggest an alternative product or regimen 118. The user also
has the
option to receive price alerts, new product launch alerts, new user comment
alerts, and
the like.
Page 107 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00346] Referring to Fig. 27, a registration page of a user interface of a
skin
care system is depicted. Information may be entered by the user, goals may be
indicated,
a security code may be entered, skin concerns, color, and/or type may be
entered, samples
may be registered for, and the like. Additionally, the user may indicate that
the want to
add a feed from the skin care system to their RSS feed, and application from
the skin care
system to a social networking site, and the like. The user may have the option
to opt-in to
alerts, to be notified of samples and products, and the like.
[00347] Referring to Fig. 28, another embodiment of a recommendation page
of a user interface of a skin care system is shown. This page may show people
in the
user's category, such as number of people of the same gender, same age group,
with
similar skin type, with similar concerns, and the like. For each stated goal,
a product may
be recommended that is most popular, has the most buzz, has been reviewed, has
been
rated, has been blogged about, and the like.
[00348] Referring to Fig. 64, the user interface may include a friend toolbar.
The friend toolbar may float over a current website, or any website, such as
by using a
plug-in. Friends may upload images and the images 6408 may be displayed on the
friend
toolbar 6402. A home key 6404 may be part of the toolbar 6402, where the whole
toolbar can be reduced to just the home key 6404. When an alert is associated
with a
friend, such as a new product being added to their beauty shelf 114 or a new
review being
written, a flag alert 6410 may pop-up next to their image on the toolbar 6402.
A bottom
bar 6412 may be used for shuffling friends or accessing other options related
to the
toolbar 6402. Referring to Fig. 65, the toolbar 6402 may auto-scroll 6502 as
the user
scrolls the webpage they are viewing. Referring to Fig. 66, objects may be
shared with
friends in the friends' toolbar 6402 using a drag-and-drop functionality 6602.
For
example, a blog posting may be shared as in Fig. 66 by dragging and dropping
the blog
title onto a friend's image. Similarly, products may be recommended to a
friend by
dragging and dropping 6702 the product into the friends' image, as in Fig, 67.
Rolling
over a friends' image may result in a pop-up, dialog box or other
manifestation of
additional information about the friend, such as a view of their user profile,
beauty shelf
114, reviews, blogs, and the like.
Page 108 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
[00349] Referring to Fig. 29, a mobile content map for a mobile user interface
of a skin care system on a mobile platform 124 is depicted. The content map
depicted
shows an example of content that can be accessed from a mobile platform 124
home
page. For example, starting from the home page, a product may be scanned or
identified
from a list and searched for using the internet on the mobile device. For
example, a bar
code may be scanned for a product and prices, reviews, ratings and the like
for the
product may be returned. The user may be helped to find something, such as an
item for
themselves, a gift for a friend, and the like. The product may be searched for
based on a
goal, an issue, a skin type, a skin color, and the like. The mobile skin care
system may
return a list of products, such as the top 10 products, and information about
the products
such as rating, impact on goals, safety, reviews, and the like. The user may
access a
Suncheck application to be given UV information by location and advice, as
well as
based on an image captured by an imaging device 108 embodied in a mobile
device, as
described previously herein.
[00350] Referring to Fig. 30, a How Good Is This Product message flow is
depicted. In the example, a bar code may be scanned to obtain product info,
the bar code
numbers may be manually entered, or the product may be chosen from a list. The
system
may return product information such as the product name, rating, ingredients,
a general
rating, a rating for a specific concern, a friend's rating, a price, where the
product can be
found, and the like. If the mobile device is enabled, a purchase may be
initiated on the
mobile platform 124.
[00351] Referring to Fig. 31, a What Should I Look For? message flow is
depicted. The message flow may begin by giving the user the option to indicate
if the
item searched for is a gift, for the user, to update a pick list, and the
like. For gifts, a
recipient may be selected from a pre-populated list or a new recipient may be
indicated.
An occasion may be indicated. Based on the recipient and occasion and any
other criteria
entered, products may be recommended along with any information associated
with the
product, a price, a location, and an option to purchase on the mobile platform
124. In
looking for something for the user, the user may indicate a goal, such as from
a drop
down menu, and receive a list of recommended products. Once a product is
selected, the
Page 109 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
user may request to locate the product at a store or initiate a purchase on
the mobile
platform 124, or the like.
[00352] Referring to Fig. 32, a Suncheck message flow is depicted. The initial
message may contain information about the user's location, the weather, a UV
index, a
sun impact rating, an indication of the maximum exposure time, and a timer for
measuring the current time in the sun. Advice may be generated based on the
information, such as what level of sun protection factor to apply, a maximum
recommended time of exposure, and the like.
[00353] Referring to Fig. 33, an Alert message flow is depicted. The user may
be linked to other users on the mobile platform 124 so that when another user
requests a
review or rating of a product, an alert may be sent to the user. The user may
respond
with a review, a rating, a chat message, an SMS, an MMS, a phone call, a
voicemail, and
the like.
[00354] Referring to Fig. 34, an Options message flow is depicted. From the
mobile platform 124 home page 3402, Options may be selected. Options 3404 may
be a
friend list, a pick list, alerts, address/location, and the like. For example,
a friend list
3408 may be accessed to pick and choose friends to follow, receive alerts from
and the
like. The friends list may indicate if the friend is online. Alerts 3410 may
also be set on
the mobile platform 124, for example to notify the user when their friends buy
something
new, notify the user when a new product that is good for them is available,
and the like.
Address / location / payment setup may allow the user to initiate purchases
from the
mobile platform 124.
[00355] The methods and systems described herein may be deployed in part or
in whole through a machine that executes computer software, program codes,
and/or
instructions on a processor. The processor may be part of a server, client,
network
infrastructure, mobile computing platform, stationary computing platform, or
other
computing platform. A processor may be any kind of computational or processing
device
capable of executing program instructions, codes, binary instructions and the
like. The
processor may be or include a signal processor, digital processor, embedded
processor,
microprocessor or any variant such as a co-processor (math co-processor,
graphic co-
processor, communication co-processor and the like) and the like that may
directly or
Page 110 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
indirectly facilitate execution of program code or program instructions stored
thereon. In
addition, the processor may enable execution of multiple programs, threads,
and codes.
The threads may be executed simultaneously to enhance the performance of the
processor
and to facilitate simultaneous operations of the application. By way of
implementation,
methods, program codes, program instructions and the like described herein may
be
implemented in one or more thread. The thread may spawn other threads that may
have
assigned priorities associated with them; the processor may execute these
threads based
on priority or any other order based on instructions provided in the program
code. The
processor may include memory that stores methods, codes, instructions and
programs as
described herein and elsewhere. The processor may access a storage medium
through an
interface that may store methods, codes, and instructions as described herein
and
elsewhere. The storage medium associated with the processor for storing
methods,
programs, codes, program instructions or other type of instructions capable of
being
executed by the computing or processing device may include but may not be
limited to
one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache
and the like.
[00356] A processor may include one or more cores that may enhance speed
and performance of a multiprocessor. In embodiments, the process may be a dual
core
processor, quad core processors, other chip-level multiprocessor and the like
that
combine two or more independent cores (called a die).
[00357] The methods and systems described herein may be deployed in part or
in whole through a machine that executes computer software on a server,
client, firewall,
gateway, hub, router, or other such computer and/or networking hardware. The
software
program may be associated with a server that may include a file server, print
server,
domain server, internet server, intranet server and other variants such as
secondary
server, host server, distributed server and the like. The server may include
one or more of
memories, processors, computer readable media, storage media, ports (physical
and
virtual), communication devices, and interfaces capable of accessing other
servers,
clients, machines, and devices through a wired or a wireless medium, and the
like. The
methods, programs or codes as described herein and elsewhere may be executed
by the
Page 111 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
server. In addition, other devices required for execution of methods as
described in this
application may be considered as a part of the infrastructure associated with
the server.
[00358] The server may provide an interface to other devices including,
without limitation, clients, other servers, printers, database servers, print
servers, file
servers, communication servers, distributed servers and the like.
Additionally, this
coupling and/or connection may facilitate remote execution of program across
the
network. The networking of some or all of these devices may facilitate
parallel
processing of a program or method at one or more location without deviating
from the
scope of the invention. In addition, any of the devices attached to the server
through an
interface may include at least one storage medium capable of storing methods,
programs,
code and/or instructions. A central repository may provide program
instructions to be
executed on different devices. In this implementation, the remote repository
may act as a
storage medium for program code, instructions, and programs.
[00359] The software program may be associated with a client that may
include a file client, print client, domain client, internet client, intranet
client and other
variants such as secondary client, host client, distributed client and the
like. The client
may include one or more of memories, processors, computer readable media,
storage
media, ports (physical and virtual), communication devices, and interfaces
capable of
accessing other clients, servers, machines, and devices through a wired or a
wireless
medium, and the like. The methods, programs or codes as described herein and
elsewhere
may be executed by the client. In addition, other devices required for
execution of
methods as described in this application may be considered as a part of the
infrastructure
associated with the client.
[00360] The client may provide an interface to other devices including,
without
limitation, servers, other clients, printers, database servers, print servers,
file servers,
communication servers, distributed servers and the like. Additionally, this
coupling
and/or connection may facilitate remote execution of program across the
network. The
networking of some or all of these devices may facilitate parallel processing
of a program
or method at one or more location without deviating from the scope of the
invention. In
addition, any of the devices attached to the client through an interface may
include at
least one storage medium capable of storing methods, programs, applications,
code
Page 112 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
and/or instructions. A central repository may provide program instructions to
be executed
on different devices. In this implementation, the remote repository may act as
a storage
medium for program code, instructions, and programs.
[00361] The methods and systems described herein may be deployed in part or
in whole through network infrastructures. The network infrastructure may
include
elements such as computing devices, servers, routers, hubs, firewalls,
clients, personal
computers, communication devices, routing devices and other active and passive
devices,
modules and/or components as known in the art. The computing and/or non-
computing
device(s) associated with the network infrastructure may include, apart from
other
components, a storage medium such as flash memory, buffer, stack, RAM, ROM and
the
like. The processes, methods, program codes, instructions described herein and
elsewhere
may be executed by one or more of the network infrastructural elements.
[00362] The methods, program codes, and instructions described herein and
elsewhere may be implemented on a cellular network having multiple cells. The
cellular
network may either be frequency division multiple access (FDMA) network or
code
division multiple access (CDMA) network. The cellular network may include
mobile
devices, cell sites, base stations, repeaters, antennas, towers, and the like.
The cell
network may be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.
[00363] The methods, programs codes, and instructions described herein and
elsewhere may be implemented on or through mobile devices. The mobile devices
may
include navigation devices, cell phones, mobile phones, mobile personal
digital
assistants, laptops, palmtops, netbooks, pagers, electronic books readers,
music players
and the like. These devices may include, apart from other components, a
storage medium
such as a flash memory, buffer, RAM, ROM and one or more computing devices.
The
computing devices associated with mobile devices may be enabled to execute
program
codes, methods, and instructions stored thereon. Alternatively, the mobile
devices may
be configured to execute instructions in collaboration with other devices. The
mobile
devices may communicate with base stations interfaced with servers and
configured to
execute program codes. The mobile devices may communicate on a peer to peer
network, mesh network, or other communications network. The program code may
be
stored on the storage medium associated with the server and executed by a
computing
Page 113 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
device embedded within the server. The base station may include a computing
device and
a storage medium. The storage device may store program codes and instructions
executed
by the computing devices associated with the base station.
[00364] The computer software, program codes, and/or instructions may be
stored and/or accessed on machine readable media that may include: computer
components, devices, and recording media that retain digital data used for
computing for
some interval of time; semiconductor storage known as random access memory
(RAM);
mass storage typically for more permanent storage, such as optical discs,
forms of
magnetic storage like hard disks, tapes, drums, cards and other types;
processor registers,
cache memory, volatile memory, non-volatile memory; optical storage such as
CD, DVD;
removable media such as flash memory (e.g. USB sticks or keys), floppy disks,
magnetic
tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable
mass
storage, off-line, and the like; other computer memory such as dynamic memory,
static
memory, read/write storage, mutable storage, read only, random access,
sequential
access, location addressable, file addressable, content addressable, network
attached
storage, storage area network, bar codes, magnetic ink, and the like.
[00365] The methods and systems described herein may transform physical
and/or or intangible items from one state to another. The methods and systems
described
herein may also transform data representing physical and/or intangible items
from one
state to another.
[00366] The elements described and depicted herein, including in flow charts
and block diagrams throughout the figures, imply logical boundaries between
the
elements. However, according to software or hardware engineering practices,
the
depicted elements and the functions thereof may be implemented on machines
through
computer executable media having a processor capable of executing program
instructions
stored thereon as a monolithic software structure, as standalone software
modules, or as
modules that employ external routines, code, services, and so forth, or any
combination
of these, and all such implementations may be within the scope of the present
disclosure.
Examples of such machines may include, but may not be limited to, personal
digital
assistants, laptops, personal computers, mobile phones, other handheld
computing
devices, medical equipment, wired or wireless communication devices,
transducers,
Page 114 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
chips, calculators, satellites, tablet PCs, electronic books, gadgets,
electronic devices,
devices having artificial intelligence, computing devices, networking
equipments,
servers, routers and the like. Furthermore, the elements depicted in the flow
chart and
block diagrams or any other logical component may be implemented on a machine
capable of executing program instructions. Thus, while the foregoing drawings
and
descriptions set forth functional aspects of the disclosed systems, no
particular
arrangement of software for implementing these functional aspects should be
inferred
from these descriptions unless explicitly stated or otherwise clear from the
context.
Similarly, it will be appreciated that the various steps identified and
described above may
be varied, and that the order of steps may be adapted to particular
applications of the
techniques disclosed herein. All such variations and modifications are
intended to fall
within the scope of this disclosure. As such, the depiction and/or description
of an order
for various steps should not be understood to require a particular order of
execution for
those steps, unless required by a particular application, or explicitly stated
or otherwise
clear from the context.
[00367] The methods and/or processes described above, and steps thereof, may
be realized in hardware, software or any combination of hardware and software
suitable
for a particular application. The hardware may include a general purpose
computer and/or
dedicated computing device or specific computing device or particular aspect
or
component of a specific computing device. The processes may be realized in one
or
more microprocessors, microcontrollers, embedded microcontrollers,
programmable
digital signal processors or other programmable device, along with internal
and/or
external memory. The processes may also, or instead, be embodied in an
application
specific integrated circuit, a programmable gate array, programmable array
logic, or any
other device or combination of devices that may be configured to process
electronic
signals. It will further be appreciated that one or more of the processes may
be realized as
a computer executable code capable of being executed on a machine readable
medium.
[00368] The computer executable code may be created using a structured
programming language such as C, an object oriented programming language such
as
C++, or any other high-level or low-level programming language (including
assembly
languages, hardware description languages, and database programming languages
and
Page 115 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
technologies) that may be stored, compiled or interpreted to run on one of the
above
devices, as well as heterogeneous combinations of processors, processor
architectures, or
combinations of different hardware and software, or any other machine capable
of
executing program instructions.
[00369] Thus, in one aspect, each method described above and combinations
thereof may be embodied in computer executable code that, when executing on
one or
more computing devices, performs the steps thereof. In another aspect, the
methods may
be embodied in systems that perform the steps thereof, and may be distributed
across
devices in a number of ways, or all of the functionality may be integrated
into a
dedicated, standalone device or other hardware. In another aspect, the means
for
performing the steps associated with the processes described above may include
any of
the hardware and/or software described above. All such permutations and
combinations
are intended to fall within the scope of the present disclosure.
[00370] Furthermore, embodiments may take the form of a computer program
product on a computer-readable storage medium having computer-readable program
instructions (e.g., computer software) embodied in the storage medium. Some
embodiments may take the form of web-implemented computer software. Any
suitable
computer-readable storage medium may be utilized including hard disks, CD-
ROMs,
optical storage devices, magnetic storage devices, or the like.
[00371] It will be understood that each block of the block diagrams and
flowchart illustrations, and combinations of blocks in the block diagrams and
flowchart
illustrations, respectively, can be implemented by computer program
instructions. These
computer program instructions may be loaded onto a general purpose computer,
special
purpose computer, or other programmable data processing apparatus to produce a
machine, such that the instructions which execute on the computer or other
programmable data processing apparatus create a means for implementing the
functions
specified in the flowchart block or blocks.
[00372] These computer program instructions may also be stored in a
computer-readable memory that can direct a computer or other programmable data
processing apparatus to function in a particular manner, such that the
instructions stored
in the computer-readable memory produce an article of manufacture including
computer-
Page 116 of 122
CA 02711519 2010-07-06
WO 2009/089292 PCT/US2009/030347
readable instructions for implementing the function specified in the flowchart
block or
blocks. The computer program instructions may also be loaded onto a computer
or other
programmable data processing apparatus to cause a series of operational steps
to be
performed on the computer or other programmable apparatus to produce a
computer-
implemented process such that the instructions that execute on the computer or
other
programmable apparatus provide steps for implementing the functions specified
in the
flowchart block or blocks.
[00373] Accordingly, blocks of the block diagrams and flowchart illustrations
support combinations of means for performing the specified functions,
combinations of
steps for performing the specified functions, program instruction means for
performing
the specified functions, and so on. It will also be understood that each block
of the block
diagrams and flowchart illustrations, and combinations of blocks in the block
diagrams
and flowchart illustrations, can be implemented by special purpose hardware-
based
computer systems that perform the specified functions or steps, or
combinations of
special purpose hardware and computer instructions.
[00374] While the invention has been disclosed in connection with the
preferred embodiments shown and described in detail, various modifications and
improvements thereon will become readily apparent to those skilled in the art.
Accordingly, the spirit and scope of the present invention is not to be
limited by the
foregoing examples, but is to be understood in the broadest sense allowable by
law.
[00375] All documents referenced herein are hereby incorporated by reference.
Page 117 of 122
