Note: Descriptions are shown in the official language in which they were submitted.
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
1
IMAGING DEVICE OF FACIAL TOPOGRAPHY WITH MULTIPLE LIGHT
SOURCE FLASH PHOTOGRAPHY AND METHOD OF BLENDING SAME
FIELD OF THE INVENTION
The present invention relates to imaging device of facial topography with
multiple light source flash photography and method of blending same which
combines a set of images of a fixed human face where light source position is
varied
between images. This blending method enhances highlights and shadows of facial
topographic details that are present in various dermatological conditions,
such as
acne, moles, keloids, scarring from trauma, accidents and/or infections, etc.
BACKGROUND OF THE INVENTION
The use of two-dimensional digital photography in dermatology has been well
documented. Techniques that have been pursued include: flash, polarized light,
and
fluorescence digital photography. Flash photography approaches employ single
image
captures in semi-standard facial positioning defined by a trained clinical
photographer. The digital flash is positioned such that salient details in a
given single
shot are enhanced. Polarized light photography encompasses approaches
utilizing
parallel polarized, perpendicular polarized and cross polarized filtering. The
use of
polarizing filters aims to reduce ambient glare reflected from the facial skin
surface to
enhance visualization of details below. Fluorescence digital photography
utilizes light
filtering to capture certain wavelengths of light emitted by areas of interest
on the face
(i.e. acne lesions) while ignoring light emission from other facial areas.
For example, US Patent Nos. 6,177,984 and 6,437,856 to Jacques propose,
among other things, a video imaging device including a light source, a
detector, and
an optical polarization system for video imaging of superficial biological
tissue layers.
The device relies on taking a set of measurement at different polarization
orientations
so as to render a new image that is independent of the light reflected from
the surface
of a tissue sample and that is independent of light scattered from deep tissue
layers.
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
2
However, such removal of specular reflection based on polarization is not
suitable for
capturing skin details, such as acne, moles, keloids, scarring from trauma,
accidents,
and/or infections, etc, in a two-dimensional image / video.
US Patent No. 7,764,303 to Pote et al. (or "Pote") proposes an imaging station
for taking a plurality of digital images of a subject under a plurality of
illuminating
conditions and storing and analyzing the digital images. However, Pote is
silent about
the analysis or processing of captured images.
US Patent Application Publication No. 2011/0304705 by Kantor et al. (or
"Kantor") proposes an optical device for imaging and measuring characteristics
of an
objects surface shape, surface spectral reflectance and structure of sub-
surface layers.
For doing so, Kantor makes various measurements to calculate surface elevation
based on light intensity, and then reconstruct a three-dimensional image
therefrom.
However, for making such measurement, specular reflections are detriment to
its
calculation and thus are considered as "noise" that need to be filtered out;
which
would increase complexity in its image capturing system/process, measurement
and/or its computation.
As shown above, many of the proposed techniques or processes are not
suitable or cost effective for characterising skin details, i.e. acne, moles,
keloids and
other surface growth, and scarring from trauma, accidents, infections etc, on
a two-
dimensional or on a flat image. As suggested in the literature, various
attempts have
been made for standardizing methods of capturing images; however, no gold
standard
has yet been established.
Therefore, there exists a need for developing an effective approach to imaging
facial surface details including elevations and depressions, that provides as
good or
better topographic information than current practice would allow practitioners
to
obtain consistent images of patients that can be easily compared over time.
SUMMARY OF THE INVENTION
The present invention provides imaging of facial topography with multiple
light source flash photography and method of blending same.
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
3
According to one aspect of the invention, it provides an imaging device of
facial topography, comprising a light source for illuminating a facial surface
of
interest from at least three different positions circumferentially around the
facial
surface of interest, and said at least three different positions are equally
circumferentially distanced around the facial surface of interest, an image
capturing
device, capturing images of the facial surface from a fixed position above the
facial
surface, each captured images having the light source illuminating the facial
surface
from different one of said at least three different positions, and an image
blending
device for blending said at least three images.
According to another aspect of the invention, it provides a method of imaging
facial topography, the method comprising the steps of capturing at least three
images
of a facial surface of interest from a fixed position, each of said at least
three images
having a light source illuminated from different one of at least three
positions which
are equally circumferentially distanced from each other, and overlaying said
at least
three images by blending to produce a blended image.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will now be described in more detail with reference to
the accompanying drawings, in which:
Figure IA illustrates a schematic view of a preferred embodiment of the
present invention;
Figure 1B illustrates a flow diagram of a preferred method of overlaying
captured images by blending in accordance with the present invention;
Figure 2A illustrates a side plan view of a three-point support device 10 of a
preferred embodiment of the present invention, being applied to a person of
interest S'
to be imaged;
Figure 2B illustrates a perspective view of the three point support device 10;
Figure 3 illustrates a schematic view of the prototype of the image capturing
device 6 used to create FIGS. 4 to 7;
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
4
Figure 4 illustrates three captured images (original images are in color, the
images have been grey-scaled) of a first representative facial surface with
three light
sources at 0, 120 and 240 degrees and one blended image created from the three
captured images using the darken blending mode of Adobe Photoshop;
Figure 4A illustrates a blended image using lighten blending mode of Adobe
Photoshop based on the three captured images shown in Figure 4;
Figure 5 illustrates three captured images (original images are in color,
images
have been grey-scaled) of a second representative facial surface with three
light
sources at 0, 120 and 240 degrees and the blended image created from the three
captured images using the darken blending mode of Adobe Photoshop;
Figure 5A illustrates a blended image using lighten blending mode of Adobe
Photoshop based on the three captured images shown in Figure 5;
Figures 6 illustrates three captured images (original images are in color,
images have been grey-scaled) of a third representative facial surface with
three light
sources at 0, 120 and 240 degrees and the blended image created from the three
captured images using the darken blending mode of Adobe Photoshop;
Figure 6A illustrates a blended image using lighten blending mode of Adobe
Photoshop based on the three captured images shown in Figure 6;
Figures 7 illustrates three captured images (original images are in color,
images have been grey-scaled) of a fourth representative facial surface with
three light
sources at 0, 120 and 240 degrees and the blended image created from the three
captured images using the darken blending mode of Adobe Photoshop; and
Figure 7A illustrates a blended image using lighten blending mode of Adobe
Photoshop based on the three captured images shown in Figure 7.
DETAILED DESCRIPTION OF THE INVENTION
The present invention provides an image capturing and blending method that
also accentuates salient topographic information using a superposition of
images
captured with varying light source directions. This method for blending images
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
represents a new approach to obtaining pseudo-three dimensional information of
a
facial skin surface in a two dimensional image through blending.
Blending images have been well described in digital image editing. Raw
images are represented numerically, the currently most common numerical
representation of colors is the one used in RGB (red, green, blue) values,
each of red,
green and blue can take values between 0 and 255, indicating the intensity of
red,
green and blue contained by a pixel in the image. Basic blend modes are
commercially available in image editing applications like Adobe Photoshop 8
and
GIMP TM (or GNU Image Manipulation Program) or a like.
The present invention utilizes a computer implemented method of blending
images to combine a set of images captured by an image capturing device (i.e.
a
digital camera) in which, while the position of the image capturing device
being fixed
in respect of the facial surface of interest / the subject being captured,
light
illumination at a different position is applied in each image. The image
capturing
device consists of a digital camera synchronized with one repositionable light
source
circumferentially around the facial surface of interest or multiple light
sources
positioned at a plurality of positions circumferentially around the facial
surface of
interest. In a preferred embodiment of the present invention, there are at
least three
different positions, and these positions are circumferentially evenly
distanced around
the facial surface of interest (i.e. at 0, 120 and 240 degrees with the equal
distance or
radius from the centre of the surface). In compiling a set of images, the
capturing
device's position relative to the facial surface of interest and the facial
orientation at
large are fixed.
To maintain the position of the face being captured, a three-point support
device resting on the chin, occiput and shoulders of the individual to be
imaged may
be employed. This three point support device would sit on top of the
individual's
shoulders and consists of adjustable arms to be in contact with the
individual's chin
and occiput.
Each image in the set consists of the same facial surface area of interest
illuminated by light from a different angle. This set of images taken with
different
combinations of light source position are combined using a blending algorithm
to
enhance light and shadow details that provide topographic information in the
facial
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
6
surface area of interest. The present invention utilizes a computer-
implemented
arithmetic blending method or mode(s), lighten and/or darken to combine the
images
in the set.
For example, each pixel of a captured raw digital image in the set would have
an RGB value (red, green, blue) between 0 and 255. The lighten blending
algorithm
combines the images in the set such that the final overlaid / combined image
consists
of the 'lightest' (i.e. largest RGB values) pixels from each individual image.
The
darken blending algorithm overlays / combines the images in the set such that
the
final overlaid / combined image consists of the 'darkest' (i.e. smallest RGB
values)
pixels from each individual image. The lighten blending typically enhances
highlights
around a feature while the darken blending typically enhances shadows around a
feature.
A prototype for the image capturing device of the present invention consists
of
a Canon 8 EOS 60D Digital SLR digital camera with a Canon EF 100 mm f/2.8 USM
lens and a Canon Speedlite 8 430EX II as the repositionable light source.
Images are
captured using the following camera settings: exposure 1/100s, aperture f/10,
focal
length 100mm, ISO speed 100. The light source is set to 1/4 flash intensity.
A constant
distance of 72 cm is maintained from the lens to the surface of interest. The
light
source is oriented at 0, 120, and 240 degrees around the surface of interest
in an arc
with a radius of 30.5 cm from the centre of the surface of interest. The light
source is
oriented to provide tangential lighting to the surface of interest during
imaging. The
light source is synchronized to the image capture by the detector. Ambient
lighting is
reduced to a minimum and images are captured in a light controlled room. Four
surfaces of interest were constructed from flesh coloured plasticine
representative of a
rounded human cheek. These surfaces of interest are approximately 15.24 cm in
diameter. These surfaces have 20 randomly distributed lesions (10 elevations
and 10
depressions of random size).
In a preferred embodiment of the present invention, the image set for each
surface of interest would contain at least three captured images with the
light source
in three different positions as described above. The three images are overlaid
and
combined in an image editing application software, for example, Adobe
Photoshop 8
using the darken blending algorithm to enhance the shadowing around elevations
and
depressions on the surface. The darken blending mode overlays and combines the
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
7
images such that the final image consists of the darkest pixel from the three
separate
images.
In another embodiment of the present invention, when more images with more
light sources are added, a fuzzy approach may be preferable and be utilized to
reconstruct or blend an image therefrom. For example, thresholds can be
selected such
that an average of the pixels with RGB intensity values above (i.e. in a fuzzy
lighten
approach) or below (i.e. in a fuzzy darken approach) the threshold are used in
the final
blended image.
FIG. 1A illustrates a schematic view of a preferred embodiment of the present
invention. A person 4 is positioned with multiple light sources 1, 2, and 3
circumferentially around to illuminate the surface of interest S. First, only
the first
light source 1 is used to illuminate the surface of interest S via the first
illuminated
light 1A synchronized to capture an image by an image capturing device 6. Raw
image data is transferred from the image capturing device 6 via communication
means
(i.e. wired or wireless communication or wired or wireless communication
network)
or stored in a data storage medium (i.e. memory card, hard disc, CD, DVD, etc,
not
shown) and is accessible from a computing device 7 (such as personal computer,
handheld computing device, embedded or dedicated computing device or module,
etc)
as an image set as a first image 1B. Then, only the second light source 2 is
used to
illuminate the surface of interest S by second light 2A to produce the second
image
2B in the set. Finally, only the third light source 3 is used to create image
3B via
illumination from third light 3A. A multitude of light sources in a multitude
of
positions may be included to produce more images in the set where the
positions of
the light sources are not coincident. The images in the set are combined by
the
computing device 7 using a blending mode (or computer-implemented image
blending method) to produce a final combined or blended image 8 that enhances
light
and shadow detail of topographic information on the surface of interest S.
FIG. 1B illustrates a flow diagram of a preferred method of overlaying images
by blending in accordance with the present invention. A plurality of images
captured
using the image capturing device 6, each of which having the light source 1, 2
or 3 (or
other, not shown) illuminating the facial surface of interest S from different
position
from others are taken at steps 100-1, 100-2 ... 100-m. Then, each of these
images are
converted into digital format / representations, such as RGB values, 201-1,
201-2, ...
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
8
201-m by a known method or equivalent processed by a computing device (not
shown). Those RGB values 201-1, 201-2, ... 201-m are then manipulated by the
computing device for overlaying the images 100-1, 100-2 ... 100-m by blending.
In a
preferred embodiment of the present invention, a darken blending mode, in
which the
lowest RGB value among 201-1, 201-2, ... 201-m at every pixel position are
selected
at step 202. In another preferred embodiment of the present invention, lighten
blending mode (i.e. selecting largest RGB values) may be used instead of
darkening
blending mode. In yet another preferred embodiment of the present invention, a
predetermined threshold is set by a user, an average light intensity (or an
average
RGB value) is calculated with the pixels of images in the given image set
above said
predetermined threshold (i.e. in a fuzzy lighten blending mode) or below said
predetermined threshold (i.e. in a fuzzy darken blending mode) to produce a
combined RGB value in the blended image. Once RGB values are calculated and
collected at step 203, the blended image is produced at step 300.
FIG. 2A illustrates a side plan view of a three-point support device 10 of a
preferred embodiment of the present invention, being applied to a person of
interest S'
to be imaged. The person of interest S' is seated in a chair 14. The support
device's
shoulder support 13 is attached to the back rest of the chair 14. The shoulder
support
13 contacts and rests on the shoulders of the person of interest S'. Occiput
support 11
is adjusted to be in contact with and to rest on the occiput of the person of
interest S'
by manipulating the length of the adjustable support arm 11A. The adjustable
support
arm 11A is extendable and retractable, and may further comprise fastening
means (i.e.
handle, screws or latching device, not shown) for releasably fixing the length
of the
adjustable support arm 11A at the desirable length. Chin support 12 may be
adjusted
to be in contact with and to rest on the chin of the person of interest S' by
manipulating the adjustable support arm 12A. Similarly, length of the
adjustable
support arm 12A is extendable and retractable, and may further comprise
fastening
means (not shown) for fastening the arm 12A at a desirable length. The three
points
of contact by the support device 10 on the person of interest S' prevent or
minimize
movement and head turning of the person of interest S' while attachment of the
shoulder support 13 to the chair 14 prevents and/or minimizes any other
movement.
FIG. 2B illustrates a perspective view of the three point support device 10 of
a
preferred embodiment of the present invention. The three point support device
10
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
9
includes occiput support 11 being attached to an adjustable occiput support
arm 11A.
The adjustable occiput support arm 11A having a threaded inserting member 21
and a
receiving member 22, where the threaded inserting member 21 slidably engaged
with
the receiving member 22, such that the adjustable occiput support arm 11A
longitudinally extends or retracts along its length. The receiving member 22
is
connected to and extends from the shoulder support 13. The receiving member 22
further comprises a fastening means 25, including a threaded aperture 24 and a
screw
23. Once the threaded inserting member 21 is adjusted to a desirable length,
the
screw 23 is driven to engage with a thread on the threaded inserting member 21
for
releas ably attaching or holding the threaded inserting member 21 to the
receiving
member 22. Various other means or structure for adjustable occiput support arm
11A
than the one shown in the present drawing may be used by a person ordinary
skilled in
the pertinent art.
Similarly, a chin support 12 being attached to an adjustable chin support arm
12A. The adjustable chin support arm 12A having a threaded inserting member 31
and a receiving member 32, where the threaded inserting member 31 slidably
engaged
with the receiving member 32, such that the adjustable chin support arm 12A
longitudinally extends or retracts along its length. The receiving member 32
is
connected to and extends from the shoulder support 33. The receiving member 32
further comprises a fastening means 35, including a threaded aperture 34 and a
screw
33. Once the threaded inserting member 31 is adjusted to a desirable length,
the
screw 33 is driven to engage with a thread on the threaded inserting member 31
for
releasably attaching or holding the threaded inserting member 31 to the
receiving
member 32. Various other means or structure for adjustable chin support arm
12A
than the one shown in the present drawing may be used by a person ordinary
skilled in
the pertinent art.
FIG. 3 illustrates a schematic view of the prototype of the image capturing
device 6 used to create FIGS. 4 to 7. A second light source 2 is used to
illuminate the
surface of interest S via light 2A. The light is synchronized with the image
capturing
device 6 to capture the raw image data. Similarly, a third light source 3 is
used to
illuminate the surface of interest S via light 3A for image capture by image
capturing
device 6. Additional light sources can be included circumferentially around
the
surface of interest S such that the light source positions are not coincident.
The
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
prototype consisted of three such light sources at positions of 0, 120 and 240
degrees
circumferentially around the surface of interest S.
FIG. 4 shows three captured images (original images are in color, images have
been grey-scaled) of a first representative facial surface with three light
sources at 0,
120 and 240 degrees and the blended image created from the three captured
images
using the darken blending mode of Adobe Photoshop O. The first representative
facial
surface of interest has ten (10) elevations and ten (10) depressions of random
size and
distribution.
FIG. 4A shows a blended image by lighten blending mode of Adobe
Photoshop based on the three captured images shown in FIG. 4.
FIG. 5 shows three captured images (original images are in color, images have
been grey-scaled) of a second representative facial surface with three light
sources at
0, 120 and 240 degrees and the blended image using the darken blending mode of
Adobe Photoshop O. The second representative facial surface of interest has
ten (10)
elevations and ten (10) depressions of random size and distribution.
FIG. 5A shows a blended image by lighten blending mode of Adobe
Photoshop based on the three captured images shown in FIG. 5.
FIG. 6 shows three captured images (original images are in color, images have
been grey-scaled) of a third representative facial surface with three light
sources at 0,
120 and 240 degrees and the blended image using the darken blending mode of
Adobe
Photoshop O. The third representative facial surface of interest has ten (10)
elevations
and ten (10) depressions of random size and distribution.
FIG. 6A shows a blended image by lighten blending mode of Adobe
Photoshop based on the three captured images shown in FIG. 6.
FIG. 7 shows three captured images (original images are in color, images have
been grey-scaled) of a fourth representative facial surface with three light
sources at 0,
120 and 240 degrees and the blended image using the darken blending mode of
Adobe
Photoshop S. The fourth representative facial surface of interest has ten (10)
elevations and ten (10) depressions of random size and distribution.
FIG. 7A shows a blended image by lighten blending mode of Adobe
Photoshop based on the three captured images shown in FIG. 7.
CA 02885775 2015-03-23
WO 2014/047712
PCT/CA2012/000886
11
It is to be understood that the embodiments and variations shown and
described herein are merely illustrations of the principles of this invention
and that
various modifications may be implemented by those skilled in the art. For
example,
an image capturing device (i.e. web camera, digital camera, etc) and
repositionable or
multiple lighting device(s) may be in communication with a computing device
(i.e. a
personal computer, embedded computing device, or a like) via a communication
means (i.e. wired or wireless communication, wired or wireless network, etc),
where
the computing device controls the image device and the lighting device(s) for
capturing images and processing captured images.
