Note: Descriptions are shown in the official language in which they were submitted.
CA 02313693 2000-07-19
n\~GI.LCt..~'. ~ , .
Patent Application ~ ~ ~j (~ ~ ~
of
W. Steve G. Mann PRO~ha i ~ '. _ _ . ~~~
for
IMPLANTABLE CAMERA OR PERSONAL IMAGING SYSTEM
WITH IMPLANTABLE PORTION
of which the following is a specification:
FIELD OF THE INVENTION
The present invention pertains generally to an apparatus, system, or the like,
for
being implanted in the body.
BACKGROUND OF THE INVENTION
Humanistic Intelligence (HI) is intelligence that arises in a natural
cybernetic way,
through having a constancy of user-interface, by way of an "always-ready"
computer
system or imaging system.
The wearable computer, and wearable photographic apparatus, invented in the
1970s and early 1980s, is an example of an apparatus that embodies HI. This
invention
and its realization of HI is described in Proceedings of the IEEE, Vol. 86,
No. 11, and
can also be found by Internet searching for keywords such as "eyetap" ,
"humanistic
intelligence" , "wearcam" , and the like.
In U.S. Pat. No. 6063117, "Porous orbital implant structure", Arthur C. Perry,
(16418 La Via Feliz, Rancho Santa Fe, CA 92067-1102) describes a porous
structure
for implantation into the orbital cavity, as well as a surgical method for
installing the
implant to obtain rapid ingrowth of vascular and connective tissues.
In U.S. Pat. No. 6033437, "Pegs for orbital implants", Arthur C. Perry
describes
a motility peg for an orbital implant. The peg is generally placed in vivo. It
also
1
CA 02313693 2000-07-19
~1~'T'E~ LF!'"~a ~ w ~ ~n('PFF~T'v
2000
PROf~r,~~ir_ ;....... :~'uELLE
allows for removable attachment to an artificial eye, typically comprising a
convex~~
surface articulating with a concave surface on the artificial eye.
In U.S. Pat. No. 5967772, "Orthodontic anchor system", James B. Gray (715
Shade Tree Ter., Roswell, GA 30075) describes an orthodontic anchor system com-
prised of an anchor for attachment (directly or indirectly to) a palate, a
member, a
tooth or one or more dental implants or subperiosteal implants.
In U.S. Pat. No. 5947723, "Titanium orthodontic appliances", Assigned to
GAC International, Mikio Mottate (Ohkuma-machi, JP), Masaaki Orikasa (Ohkuma-
machi, JP), and Kikuo Nishi (Haramachi, JP), describe orthodontic appliances
of
titanium alloy which avoid toxic or allergic reactions.
In U.S. Pat. No. 6077073, "Light emitting diode-array light apparatus",
Gregory
S. Jacob (9672 Reding Cir., Des Plaines, IL 60016) describes an array of low
voltage
LEDs in a clear housing conforming to an approximate shape of dentition.
During use
the apparatus of the invention emits light for curing adhesives, sealants,
whitening or
coloring agents, or the like.
In U.S. Pat. No. 6059571, "Method for embedding mark in denture and im-
plement for making recess used therefor", Hisashi Kishigam (2-25-21,
Shimizugaoka,
Sumiyoshi-ku, Osaka, JP) describes a method of embedding a microprocessor
inden-
tification device a denture, allegedly for management of dentures at a
dentist's or
dental mechanic's office.
In U.S. Pat. No. 6083248, "World wide patient location and data telemetry
system for implantable medical devices", assigned to Medtronic, Inc.
(Minneapolis,
MN), David L. Thompson (Fridley, MN), describes an implantable telemetry
system
with two way communications capability. The system locates a device implanted
in
an ambulatory patient patient to selectively monitor device function, alter
device op-
erating parameters and modes, and to communicate with a patient using a
transceiver
for communicating between the implanted device and an external patient communi-
cations device that is worn or kept by the patient in close range. The system
includes
2
CA 02313693 2000-07-19
iNTELI.FCTI.IeI ~PpPERTY
r,". .
J U ~. ~ ~ 2000
~..pRO'~r~'~~ic,.:., -. ~:i~;TUEt~
a communication link with a medical support network, global
posi~~i'g'S~'2~~°'~~'
receiver, and a patient activated link.
In Canadian Patent 2249976, WEARABLE CAMERA SYSTEM WITH VIEWFINDER
MEANS Nov. 2 , 1999 (filed Oct. 15, 1998), Mann (the inventor in the present
ap-
plication) describes an EyeTap (TM) device for providing a Point Of Eye (POE)
imaging system that is preferably responsive to rays of light that would
otherwise
pass through the center of projection of a lens of an eye of the wearer of the
EyeTap
device.
SUMMARY OF THE INVENTION
A preferred embodiment of the invention comprises a light sensitive element,
light
sensor, or the like, for being implanted in the body together with a processor
for
reading out quantities of light, or quantities that are in some way responsive
to
quantities of light.
In this disclosure, the term "implanted" includes an apparatus such as an
occular
implant, an artificial eye, a denture, mouthguard, or the like, that is
inserted into the
body where it could remain in the body after removal of clothing, or the like,
and in
which insertion does not necessarily require cutting the body open in surgery,
or the
like.
In a typical embodiment of the invention, there are two or three components:
~ a power source;
~ an implantable apparatus having a power receiver and picture sender; and
~ a picture receiver.
The power source for the implantable apparatus may be derived, at least
partially,
from the body itself, or it may be wholly supplied externally, possibly with
only a
local power storage or filter capacitor internally.
3
CA 02313693 2000-07-19
In embodiments where the power is supplied, at least in part, externally, the
user
wears a device that transmits power to the implant, either to run it, or to
charge up
its internal storage system.
In the preferred embodiment, a higher level of power is supplied externally to
charge the implantable apparatus, overnight and a lesser level of power is
supplied to
the implantable device from a wearable apparatus during daily operation.
Preferably
a high power charger charges both the wearable apparatus and the implantable
ap-
paratus, and a low power charger in the wearable apparatus continues to charge
the
implantable apparatus during daily use.
There are some situations where it is desired that the implant continue to run
when separated from the wearable apparatus. For example, in using the
invention as
a Personal Safety Device (PSD) for crime reduction, it is preferable that the
device
continue to operate despite theft of clothing and personal belongings. In this
way,
there is better chance of capturing and identifying the perpetrator of the
theft. A
vicious attack, rape, or theft of clothing, could therefore be punished by
catching
the perpetrator, and further using image data as evidence in a courtroom, for
the
prosecution of criminal activity.
In some embodiments, the implantable device is housed in dental apparatus such
as orthodontics, a mouthguard, dentures, dental work, or the like, where the
image
sensor and associated optics are typically aimed out through a gap in the
front teeth,
and image capture is triggered by opening the mouth. In other embodiments, the
apparatus takes the form of an occular implant, artificial eye, or the like,
such that it
may be used to capture Point Of Eye (POE) documentary of sporting events such
as
summer Olympics freestyle, butterfly, breaststroke, IM, etc., from the
perspective of a
participant, which could not otherwise be captured by traditional wearable
computing
apparatus of the wearable wireless webcam or EyeTap (TM) variety, as described
in
IEEE Proceedings of ISWC-98, Pittsburgh, Pennsylvania, October 19-20, 1998 or
by
an Internet search on keywords "wearcam" and "eyetap" .
4
CA 02313693 2000-07-19
Moreover, the POE embodiment of the invention captures a true Point Of Eye,
and not just a Point Of View (POV) as might be captured by a headworn camera.
l~Moreover, although both the POE embodiment of the invention and the EyeTap
(TM) camera both capture a true Point Of Eye, in the sense that the eye itself
is,
in effect, the camera in both cases, the present invention carries this effect
one step
further, by including the eyelids and eyelashes, as seen from within the eye,
in the
view. Thus, through appropriate optics, a natural inside-the-eye view is
established.
Additionally, using a CrossEyeTap (TM) system, in which the POE camera sends
a picture over to a display in view of the other eye, a natural looking
occular prosthesis
evolves.
In particular, loss of an eye due to trauma, disease, or the like, has
negative
impact on the ability to do that requires depth perception, such as is desired
in
leisure activities of gaming (e.g. tennis, volleyball, and the like) as well
as job-related
activities). Moreover, the negative impact on self-image arising from poor
aiming of
an artificial eye, during conversation or the like, has an adverse effect on
self confidence
and self esteem.
The CrossEyeTap system may be embodied in the invention in order to give a
more lifelike quality to the artificial eye. Even if image capture is not
needed, there
is thus benefit to merely monitoring the output of the artificial eye in the
normal
eye. Thus by sending the output of the artifial eye into the normal eye, there
arises
a feedback mechanism wherein the artificial eye comes to life again as a
lifelike input
into the visual system. As the brain adapts to this new way of seeing, the
artificial
eye takes on a new life to greatly improve the self esteem and self confidence
of the
wearer. Simply knowing that the other eye is not "dead" provides a good deal
of such
self confidence. Monitoring its output continues to build not only the self
esteem, but
also the movement control of the eye, such that the vergence of gaze returns.
By displaying the material distinctly (e.g. in red, or in processed video) on
the
other eye, a feedback system evolves to control the artificial eye. Moreover,
the
CA 02313693 2000-07-19
additonal POE information visible to the normal eye helps in regaining greater
sense
of depth perception.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention will now be described in more detail, by way of examples which
in no way are meant to limit the scope of the invention, but, rather, these
examples
will serve to illustrate the invention with reference to the accompanying
drawings, in
which:
FIG. 1A illustrates the implantable and external portions of the personal
imaging
system.
FIG. 1B illustrates the implantable portion of the personal imaging system.
FIG. 2 illustrates a dental embodiment.
FIG. 3A illustrates a cross section of the eye portion of an eye embodiment.
FIG. 3B illustrates the components of the eye portion an eye embodiment.
FIG. 4A illustrates an artificial iris of an artificial eye implemented with
two
durable glass conductive ITO (indium-tin oxide) coatings having transmissivity
in
the visible region of light.
FIG. 4B illustrates an electromagnetically powered artificial iris of an
artificial
eye.
FIG. 4C illustrates a photonically powered artificial iris of an artificial
eye.
FIG. 5 shows a crosseye embodiment in which an aremac for viewing by a first
eye is responsive to an output of an artificial eye.
FIG. 6 shows an embodiment with a stereo depth processor for being responsive
to rays of light passing into both eyes, an output of the stereo depth
processor for
being viewed by one eye.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
6
CA 02313693 2000-07-19
While the invention shall now be described with reference to the preferred em-
bodiments shown in the drawings, it should be understood that the description
is not
to limit the invention only to the particular embodiments shown but rather to
cover
all alterations, modifications and equivalent arrangements possible within the
scope
of the appended claims.
In all aspects of the present invention, references to "light sensor" mean any
device
to measure, quantify, or be responsive to light, in either continuous or
discrete steps.
References to "processor" , or "computer" shall include sequential
instruction, par
allel instruction, and special purpose architectures such as digital signal
processing
hardware, Field Programmable Gate Arrays (FPGAs), programmable logic devices,
Programmable Interface Controllers (PICs), as well as analog signal processing
de-
vices, digital or analog or other camera control units (CCUs), and the like.
When it is said that object "A" is "borne" by object "B", this shall include
the
possibilities that A is attached to B, that A is bonded onto the surface of B,
that A
is embedded inside B, that A is part of B, that A is built into B, or that A
is B.
FIG. lA shows a personal imaging system. With reference to Fig lA, an im-
plantable portion 115 is powered by at least one of:
~ an externally stored power source 124;
~ an external power source 125;
~ an internally generated power source 126; or
~ an internally stored power source 127,
each of which is denoted in dashed lines since it will not necessarily be
present in
all embodiments of the invention. If present in a particular embodiment,
externally
stored power source 124 preferably takes the form of a body worn battery pack
which
also powers a body worn computer system. If present in a particular
embodiment,
external power source 125 wirelessly sends power to implantable portion 115.
In the
7
CA 02313693 2000-07-19
preferred embodiments, the sending of power is by inductive coupling. An
internally
generated power source 126 may also be present, in the form of a source that
de-
rives power from body muscle movement, from the environment (as received
light,
or received electromagnetic energy possibly in the absence of a specific
source 125
thereof), or the like.
Implantable portion 115 may also be powered by an internally stored power
source
127. Source 127 may be a primary battery, but in the preferred embodiments it
is
a secondary (e.g. rechareable) source deriving charge from either sources 125
or 126
depending on which is present.
External power source 125 may be mainly for charging source 127, or it may be
mainly for powering portion 115, or for both charging and powering portion
115. In
the case of charging, source 127 may take the form of an inductive coil in a
pillow for
sleeping, to charge portion 115 overnight.
Source 125 may also be built into eyeglasses worn near the portion 115, to
operate
it, or to charge it. Source 125 may provide a combination of operation and
charging,
and source 125 may also be split into parts, e.g. a source 125 in eyeglasses
as well as
a source 125 in a pillow, etc., so that a user of the apparatus may have more
than
one external power source 125.
Sources 126 and 127, if either or both are present in a particular embodiment,
are
implanted in the body.
Implantable portion 115 sends picture information to a picture receiver 1408.
Picture receiver 1408 is optinally powered by source 125, by its own source,
or by a
combination of source 125 and its own source. A wireless link 140L carries
picture
information from portion 115 to receiver 1408.
An optional display 140D shows the user a displayed output of his or her im-
plantable portion 115.
In a preferred embodiment of the invention, source 125 and receiver 1408 are
housed in a single housing 140H worn by the user. Additionally, if present,
display
8
CA 02313693 2000-07-19
140D is preferably located in the same housing 140H.
FIG. 1B shows the implantable portion 115 of the personal imaging system that
was previously shown in Fig. lA. A power receiver 120 receives power from one
or
more sources (125, 126, 127 of Fig. lA). In a preferred embodiment, power
receiver
120 receives power electromagnetically, and is denoted as power receiver 120A.
In
other embodiments power receiver 120 is located in a light sensitive system
130, and
in this location, the power receiver is denoted 120B. Power receiver 120B, if
present,
receives its power from a light sensor 131 which is preferably also an imaging
array.
In the case of power receiver 120A, power is said to come from an external
power
source (125 of Fig. lA), wheras in the case of power receiver 120B, power is
said to
come from an internally generated power source (126 of Fig. lA) in which
source 126
may be considered to be sensor 131.
Power connections 121 convey power from either source 120A by way of connec-
dons 121A, or source 120B by way of connections 121B. Power is conveyed to a
light
sensitive system 130 as well as a picture transmitter 140. Optionally a
controller
150 accepts input 160 from the user 100. Preferably the implantable portion
(115 of
Fig. lA) is in the head 110 of user 100.
Sensor 131 is responsive to light, and supplies picture signals to processor
132,
which provides a picture signal to picture transmitter 140. Optics for sensor
131 is
not shown, and is either implicit, explicit, or may be manifest as, for
example, the
lens of an artificial eye.
FIG. 2 shows a dental embodiment of the implantable portion 115 of Fig. lA. A
retainer, denture, or other similar dental element 200 attaches to teeth 201,
either by
friction fit, by dental implant, or by attaching a place where teeth are
absent, if teeth
are wholly, or partially absent.
A power receiver 220A receives power electromagnetically from turns of wire in
a pickup coil 220B. This power is conveyed by connections 121A to a light
sensitive
system 230. The light sensitive system 230 contains an image sensor. An
aperture
9
CA 02313693 2000-07-19
130A may be formed by a crack in two front teeth, giving rise to a pinhole
camera.
In a preferred embodiment, an additonal lens system with aperture stop in the
region
of the gap in the front teeth is used for aperture 130A.
Connections 134 convey picture signals to a processor and transmitter 240,
having
a dipole antenna made from lengths of conducative material 240A. An input by
way
of terminals 260 allows the apparatus to be controlled by the toungue.
Terminals 260
are connected to the processor portion of processor and transmitter 240.
Terminals 260 are preferably located at the back of the mouth, so that they
are
not accidentally activated, and to conceal the movement of the tongue.
Alternatively,
muscle sensors activate taking a picture when the user smiles, as this causes
the mouth
to open. This feature may provide reciprocity: in smiling for a camera the
user can
also take a picture of the photographer.
In another preferred embodiment, a smile activated camera system is realized
by
light sensing (when sufficient or complete light enters in aperture 130A),
such that a
picture is automatically taken when there is complete input of an entire
picture frame.
A small electrical stimulus is preferably produced by processor and
transmitter 240
so that the user can taste the status of the image capture, both to remind the
user
that a picture was taken, as well as to provide additional information.
The additional information can denote how much hard drive space is left on a
body worn computer receiving the images, as well as the number of viewers on a
"hit counter" on a WWW page viewing the output of the camera. These features
prefer that picture receiver 1408 and picture transmitter 140 are both
transceivers.
Preferably they are both data packet transceivers, for communication to a body
worn
computer that servers as a repeater and Internet gateway as well as local
storage and
exposure indication. Exposure indication is calculated in processor and
transmitter
240, or may also be supplied by a remote expert viewing the picture
information. The
remote expert guide to exposure may also include head orientation and
composition
information conveyed back by taste (e.g. voltages applied inside where they
can be
CA 02313693 2000-07-19
tasted by the tongue).
Small voltages like 6 volts are easy to taste, to about a 1 volt precision, so
that
the voltage can also be used to indicate exposure and other related
information.
Voltage profiles across the apparatus can also convey spatialization and
composition
information.
FIG. 3A shows an eye implant version of the implantable portion 115 that was
shown in Fig. lA. A lens 311 of an artificial eye 310 focuses light on a
motility peg
330. The motility peg 330 contains a light sensitive system as shown in Fig.
1B, with
sensor array and processor. Connections 331 bring picture information to
processor
and transmitter 332. Preferably these connections are such that the motility
peg can
be installed after the orbital implant 300 is installed and attachment has
occurred
(e.g. a few months after orbital implant 300 is installed).
The method of installation comprises the steps of:
~ installing orbital implant 300 in the eye socket of the recipient;
~ allowing implant 300 to attach itself to tissue (muscle tissue, etc.);
~ after sufficient elapsed time for attachment, locating an appropriate
optical axis
center on implant 300;
~ drilling implant 300;
~ inserting motility peg 330 such as to form connections 331 thereto;
~ applying artificial eye 310.
Prior to installation, orbital implant 300 preferably already contains
processor
and transmitter 332 as well as pickup coil 320B for supplying power to
rectifier 320A.
Rectifier 320A has connections 321A to processor and transmitter 332, as well
as
connections 321B to the region where motility peg 330 will be later installed.
Since the exact location of motility peg 330 may depend on how implant 300
adheres to tissue, connections 331 preferably span a region of space to accept
a drill
11
CA 02313693 2000-07-19
hole from a variety of possible directions, and preferably also are resistant
to damage
from drilling.
FIG. 3B shows the components, as supplied to the installer. There are three
components to be installed in the following order:
~ orbital implant 300;
~ motility peg 330;
~ artificial eye 310.
FIG. 4A shows an artificial iris. The depth of focus of lens 311 (Fig. 3A and
3B) is
preferably variable, and responsive to light to arise in a true eye view in
which depth
of focus increases in bright light.
Alternatively it is desired to control the depth of focus away from what would
naturally occur, or from a fixed depth of focus. For example, it might be
desired to
increase depth of focus to see the eyelashes for effect, especially since the
blinking of
the eye will a provide good cinematographic effect of a true first person
perspective.
Accordingly, photochromic materials for the former case, or electrochromic
mate-
rials for the latter case, may be used to create an artificial iris.
The iris of Fig. 4A is the electrochromic variety with a processor 400 in an
arti-
ficial eye 310. Artificial eye 310 also has a ground connection 411 to a layer
410 of
durable glass with a conductive ITO (indium-tin oxide) coating having
transmissiv-
ity in the visible region, and typically having resistivity in the range of 10
to 10,000
ohms/square.
A first coating of layer 410 is separated from coatings of concentric layers
420,
430, etc., by an insulating layer. The insulating layer contains the portion
of the
artificial eye that is to be controlled or switched. The transparency of the
various
concentric rings are thus individually controlled by way of connections 421,
431, etc.,
to processor 400. Thus processor 400, receiving control information wirelessly
from
12
CA 02313693 2000-07-19
within the orbital implant, may control the depth of field and the amount of
light
admitted by lens 311.
Connections 421, 431, etc., are preferably of copper with a red enamel, and
erratic
shape, so as to have the appearance of blood vessels in the eye.
FIG. 4B shows an electromagnetically powered artificial eye, and answers the
question as to how the artificial eye will receive power. Power is received
from source
125, 126, or 127 (of Fig. lA). This power may come directly from the outside,
or
from orbital implant 300, or from a combination of these sources, or
indirectly from
outside, through the orbital implant, resending thereto, or a combination
thereof.
Power is received by way of windings 400 which also preferably have an
appearance
of blood vessels in the eye, by way of erratic winding and red enamel on fine
copper
wire. These windings power the iris connections at two iris input sections 450
and
451.
FIG. 4C depicts a photonically powered embodiment of the artificial eye, in
which
a solar cell comprised of plate regions 480 and 481 powers the input sections
450 and
451 through connections 441. Again connections 441 preferably are of red
enamelled
fine copper wire.
FIG. 5 depicts a crosseye embodiment in which implant 300 is responsive to
rays
of light in a cone of light bounded by rays 510 and 520. light within this
region
contributes to a picture signal sent by way of link 140L to a picture
processor and
picture receiver 532 housed in housing 140H together with an aremac 540D which
displays the picture information to left eye 500.
An aremac is a device for displaying picture information to an eye. An aremac
is
to flatbed scanner as a camera is to a projector. A camera maps the 3D world
to a
2D image. A scanner maps the 2D world (e.g. the page of a book, or a newspaper
article being scanned) to a 2D image. A projector takes a 2D image and sends
it out
into a 2D world. An aremac takes a 2D image and sends it out into a possibly
3D
world, e.g. the eye in which distance of focus may be arbitrary. Thus an
aremac is a
13
CA 02313693 2000-07-19
display device that preferably has some depth of field control. An ordinary
display is a
special case of an aremac, but the notion of an aremac also includes devices
that image
into the eye, such as blurry information displays and laser eye input devices.
The
etymology of the word "aremac" derives from spelling the word "camera"
backwards.
Preferably a diverter 516 comprising either a mirror or beamsplitter or
similar
reflective optical element diverts light from aremac 540D into eye 500.
Preferably
diverter 516 is curved with the concave side facing the eye 500 and aremac
540D. With
regards to the transmissivity of diverter 516 there are two preferred
embodiments:
~ a first embodiment in which diverter 516 has a partially silvered portion
515
that eye 500 can see around and, to some extent, through; and
~ a second embodiment in which an aremac is concealed in an eyepatch 540P,
possibly without the use of a diverter.
FIG. 6 depicts an embodiment with a crosseyetap camera used with an artificial
eye. Implant 300 sends a signal to depth processor 635 by way of link 140L.
Depth
processor 635 also receives an input from the viewpoint of the other eye, by
way of
an eyetap camera for the other eye. This eyetap camera is comprised of
diverter 616,
optics 632; and sensor array 631.
Depth processor 635 then computes depth information based on the inputs from
the two cameras.
Optionally, depth processor 635 may supply a picture signal to aremac 140, in
which the picture signal is responsive to depth information calculated from
the two
cameras. In this case, aremac 140D is seen in the back side of diverter 616.
A key inventive concept of the apparatus of Fig. 6 is the use of two devices
that
behave in a cameralike way. One device, an eye implant, is responsive to rays
of light
passing through the center of a lens of an artificial eye, whereas the other
is responsive
to rays of light passing through another point, in this case, the other point
being the
center of the lens of the other eye, giving rise to an auxiliary camera
tapping the real
14
CA 02313693 2000-07-19
eye 600.
This is the preferred embodiment in which the first camera is implant 300 and
the
auxiliary camera is on eye 600. Other embodiments are possible. For example,
the
auxiliary camera may be headworn, or eyeglass worn, and might, for example, be
in
the nosebridge of a pair of eyeglasses, rather than necessarily being
effectively at eye
600.
From the foregoing description, it will thus be evident that the present
invention
provides a design for a personal imaging system with at least an implantable
portion.
As various changes can be made in the above embodiments and operating methods
without departing from the spirit or scope of the invention, it is intended
that all
matter contained in the above description or shown in the accompanying
drawings
should be interpreted as illustrative and not in a limiting sense.
Variations or modifications to the design and construction of this invention,
within
the scope of the invention, may occur to those skilled in the art upon
reviewing
the disclosure herein. Such variations or modifications, if within the spirit
of this
invention, are intended to be encompassed within the scope of any claims to
patent
protection issuing upon this invention.
