Note: Descriptions are shown in the official language in which they were submitted.
CA 02275798 1999-06-29
INTc'rA PERTY
..,ry.,- . ~..'.e.F
CSI,;.,
Patent Application , ,
of Bui ,: , , . : ~, ;nCT
PFiOPRiETE ~ui~,
W. Steve G. Mann
for
WRISTWORN OR HANDHELD VIDEO PRODUCTION FACILITY
OR VIDEOCONFERENCING SYSTEM
of which the following is a specification:
FIELD OF THE INVENTION
The present invention pertains generally to a wristworn or handheld portable
elec-
tronic camera system with computer and computer screen that functions also as
a
viewfinder.
BACKGROUND OF THE INVENTION
In photography (and in movie and video production), it is often desirable to
cap-
ture events in a natural manner with minimal intervention or disturbance. In
the
present invention, the scenario to be considered is that of face-to-face
conversation
between two individuals, where one of the individuals wishes to either make an
an-
notated video diary of the conversation, or at least capture a picture of the
other
individual, without disrupting the natural flow of the conversation. In this
context,
it is desirable to create a personal video diary or personal documentary, or
to have
some kind of personal photographic or videographic memory aid that forms the
visual
equivalent of what the electronic organizers and personal digital assistants
do to help
us remember textual or syntactic information.
Current state-of-the-art photographic or video apparatus creates a visual dis-
turbance to others and attracts considerable attention on account of the
gesture of
bringing the camera up to the eye. Even if the size of the camera could be
reduced
1
CA 02275798 1999-06-29
to the point of being negligible (e.g. no bigger than the eyecup of a typical
cam-
era viewfinder, for example), the very gesture of bringing a device up to the
eye is
unnatural and attracts considerable attention, especially in establishments
such as
department stores where photography is often prohibited. However it is in
these very
establishments in which a customer may wish, for example, to have a video
record of
the clerk's statement of the refund policy or the terms of a sale. Just as
department
stores often keep a video recording of all transactions (and often even a
video record-
ing of all activity within the establishment, sometimes including a video
recording
of customers in the department store fitting rooms or changerooms), the goal
of the
present invention is to assist, for example, a customer who may wish to keep a
video
record of a transaction, interaction with a clerk, manager, refund
explanation, or the
like, or at the very least, a system that enables the customer to capture a
picture of
an official who refuses to identify himself/herself.
Although there exist a variety of covert cameras such as cameras concealed
beneath
the jewels of necktie clips, cameras concealed in baseball caps, and cameras
concealed
in eyeglasses, these cameras tend to produce inferior images, not just because
of the
technical limitations imposed by their small size, but, more importantly
because they
usually lack a viewfinder means (e.g. means of viewing the image to adjust
camera
angle for best composition). Because of the lack of viewfinder means, the
subject
matter is not necessarily centered well in the viewfinder, or even captured by
the
camera, and thus these covert cameras are not well suited to personal
documentary
or for use in a personal photographic/videographic memory assistant.
A wearable camera system (called WearCam) has been disclosed in, for example,
IEEE Computer, Vol 30, No 2, or equivalently
http://wearcam.org/ieeecomputer.html
(see also http://wearcomp.org/wearpubs.html), as well as in Proceedings of the
IEEE,
Vol 8fi, No 11, or equivalently http://hi.eecg.toronto.edu/hi.htm.
Unfortunately em-
bodiments of the WearCam invention typically require special preparation such
as
special clothing, and special wiring of the body a-priori. Such systems will
work for
2
CA 02275798 1999-06-29
those dedicated to being constantly wired with various computational and
sensory
apparatus incorporated into special clothing, but there are a good many
individuals
who may wish to obtain a similar kind of convenience without a change of
lifestyle.
The invention is related to the Personal Digital Assistants (PDA) such as the
Palm
Pilot, the so-called "Paper-Based AudioNotebook" (Lisa J. Stifelman Augmenting
Real-World Objects: A Paper-Based Audio Notebook CHI'96 Conference Companion,
pp. 199-200, April 1996), and the general ideas of personal electronic devices
such as
wristwatches, that can be used naturally while standing face-to-face with a
person
such as a department store manager or other official.
There is also an important connection to portable computers, mobile computing,
and wearable computers (See for example, IEEE Computer, Vol 30 No 2).
U.S. Pat. No.4780864 describes a combination wristwatch and bike computer.
U.S. Pat. No.5852401 held by Casio corp., describes a distress sending
wristwatch
that transmits the last known GPS location it has sensed, during time of
distress.
U.S. Pat. No.5738104 describes a wristwatch that senses EKG.
U.S. Pat. No.4106283 describes a wristwatch television.
DESCRIPTION OF THE INVENTION
The basic principle of the invention is that a camera is concealed in a device
that
has another purpose, where the other purpose of the device would involve
looking at
a display. This other non-picture-taking purpose of the device provides for a
con-
comitant cover activity. For example, the camera may be concealed in a
wristwatch,
and the display of the wristwatch may also serve as the viewfinder of the
camera.
When the user wishes to take a picture, he or she pretends to check the time
on the
wristwatch. Checking the time is therefore a concomitant cover activity. The
camera
is mounted in the watch in such a way that it points out in front of the user,
so that
a person the user is talking to will be in field of view of the camera while
the user
is pretending to check the time. Moreover, the user can pretend he/she has
trouble
3
CA 02275798 1999-06-29
focusing on the watch (as when a person who would normally wear bifocals has
for-
gotten his/her eyeglasses) and hold the watch some distance out from the eyes,
so
that it will be very close to the clerk or official that the user is taking a
picture of.
This may be useful to capture close up pictures of identifying features of the
official,
such as the shape of pores in the official's skin - features that may help
later provide
positive identification of the perpetrator of a crime.
The wearer's eyeglasses may also be special eyeglasses that either contain a
display
device to receive input from a wristwatch mounted camera, or a downprism so
that
the wearer of the special glasses can look downward at a wristwatch or the
like, with
a portion of the vision. The downprism can be concealed to look like ordinary
bifocal
eyeglasses, where the lower portion of the eyeglasses comprises a prism that
directs
light coming from below into the eyes, so that the wearer can see almost
directly
downward within a small portion of the view.
In an alternative embodiment, the camera is mounted inside a PDA. In this
case,
pulling out a PDA and holding it in front of the body while looking at the
display is
a concomitant cover activity for taking a picture.
In another embodiment, the camera is mounted in a clipboard, so that writing
or
a clipboard is a concomitant cover activity. Thus the user can either pretend
to take
notes, or actually take notes, while capturing pictures or video at the same
time.
The invention includes a camera and viewfinder for unobtrusively capturing
video
of exceptionally high compositional quality and possibly even artistic merit.
In summary, the apparatus of the invention works as follows:
~ the user engages in a concomitant cover activity;
~ the concomitant cover activity involves looking downward, intently at the
top
of a device that is held in front of the body and slightly below the head;
~ a camera in the device is aimed at a person or persons the user is talking
to;
~ the electronic display is responsive to an output of the camera, so that it
can
4
CA 02275798 1999-06-29
serve as a viewfinder, and perhaps also display other information such as
zebra
stripes in regions of overexposure, or focusing/exposure adjustment aids, as
well
as possibly compositional aids received from a compositional assistant (either
software or one or more remotely located persons assisting the user);
~ the concomitant cover activity also includes interacting with the device
(such
as pushing a button on the wristwatch or writing on the screen of a PDA).
The concomitant cover activity may involve something as simple as resting the
arm on a counter top, while wearing a wristwatch embodiment of the invention.
In addition to the fact that covert versions of the apparatus can be used to
create
investigative documentary videos having very good composition, the device need
not
necessarily be covert. Instead, it may merely be designed so that the subject
of the
picture or video cannot readily determine whether or not the apparatus is in
use
recording. Just as department stores and the like often have dark domes on
their
ceilings, so that customers do not know whether or not there are cameras in
the
domes (or which ones have cameras and even which way the cameras are pointed
if
there are cameras in the domes), the proposed invention creates a situation in
which a
department store clerk or the like will not know whether or not a customer's
personal
memory assistant is recording video. This uncertainty is created by having the
camera
positioned so that it will typically be pointed at a person at all times,
whether or not
it is actually being used.
In a handheld embodiment of the invention, a handheld pen-based computer or
a combination clipboard and pen-based computer input device, hereafter
referred to
as "clipboard" (whether it be an electronic organizer, PDA, or a real pen and
paper
clipboard into which some electronic devices have been added) is used as the
basis
for the secondary activity.
A camera is built into the clipboard with the optical axis of the lens facing
the
direction from bottom to top of the clipboard, so that during normal face-to-
face
conversation, the person holding the clipboard will tend to point the camera
at the
CA 02275798 1999-06-29
other person while taking written notes of the conversation. In this manner,
the
intentionality (whether or not the person taking written notes is intending to
point
the camera at the other person) is masked by the fact that the camera will
always
be pointed at the other person by virtue of its placement in the clipboard.
Thus the
camera lens opening need not necessarily be covert, and in fact, could be
deliberately
accentuated (e.g. made more visible) if desired. To understand why it might be
desirable to make it more visible, one can look to the cameras in department
stores,
which are often placed in large dark smoked plexiglass domes. In this way they
are
neither hidden nor visible, but rather, they serve as an uncertain deterrent
to criminal
conduct. While they could easily be hidden inside smoke detectors, ventilation
slots,
or small openings, the goal of the dome is to make the camera conceptually
visible
yet completely hidden. In a similar manner, a large lens opening on the
clipboard
may, at times, be desirable, so that the subject will be reminded that there
could be a
recording, but will be uncertain as to whether or not such a recording is
actually taking
place. Alternatively, a large dark shiny plexiglass strip, made from darkly
smoked
plexiglass (typically lcm high and 22cm across) is installed across the top of
the
clipboard, as a very subtle yet visible deterrent to criminal behaviour and
disrespect.
One or more miniature cameras are then installed behind the dark plexiglass,
looking
forward through it. In other embodiments, a camera is installed in a PDA, and
then
the top of the PDA is covered with dark smoky plexiglass.
The clipboard is fitted with an electronic display system which includes the
capa-
bility of displaying the image from the camera, so that it can serve as a
viewfinder
for aiming the camera at the subject. Moreover, the display is typically
constructed
so that it is visible only to the user of the clipboard, or, at the very
least, so that the
subject of the picture cannot readily see the display. Such concealment of the
display
may be accomplished through the use of a honeycomb filter placed over the
display.
Such honeycomb filters are common in photography, where they are placed over
lights
to make the light sources behave more directionally. They are also sometimes
placed
6
CA 02275798 1999-06-29
over traffic lights where there is a wye intersection, so that the lights can
only be seen
from one direction, for example, so that the traffic lights do not confuse
drivers on
another branch of a wye intersection that faces almost the same way.
Alternatively,
the display may be designed to provide an inherently narrow field of view, or
other
barriers may be constructed to prevent the subject from seeing the screen.
The function of the camera is integrated with the clipboard so that textual in-
formation, as well as drawings, may be stored in a computer system, together
with
pictures or videoclips (hereafter still pictures and segments of video will
both be re-
ferred to as videoclips, with the understanding that a still picture is just a
video
sequence which is one frame in length).
Since videoclips are stored in the computer together with other information,
these
videoclips may be recalled by an associative memory working together with that
other information. Thus tools like the UNIX "grep" command may be applied to
videoclips by virtue of the associated textual information which typically
resides as a
videographic header. In this way, for example, one could grep for the word
"meijer",
and may find various videoclips taken during conversations with clerks in the
Meijer
department store. Thus such a videographic memory system may give rise to a
memory recall of previous videoclips taken during previous visits to this
particular
department store, provided that one has been diligent enough to write down
(e.g.
enter textually) the name of the department store upon each visit.
Videoclips are typically time-stamped (e.g. there exist file creation dates)
and
GPS-stamped (e.g. there exists global positioning system headers from last
valid
readout) so that one can search on setting (time+place).
Thus the video clipboard may be programmed so that the act of simply taking
notes causes previous related videoclips to play back automatically in a
separate
window (in addition to the viewfinder window which should always remain active
for
continued proper aiming of the camera). Such a video clipboard may, for
example,
assist in a refund explanation by providing the customer with an index into
previous
7
CA 02275798 1999-06-29
visual information to accompany previous notes taken during a purchase. This
system
is especially beneficial when encountering department store representatives
who do
not wear name tags and who refuse to identify themselves by name (as is often
the
case when they know they have done something wrong such as disrespect a
customer).
Thus the device is responsive to an input from the user, and the operation of
the camera in recording or recall is thus under user-control. An alternative
form of
user input for wearable embodiments of the invention (e.g. the wristwatch
camera)
includes physiological input signals. The wristwatch camera, for example,
could be
responsive to heart rate and skin resistance, so that the capture rate
increases when
the heart beats faster and when the wearer sweats more. In this way, the user
input
does not require pressing buttons. During an argument with a clerk or
official, the
user's heart rate increases, and the user sweats profusely, triggering capture
of the
event.
The user input may also be in the form of speech recognition. For example, a
wristwatch embodiment of the invention containing a microphone could recognize
the
phrase "it's getting late" as an instruction to the computer to capture a
picture or
to begin recording video. Thus the wearer could pretend to check the time on
his
wristwatch and at the same time, while sighting the shot by looking at the
display
(functioning as a camera viewfinder) remark, ostensibly to the person he is
talking to
(when in reality the wearer is making the remark to the watch), "oh well, it's
getting
late".
Variations of the same phrase could do different things. For example, the
phrase
"it's getting kind of late" could capture a low resolution picture for rapid
transmission
to the Internet, while the phrase "it's getting awfully late" could capture an
awfully
big file that might take longer to send, and the phrase "it's getting very
late" could
capture a picture at very high resolution.
Other phrases could trigger videoconferencing modes, for example, the phrase
"oh, I've got to go pick up my wife" could open up a videoconferencing channel
to a
8
CA 02275798 1999-06-29
specific person (e.g. the wearer's spouse). These commands could be combined,
so
for example if the wearer wishes to capture a picture and also send it to his
wife, he
could remark: "oh well, it's getting very late! I've got to go pick up my
wife.".
Alternatively, a switch elsewhere on the body can trigger the capture. For ex-
ample, a switch concealed in the frame of a pair of eyeglasses works well when
the
wearer of the watch and glasses is pretending he has trouble focusing on the
watch,
since people often grab onto their glasses and pull them out to the end of
their nose
when they are having trouble focusing close. This could trigger transmission
of video
that would end when the glasses are pushed back up to the top of the nose.
Such an apparatus might also be of use in personal safety. Although there are
a
growing number of video surveillance cameras installed in the environment
allegedly
for "public safety", there have been recent questions as to the true benefit
of such
centralized surveillance infrastructures. Most notably there have been several
exam-
ples in which such centralized infrastructure has been abused by the owners of
it (as
in roundups and detainment of peaceful demonstrators). Moreover, "public
safety"
systems may fail to protect individuals against crimes committed by members of
the
organizations that installed the systems. Accordingly, embodiments of the
invention
often implement the storage and retrieval of images by transmitting and
recording im-
ages at one or more remote locations. In some embodiments of the invention,
images
are transmitted and recorded in different countries, so that they would be
difficult to
destroy, in the event that the perpetrator of a crime or other misconduct in a
high
position of power and influence might wish to destroy evidence of his or her
miscon-
duct. By transmitting images to a wide variety of receive sites, there will
hopefully be
at least one such site out of reach of the perpetrator's sphere of
totalitarian control.
Thus an Internet connected portable recording device, such as one that
transmits
pictures to the World Wide Web, or the like, ofFers the capability of
preventing, or at
least documenting misconduct of officials who might otherwise attempt to
prohibit
photography or videography inside their possibly corrupt establishments.
9
CA 02275798 1999-06-29
The invention allows images to be captured in a natural manner, without giving
an unusual appearance to others (such as an abusive official).
Moreover, as an artistic tool of personal expression, the apparatus allows the
user
to record, from a new perspective, experiences that have been difficult to so
record
in the past. For example, a customer might be able to record an argument with
a
fraudulent business owner from a very close camera angle. This is possible
using,
for example, the clipboard embodiment of the invention, because a clipboard
may be
extended outward toward the person without violating personal space in the
same
way as might be necessary to do the same with a camera hidden in a tie clip,
baseball
cap, or sunglasses. Since a clipboard may extend outward from the body, it may
be placed closer to the subject than the normal eye viewpoint in normal face-
to-
face conversation. As a result, the camera can capture a very close-up view of
the
subject. Such a view is often more desirable than the view obtainable from a
telephoto
lens located in eyeglasses (e.g. telephoto embodiment of the WearCam
invention)
because the close-up view also provides a dramatic and powerful composition,
from
underneath the subject's face, and is therefore most suitable,
cinematographically,
for best capturing the menacing nature of a corrupt or disrespectful official.
SUMMARY OF THE INVENTION
In one aspect this invention provides a method of positioning a camera to take
a
picture or video of a subject, where there is a concomitant cover activity, so
that the
camera is positioned without the subject's knowledge or at least without the
subject's
being certain as to whether or not said picture or video is being taken.
In another aspect this invention provides a means of recording a picture or
video
where the spatial extent (field of view) of the image may be ascertained
without
having to hold any device directly up to the eye.
In another aspect this invention provides such a method of taking a picture or
video
where the tonal characteristics of the picture may be ascertained without
having to
hold any device directly up to the eye.
CA 02275798 1999-06-29
In another aspect this invention provides such a method of taking a picture or
video where no apparent difference in body movement or gesture between when a
picture is being taken and when no picture is being taken is discernible by
others.
In another aspect this invention provides the user with a means of determining
the composition of the picture from a display device that is located such that
only the
user can see the display device, and so that the user can ascertain the
composition
of a picture or take a picture or video and transmit images) to one or more
remote
locations without the knowledge of others in the immediate environment, or
without
others in the environment being certain whether or not such picture or video
is being
captured.
In another aspect this invention provides a means and apparatus for a user to
capture a plurality of images of the same scene or objects, in a natural
process of
simply rotating a small device around on a countertop, and then have these
images
combined together into a single image of increased spatial extent, spatial
resolution,
dynamic range, or tonal fidelity.
In another aspect this invention provides a camera with viewfinder means for
col-
laboration between the user of the apparatus and one or more other persons at
remote
locations through the manipulation of virtual objects such as cursors, or
computer
graphics renderings displayed upon the camera viewfinder.
In another aspect this invention allows multiple users of the invention,
whether
at remote locations or side-by-side, or in the same room within each other's
field
of view, to interact with one another through the collaborative capabilities
of the
apparatus.
In another aspect this invention allows multiple users of the invention, at
remote
locations, to collaborate in such a way that a virtual environment is shared
in which
camera-based camera-tracking of each user results in acquisition of video and
subse-
quent generation of virtual information being made available to the other(s).
In another aspect this invention allows multiple users of the invention, at
the same
11
CA 02275798 1999-06-29
location, to collaborate in such a way that multiple camera viewpoints may be
shared
among the users so that they can advise each other on matters such as
composition,
or so that one or more viewers at remote locations can advise one or more of
the users
on matters such as composition or camera angle.
In another aspect this invention allows multiple users of the invention, at
different
locations, to collaborate on a project or endeavour that may not pertain to
photogra-
phy or videography directly, but a project or endeavour nevertheless that is
enhanced
by the ability for each person to experience the viewpoint of another.
In another aspect this invention allows one or more remote participants at con-
ventional desktop computers or the like to interact with one or more users of
the
invention, at one or more other locations, to collaborate on a project or
endeavour
that may not pertain to photography or videography directly, but a project or
en-
deavour nevertheless that is enhanced by the ability for one or more users of
the
invention to either provide or obtain advice from or to another individual at
a remote
location.
In another aspect this invention to facilitates a very close-up camera angle,
from
slightly below the subject of the picture or video, most notably so that the
camera can
be brought closer to the subject than would be acceptable for the body of the
camera
operator, so that the personal space of the subject can be violated more
effectively
than with a body-worn camera such as that of the WearCam invention.
In another aspect this invention facilitates the creation of a database
comprising
video records linked to associated user input such as hand-written notes, user
heart
rate, user sweatiness, etc..
In another aspect this invention facilitates concomitant cover activity
together
with user input.
In another aspect this invention provides the user with means of sending
pictures
to least one remote location.
In another aspect this invention provides the user with a personal safety
device
12
CA 02275798 1999-06-29
including means of notifying others at at least one remote location of
potential danger.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention will now be described in more detail, by way of examples which
in no way are meant to limit the scope of the invention, but, rather, these
examples
will serve to illustrate the invention with reference to the accompanying
drawings, in
which:
FIG. 1 is a diagram of a simple embodiment of the invention in which there is
a camera borne by a personal digital assistant (PDA) which has a separate
display
attached to it to function as a viewfinder for the camera.
FIG. 2 is a diagram of an alternate embodiment of the camera system in which a
graphics tablet is concealed under a pad of paper and an electronic pen is
concealed
inside an ordinary ink pen so that all of the writing on the paper is captured
and
recorded electronically together with video from the subject.
FIG. 3 illustrates a wristwatch embodiment of the invention in which a camera
concealed inside the watch sends video to a covert body-worn computer system
which
transmits the video to a remote site where it is processed by one or more
remote
experts and advice from the remote experts) is displayed on a computer screen
on the face of the watch together with time, date, and video from the camera
(for
viewfinder purposes).
FIG. 4 depicts a close-up view of the wristwatch display, which happens to be
the display of a body-worn computer running XFree86 on top of the Linux
operating
system, which facilitates the display of an "xclock" together with other
material such
as video from the camera (so the display can function as a viewfinder), and
various
menu items suitable for a wristwatch videotelephone.
FIG. 5 depicts the rightward orientation of the camera for a wristwatch
housing
worn on the left hand, when the hand is extended outward with a long reach to
get
the camera closer to the subject.
13
CA 02275798 1999-06-29
FIG. 6 depicts a side view of the downward orientation of the camera, showing
a
side view of the wearer's wrist, and further details of the apparatus of the
invention.
FIG. ? depicts a strain relief attachable to the inside of a shirt sleeve.
FIG. 8 depicts the block diagram of an embodiment of the invention including
various involuntary inputs (biosensors of the invention for measuring
physiological
quantities) and voluntary inputs (pushbuttons, voice recognizer, etc.) that
affect the
capture of pictures.
FIG. 9a depicts the general appearance of the wristwatch videophone embodiment
of the portable video production facility.
FIG. 9b depicts the nontransparent clockface in detail.
FIG. 9c depicts the transparent clockface in detail.
While the invention shall now be described with reference to the preferred em-
bodiments shown in the drawings, it should be understood that the intention is
not
to limit the invention only to the particular embodiments shown but rather to
cover
all alterations, modifications and equivalent arrangements possible within the
scope
of appended claims.
In all aspects of the present invention, references to "camera" mean any
device or
collection of devices capable of simultaneously determining a quantity of
light arriving
from a plurality of directions and or at a plurality of locations, or
determining some
other attribute of light arriving from a plurality of directions and or at a
plurality
of locations. Similarly references to "display" shall not be limited to just
television
monitors or traditional televisions used for the display of video from a
camera near or
distant, but shall also include computer data display means, computer data
monitors,
other video display devices, still picture display devices, ASCII text display
devices
and the like.
References to "video" shall include any signal that includes one or more
streams
of visible pictures, together with one or more audio channels from one or more
mi-
crophones. Thus the term "video" may, for example, refer to a signal recorded
from
14
CA 02275798 1999-06-29
two cameras and several microphones arranged in an array, to be later
processed for
beamforming purposes or to be processed in such a way as to project a
nullspace
toward unwanted audible interference such as the sound of a vacuum cleaner in
the
appliances section of a Sears department store during the shooting of a
documenatary
video there. The term "video" shall also be taken to include intermediate
representa-
tions of signals derived from cameras, microphones, and associated auxiliary
sensors.
Examples of such intermediate representations shall include, but not be
limited to,
recording of some number M of audio channels from some number N of
microphones,
where M may be less than N. Examples of "video" may also include image
represen-
tations from multiple electromagnetic sensors represented as a single signal.
DETAILED DESCRIPTION OF THE INVENTION WITH
REFERENCE TO DRAWINGS
Fig 1 shows an embodiment of the invention built into a PDA 100. A video
camera
110 captures a view of a person standing in front of the user of the PDA and
displays
the image on an auxiliary screen 120, which may be easily concealed by the
user's
hand while the user is writing or pertending to write on the PDA's screen 130
with
pen 140. In commercial embodiments of the invention the auxiliary screen 120
may
not be necessary, as it may be implemented as a window displaying the camera's
view on a portion of the main screen 130. Annotations made on screen 130 with
pen
140 may also be captured and stored together with videoclips from camera 110,
so
that there is a unified database in which the notes and annotations are linked
with
the video. An optional second camera 150 may be used if the user wishes to
make
a video recording of himself/herself while recording another person with
camera 110.
In this way, both sides of the conversation may be simultaneously recorded by
the
two cameras, so that the resulting recordings could be edited later, so that
there
could, for example, be a cut back and forth between the two cameras to follow
the
natural flow of the conversation. Such a recording might, for example, be used
for an
investigative journalism story on corrupt department store owners who
illegally chain
CA 02275798 1999-06-29
their fire exits shut. In the early prototypes, an additional wire 160 was run
up the
sleeve of the user into a separate body worn pack 170 powered by its own
battery pack
172. The body worn pack typically contains a computer system 174 which houses
video capture hardware and is connected to a communications system 176
comprising
packet radio terminal node controller (high level data link controller with
modem) and
radio, which typically establishes an Internet connection by way of antenna
178. The
body-worn pack 170 would likely disappear in the final embodiment of the
invention
which would contain the functionality of the pack 170 inside the hand-held
device
100.
Fig 2 depicts an alternate embodiment of the invention depicted in Fig 1 in
which
the invention is built into an ordinary clipboard 200. Video camera 210
displays on
a miniature screen 220 mounted to the clipboard. A folded back piece of paper
230
conceals the screen 220. The rest of the sheets of paper 231 are placed
slightly below
the top sheet 230, so that the user can write on them in a natural fashion.
From
the perspective of someone facing the user (the subject), the clipboard will
have the
appearance of a normal clipboard in which the top sheet 230 appears to be part
of
the stack 231. Pen 240 is a combined electronic pen and real pen, so that the
user
can simultaneously write on the paper with real ink, as well as make an
electronic
annotation by virtue of a graphics tablet below the stack of paper 231,
provided that
the stack is not excessively thick. In this way there is a computer database
linking the
real physical paper with its pen strokes and the video recorded of the
subject. From
a legal point of view, real physical pen strokes may have some forensic value
that the
electronic material may not (e.g. if the department store owner asks the
customer to
sign something, or even just to sign for a credit card transaction, the
customer may
place it over the pad 231 and use the special pen 240 to capture the signature
in the
customer's own computer and index it to the video record). Wire 260 going from
the
clipboard, up the sleeve of the user, would be eliminated through construction
of a
self-contained video clipboard.
16
CA 02275798 1999-06-29
Fig 3 depicts a wristwatch embodiment 300 of the invention depicted in Fig 1.
Camera 310 points forward such that, for example, while a customer is wearing
the
wristwatch embodiment of the invention and is standing at a counter, he or she
can
place his or her arm naturally upon the counter and aim the camera 310 at the
of
ficial behind the counter without appearing unusual. The video of the official
seen
by camera 310 is displayed on display unit 320, so that display unit 320
functions as
a viewfinder to assist the customer in aiming the camera 310. An optional
second
camera 350 may be included if the customer wishes to record himself/herself so
that
both sides of the conversation would be recorded. Cabling 360 runs from the
cam-
eras) 310 (and 350) inside the wrist strap, emerging at the back of the wrist
strap
where it will be concealed from view by the official. Cabling 360 runs up the
sleeve of
the wearer of the watch, to an Internet-connected body-worn computer system,
the
output of which runs back down cabling 360 out to the display 320. In this way
video
from camera 310 may be transmitted and recorded at remote sites, while the
wearer
of the wristwatch may be advised by a remote legal expert on the best approach
for
dealing with the corrupt or disrespectful official. In future embodiments, the
cabling
360 may be eliminated, so that the unit would be either entirely self-
contained, or
would communicate wirelessly with a body-worn computer/repeater/internet gate-
way station or a computer/repeater/internet gateway station carried in a
briefcase or
the like.
Interaction with the wristwatch version of the invention depicted in Fig 3 may
be
done through a pen-based or touch-based interface to the screen, in the form
of a
so-called "pie menu" as described in Callahan, Hopkins, Weiser and
Shneiderman,
1988, or by using a "unistroke shorthand" as invented by Goldberg and
Richardson,
1993.
Fig 4 depicts a natural choice of pie menu for a wristwatch display. Display
400
is typically a computer screen with 480 pixels down and 640 pixels across,
measuring
approximately 0.7 inches on the diagonal. Upon display 400 is the image of a
clock
1?
CA 02275798 1999-06-29
face, superimposed on top of a video signal from the camera. Time is displayed
as a
video picture, similar to the way that time is displayed on a UNIX workstation
using
the "xclock" program. Thus the "hands" of the clock 410 are realized by a
video image
in which pixels are darker (or lighter) in the region of the hands, and in
this way the
embodiment of the camera invention tells time. In the figure depicted here,
the time is
4:03. Thus the device truly looks like an ordinary wristwatch (although one in
which
the hands are displayed electronically) because it is in fact a wristwatch,
among other
things. It is natural for such a wristwatch to have a circle 420 displayed on
the screen,
and to have numbers 430 displayed around the periphery of said circle. In this
way
it is easier to tell time, and also the numbers may be assigned a secondary
meaning
(e.g. select "0" to stop recording, "4" to kill all processes and halt the
processor, "7"
to wake up the system from sleep mode, etc.).
Since humans are quite good at telling time, often the numbers are missing
from
many commercial wristwatches, and some wristwatches do not even have markings
for
each hour. Instead, we often rely on our heightened sense of visual acuity to
discern
the angle of the hands upon the clockface. Thus it is no surprise that the
clock menu
is usable without paying much attention to the face of the clock. The user
just needs
to stroke the face of the clock in the direction desired (e.g. stroke the
clockface in the
2:00 direction to enter the number "2").
The entry of numbers on a touch sensitive clockface in the context of the
current
invention may be done as vectors (e.g. with no regard to location, only regard
to
direction). Thus a stroke from left to right is regarded as the number "3"
regardless
of where the stroke begins or ends. A downwards stroke (e.g. from top to
bottom) is
regarded as the number "6" regardless of where the stroke begins or ends, and
so on.
Thus telephone numbers can be easily entered into the device, and similarly an
alphabet 440 can be constructed much like the alphabet of an automated DTMF
answering system used for voicemail and the like in telephony.
Such a menu may be implemented with touchscreen, or alternatively small push-
18
CA 02275798 1999-06-29
buttons around the periphery of the clock face may be used. Since there are 12
pushbuttons on a telephone, and there are also 12 hours on a clock face, there
can be
a one to one correspondence between the numbers of the clockface and those of
the
telephone. The hours 10:00 and 11:00 are used for the symbols "*" and "#" of
the
telephone touchpad.
The data entered by way of the clock face menu is typically combined with the
video recording made from the scene. Of course the amount of information taken
down
may be less than that in the embodiments of the invention described in Fig 1
and
Fig 2, but the clock face menu is sufficient for entering a department store
manager's
name, which may be appended to the video file header, so that later a large
database
of recorded video may be navigated using these short text headers. Of course
GPS
and file creation are also important as in the other embodiments of the
invention.
Moreover, due to direct contact between the watch and the body, the pulse
(heart
rate), as well as skin conductivity (sweatiness index) of the wearer may be
determined
and this information may be appended to or recorded with the video signals.
This
may facilitate, for example, a future search through all video in which the
wearer's
heart rate exceeds a certain threshold. It has been found that when a
department
store manager is dishonest with respect to refund policies, or a clerk refuses
to tell
a customer his~her name, that the customer's heart rate increases
dramatically, and
the customer often sweats profusely. Thus this extra information can later
help locate
moments of tension in a previously recorded argument at the refund counter.
Fig 5 depicts a rightward angling of the camera suitable for use when the
wrist-
watch is worn on the left hand which is extended outward toward the subject.
The
new camera orientation 510 is such that the camera aims to the right of where
it
would have with the original camera orientation 310. The camera may swivel so
that the user can select an orientation, or if the camera is fixed in
manufacture, it
may be desirable that it be angled slightly to the right, e.g. somewhere
between the
orientation of 310 and that of 510.
19
CA 02275798 1999-06-29
Fig 6 depicts a downward angling of the camera suitable for use when the
wearer's
arm 601 is extended outward toward the subject 605. The new camera orientation
610 is such that the camera aims downward slightly, relative to the housing
630. Thus
the wearer 600 can extend the wrist 602 outward a little further and tip the
watch
face display 620 back toward wearer 600, and then the camera 610 will point
straight
ahead.
The wrist 602 is shown in cross section, and is shown greatly enlarged
compared
to the arm 601 of the wearer 600.
The wristwatch has a watchband made in two sections, a first inward portion
640,
with velcro receptor 641 and a second outward portion 650 with velcro hooks
651.
The velcro strap is shown undone in the figure for clarity, while it will be
understood
that the velcro hooks 651 will be pressed against the velcro receptor 641 in
actual
use.
Wrist orientation 660 will be defined as "inward", and wrist orientation 670
will
be defined as "downward", for purposes of this disclosure.
Wires from downward oriented camera 610 and display 620 emerge within insu-
lation harness 680 which passes beneath inward portion 640 of the wrist band.
This
wiring harness 680 emerges from the wristband 640 at a point between the down-
wardmost point 671 and the innermost point 661. Wiring harness 680 is shown as
a
dashed line beyond this point, where it runs inside a shirtsleeve of the
wearer, who
is wearing a long sleeved shirt. Wiring harness 680 is anchored inside the
shirtsleeve
at a point 681 between the wrist and elbow. Wiring harness 680 continues on
inside
the wearer's shirt, to a body worn computer system 690, which is the processor
that
processes images from camera 610. Display 620 is connected to an output of
body
worn computer system 690, so that a camera viewfinder function on display 620
can
be controlled by software. This control also facilitates videoconferencing in
which
display 620 can display video from the wearer of another similar device,
instead of,
or in addition to video from camera 610. Display 620 also shows output from
com-
CA 02275798 1999-06-29
puter system 690, such as ping statistics, netstat, and other functions of
Internet
connectivity, as well as text messages from a person at a remote location.
Computer system 690 typically runs the GNUX (GNUS-Linux) operating system
with XF86 variant of X-windows. The screen 620 is typically 640 by 480 pixels
in true
24 bit color, so that good image fidelity results. The GNUX (GNU+Linux)
command
oclock -transparent -geometry 480x480+80+0 is typically used to provide a nice
clockface that fills almost the entire screen, yet is transparent so that live
video can
be seen in the root window beneath it. In this manner, the entire apparatus
has the
appearance of a wristwatch.
For the purposes of this disclosure, the term "oclock" shall be taken to mean
any
electronically displayed round clock face with electronically displayed hands,
and not
just that which typically runs under UNIX, GNUX, or X-windows. The oclock may
comprise only two electronically displayed hands, a minute hand and an hour
hand, or
it may also include an electronically displayed sweep second hand, to give the
clock
a more lively appearance. An oclock may include an "O"-shaped ring encircling
around the outside of the hands, but it is not necessary that this ring be
present.
The clockface may or may not have numbers, depending on personal taste. It may
also have an electronically displayed "jewel" at the top of the clockface, to
make it
easier for others to see the time, even if shown the display of the watch
upside-down
or sideways (as might be the case when onlookers ask the werer for the time).
The clockface of the oclock will typically be transparent so that only the
hands,
and possibly a jewel or "0"-shaped ring encircling around the outside of the
hands
will block what is underneath. The hands and possibly a jewel or "O"-shaped
ring
encircling around the outside of the hands may also be rendered partially
transpar-
ent, either by selection of an alpha blending parameter, or by other means,
such as
exclusive or of the pixel values in the region defined by these hands and
possibly other
clockface features.
Fig 7 depicts the strain relief for the wristwatch embodiment in which the
sender
21
CA 02275798 1999-06-29
sending pictures from camera to computer and from computer to display comprise
wires. Cabling 360 runs up the forearm and is strain relieved inside the
sleeve of the
forearm with forearm strain relief 710. It then runs through a small loop 720
located
at the elbow joint of the arm. Loop 720 provides enough slack that the arm can
fold
at the elbow without tugging at cabling 360, hence ensuring that cabling 360
will
have a long life. Strain relief 730 is a further strain relief that clips to
the inside of
the shirt at the aftarm, near the biceps. Loop 740 is located at the shoulder
joint to
allow enough slack for the arm to rotate at the shoulder. Optional strain
relief 750 is
a further strain relief that clips inside the shirt at the torso area of the
body.
Not all three strain reliefs 710, 730, and 750 are necessary, but preferably
there is
at least one such strain relief used.
Fig 8 depicts an embodiment of the invention in block diagram form. Processor
800 receives input from camera 810. Processor 800 may also send information
back
to camera 810 by way of camera control input 811, so that shutter speed,
integration
time, aperture, focus, or the like can be controlled by processor 800.
Alternatively,
camera 810 may have a bidirectional link sending packets of video to processor
800
with a high level protocol that involves bidirectional communication to
confirm re-
ceived packets and possibly retransmit when packets are lost. In this way
control
input 811 is not explicitly needed, and may be absorbed into a packet protocol
that
encompasses bidirectional data communications between processor 800 and camera
810.
Display 820 is responsive to an output of processor 800 so that it can show
the
wearer of the wristwatch the time of day, which is kept by a realtime clock of
processor
800. Time of day is typically kept within a clock that has its own battery
backup so
that when processor 800 is turned off and later turned on, the time is still
correct.
Time is typically set once (e.g. using the GNUX date command, followed by the
GNUX /sbin/clock -w command), and then remains correct thereafter. A global
positioning system built into the wristwatch may receive satellite signals so
that the
22
CA 02275798 1999-06-29
clock would automatically be set appropriately for a given time zone, and in
this
case, the time may be set to UTC or GMT at all times, with a correction for
the time
displayed in a given timezone.
In many embodiments, there is at least one microphone built into the watch for
recording the audio portion of the video that is recorded by the camera 810.
Moreover,
the microphone can also serve as a speech recognizes to accept commands from
the
user.
To watch previously recorded video upon display 820 there is typically one or
two earpieces 840 that carry the audio portion of the video displayed on
display 820,
earpieces) 840 being responsive to an output of processor 800.
At least some of the functionality of processor 800 is typically controlled by
vol-
untary inputs 850 which may include pushbutton switches on the wristworn
housing.
Additionally, heart rate, or full EKG waveform information determined from
sensors
in the watchband may also feed into processor 800. Other physiological
quantities of
the wearer, such as blood volume, blood flow related, etc., and a measure of
sweatiness
or skin conductivity may also feed into processor 800. These involuntary
inputs 860
can be used by processor 800 to make inferences about such things as video
capture
frame rate, or whether or not to take one or more still pictures.
Additional outputs 870 may include direct electrical stimulation of the skin
by
way of a very small electric shock or the like, so that the wearer can
unobtrusively
obtain information from processor 800. Such information may include the time
of
day, encoded as various combinations of electric shocks. Vibrotactile devices
may be
used in place of electrodes, so that vibration is used instead of the electric
shocks. A
combination of electric shocks and vibration may also be used to give greater
overall
resolution, allowing a larger number of distinctly perceptible output channels
to be
used.
Fig 9a depicts a wristwatch videophone embodiment of the portable video pro-
duction facility, showing the buttons 900 on the watch, which may be used to
set the
23
CA 02275798 1999-06-29
time, select various modes, open up a secret videoconferencing channel, choose
the
form of encryption, select image compression quality, video capture frame
rate, etc.,
as well as choose which clockface will be displayed.
The clock is represented electronically as an array of time-varying pixels,
with a
clockface 910 that has electronically represented hands upon it. Behind
clockface 910
there is the main window 920 of the screen 620, showing the text of this
patent ap-
plication, which was prepared using the wristwatch-based computer with an
external
keyboard plugged into it.
Fig 9b depicts a closeup of the screen 620 of the wristwatch videophone embodi-
ment of the portable video production facility. A clockface 910 has an
electronically
represented minute hand 911 and hour hand 912, as well as a second hand 913.
The
second hand provides a lively appearance to the oclock, and would convince the
aver-
age person looking at the screen 620 that the apparatus was a wristwatch,
especially
if main window 920 were blacked out by selecting a blackout option from
buttons
900. Pressing one of buttons 900 will then reveal the secret window 920 behind
the
clockface, whereupon the secret text of this patent application appears.
The oclock has displayed a jewel 914 so that a person asking the wearer what
time
it is can easily read the time even if the watch is seen upside down or
sideways from
the extended arm of the wearer. The "0"-shaped border 915 also helps further
to
define the shape of the oclock.
The oclock has a large border 919 that hides part or all of window 920, so
that
the clock is easier to read, and also so that the main window 920 is harder to
read. A
single pressing of one of buttons 900 rapidly conceals window 920 behind
clockshroud
919, which in actual use fills completely the screen 620 rather than just part
of the
screen as illustrated in the drawing. Thus the clockshroud would extend
outward all
the way to the edges of screen 620, so that none of main window 920 would be
visible.
Text cursor 921 would also be hidden by clockshroud 919.
Fig 9c depicts a closeup of the screen 620 in secret mode, after a single
pressing of
24
CA 02275798 1999-06-29
one of the buttons 900 unconceals window 920 by making clockshroud 919
disappear.
In this mode the portion of the screen 929 that was covered by clockshroud 919
can
now be read. Moreover, the jewel is eliminated, because this secret mode is
for the
eyes of the wearer only, and there is no longer a need to tell time when the
clock is
being observed from odd viewpoints. The second hand is also eliminated so it
is not
distracting to the wearer, who can now more easily read the entire screen 620.
A small handheld chortling keyboard such as that sold under the tradename Twid-
dler (TM) manufactured by Handykey Corp, can be used to type with. A wireless
version of a chortling keyboard may also be concealed in a pocket, so that
with one
hand in a pocket, the wearer can type short messages or computer commands
while
looking at the wristwatch display 620.
Text cursor 921 thus functions in a normal fashion once window 920 is
selected.
There may be multiple windows like window 620, and some of these may be text,
some may be graphics, and some may be live video or still pictures. Window 620
may
be used to display live video from the camera, or it may be used to play back
video.
Microphones and one or more earpieces are also part of the apparatus for use
as a
video recorder and player.
The function of the apparatus that allows windows to be quickly concealed by
an
opaque clock, or the like, is called a SECRETER, and the function of the
apparatus
that allows windows to be revealed is called an OPENER.
A SECRETER may perform a variety of functions such as hiding windows under
an opaque barrier, or under a clock, and making the clock become more complex
and
busy. The OPENER may perform a variety of functions such as making the clock
simpler, more transparent, or perhaps causing the clock to disappear
altogether.
The OPENER may cause the clock hands to exclusive or with the background, so
that the wearer can see right through the hands themselves.
Alternatively, some ore all of the information on the screen may be
polarizationally
covert, so that it is visible only while wearing polarized glasses. Material
that is visible
CA 02275798 1999-06-29
to everyone, such as the clockface, is not polarizationally dependent.
Material meant
for the wearer only to see is polarizationally dependent, so that, for
example, right
circular polarized light represents the image for the wearer to see, and left
circular
polarized light is produced in the negative of the image, so that it cancels
the right
circularly polarized light, except for the wearer who wears special polarizing
glasses
that filter out the left circularly polarized negational light, which then no
longer
cancels the right circularly polarized light.
BENEFITS OF THE INVENTION
The vast proliferation of video surveillance throughout various establishments
such
as department stores, public parks, streets, and even changerooms (Newsweek,
July
17, 1995) is propagated with a promise to a better, safer world. However,
represen-
tatives and architects of this surveillance superhighway may continue to be
corrupt.
For example, although cameras reduce and deter shoplifting, there continue to
be sit-
uations where shopkeepers illegally chain shut fire exits, or refuse to be
accountable
in other ways {such as making promises with respect to refund policy and then
failing
to keep these promises later). Extreme situations include the murder of a
customer
suspected of {and falsely accused of) shoplifting by a department store owner
(see,
for example, the Latasha Harlins case) where the department store owner shot
the
customer in the back as she turned to walk out of the store.
Thus one benefit of the invention is to provide personal safety, and to ensure
accountability to those who might otherwise escape accountability. These
benefits are
especially useful in a totalitarian regime (a regime which wishes to know
everything
about others but reveal nothing about itself). A typical department store
(where video
surveillance is used extensively yet photography and video by customers is
prohibited)
is a prime example of where the proposed invention may assist in providing
some
degree of balance with respect to mutual accountability.
Furthermore, the invention is useful as a new communications medium, in the
context of collaborative photography, collaborative videography, and
telepresence.
26
CA 02275798 1999-06-29
One way in which the invention can be useful for telepresence is in the
creation of
video orbits (collections of pictures that exist in approximately the same
orbit of the
projective group of coordinate transformations). A video orbit can be
constructed
using the clipboard embodiment in which a small rubber bump is made on the
bottom
of the clipboard right under the camera's center of projection. In this way,
when the
clipboard is rested upon a surface such as a countertop, it can be panned
around
this fixed point, so that video recorded from the camera can be used to
assemble a
panorama or orbit of greater spatial extent than a single picture. Similarly
with the
wristwatch embodiment, a small rubber bump on the bottom of the wristband
allows
the wearer to place the wrist upon a countertop and rotate the entire arm and
wrist
about a fixed point. Either embodiment is very well suited to shooting a high
quality
panoramic picture or orbit of an official behind a high counter, as is
typically found
at a department store, bank, or other organization.
Moreover, the invention may perform other useful tasks such as functioning as
a
personal safety device and crime deterrent by virtue of its ability to
maintain a video
diary transmitted and recorded at multiple remote locations. As a tool for
photo-
journalists and reporters, the invention has clear advantages over other
competing
technologies.
OTHER EMBODIMENTS
From the foregoing description, it will thus be evident that the present
invention
provides a design for a hand-held or wrist-worn camera with viewfinder means.
As
various changes can be made in the above embodiments and operating methods
with-
out departing from the spirit or scope of the following claims, it is intended
that all
matter contained in the above description or shown in the accompanying
drawings
should be interpreted as illustrative and not in a limiting sense.
Variations or modifications to the design and construction of this invention,
within
the scope of the appended claims, may occur to those skilled in the art upon
reviewing
27
CA 02275798 1999-06-29
the disclosure herein. Such variations or modifications, if within the spirit
of this
invention, are intended to be encompassed within the scope of any claims to
patent
protection issuing upon this invention.
28
