Note: Descriptions are shown in the official language in which they were submitted.
CA 02662318 2014-03-13
IMMERSIVE COLLABORATIVE ENVIRONMENT USING
MOTION CAPTURE, HEAD MOUNTED DISPLAY, AND CAVE
BACKGROUND
2. Field of Invention
[0002] The present invention relates generally to virtual reality and
motion capture, and, more
particularly, to systems and program products which allow persons to interact
with real and
artificial environments using motion capture.
3. Background
[0003] Various techniques and technologies exist which allow users to
interact with or
analyze their environment. For example, motion capture techniques are used in
the fields of
Page 1
CA 02662318 2009-04-09
sports, medicine, and entertainment, especially video gaming and animation. In
sports, for
example, motion capture enables a golfer's swing to be digitally recorded for
analysis. In
medicine, orthopedic rehabilitation can employ motion capture to provide
feedback to the
patient, illustrating correct or incorrect techniques with the patient's
movements during walking,
for example. In animation, motion capture allows for an actor's movements and
even facial
expressions to be digitally recorded in a computer model. Later, animators use
the actor's
recorded motions as the basis for the motions of a computer-generated
character. Likewise,
video games use motion capture to facilitate the animation of life-like
characters within the
games.
[0004] Virtual reality technologies allow a user to interact with a
computer-simulated
environment. Most virtual reality environments rely on computer screens or
stereoscopic
displays and are primarily visual experiences. A popular example of virtual
reality technology is
a flight simulator video game, in which the player pilots a virtual aircraft
in a computer-
simulated environment.
[0005] Telepresence refers to technologies which allow the user to
experience, or be present
at, a remote location. For example, telepresence includes a remote video
camera in which the
user can control the pan, tilt, and zoom, if the display is of sufficient size
and quality to allow the
user to feel present at the remote location.
[0006] None of these technologies alone provide a collaborative immersive
environment for
evaluating a design through interaction, virtual training on a task, and
validating a simulation
with a life video.
SUMMARY OF INVENTION
Page 2
CA 02662318 2009-04-09
[0007] In view of the foregoing, embodiments of the present invention, for
example, provide
a collaborative visualization system, which integrates motion capture and
virtual reality, along
with kinematics and computer-aided design (CAD), for the purpose of, amongst
others,
evaluating a design. Embodiments of the present invention also provide, e.g.,
portable motion
capture systems, which allow one or more persons to interact with real and
artificial
environments, and head mounted displays for evaluating a design, virtual
training on a task, and
validating a simulation with a real-world video, amongst others. Embodiments
of the present
invention further provide, for example, objects to be tracked by a motion
capture system and
incorporated into a virtual reality simulation. Embodiments of the
collaborative visualization
system can include, for example, an immersive environment with one or more
simultaneous
users, with one or more external observers, with real-time interactions and
scaling, and with the
ability to switch between a simulation and a telepresence view. An immersive
environment can,
for example, generate a three-dimensional or stereoscopic image which appears
to surround the
viewer. That is, the viewer is "immersed" in the artificial environment.
[0008] Embodiments of the present invention include, for example, a virtual
reality simulator.
The virtual reality simulator can receive data from CAD designs and display a
simulation
constructed from the data to a user via, for example, a head mounted display,
resulting in full-
scale and stereoscopic images. The images can be, for example, so detailed
that the user can
read the wording on the working knobs and switches within the simulation.
[0009] Embodiments of the present invention further include, for example, a
motion capture
system incorporated into the virtual reality simulator. The motion capture
system, for example,
can track the movements and interactions of a user (who is wearing a motion
capture ensemble
with flexible and adjustable level of detail, from just the head to a full
body suit and gloves)
Page 3
CA 02662318 2009-04-09
within the virtual reality simulation so that the when the user's head rotates
or tilts, the view of
the simulation rendered in the head mounted display changes accordingly. The
motion capture
system, for example, can store the tracked movements and interactions of a
user for later use,
including, for example, training or design evaluation purposes. The motion
capture system also
can track the movements and interactions of multiple users within the virtual
reality simulation
such that the multiple users are represented by avatars in real time within
the simulation. Thus,
multiple users can simulate a coordinated activity within the simulation, such
as performing
routine maintenance on an aircraft, for example. In addition, the motion
capture system, for
example, can track the movements and interactions of objects, including tools
and props used by
a user.
[0010] Embodiments of the present invention also include, for example, an
immersive
observation system where multiple observers can view, in real-time, the
virtual reality simulation
including the interactions of the avatars in a common, immersive environment
so as to evaluate
the CAD design. The immersive observation system can include a CAVE (Cave
Automatic
Virtual Environment), such as, a reconfigurable 8-foot-high by 10-foot-wide by
10-foot-long
room with displays on three walls and the floor, where one or more observers
view a common
environment in an immersive and interactive way, including stereoscopic and
full-scale images.
Advantageously, using the CAVE, the designers of the CAD design can observe
end-users
interacting extensively with the proposed design in the simulator to analyze
and evaluate the
design without having to build expensive prototypes. For example, observers
can view users
within the simulation performing the routine maintenance operations on the
aircraft. In addition,
trainees can observe, for example, trainers performing various tasks by
viewing avatars of the
trainers responsive to recorded motion capture data.
Page 4
CA 02662318 2009-04-09
[0011]
According to the embodiments of the present invention, the virtual reality
simulator
can scale each avatar in real time. That is, a 5' 4" user within the
simulation can be scaled in
real-time to be a 6' 2" avatar within the simulation. The images rendered in
the 5' 4" user's head
mounted display will correspond to the perspective expected by someone 6' 2".
An observer of
the simulation will see a 6' 2" avatar. Scaling may be accomplished by
applying a ratio to each
aspect of the data in the translation from motion capture data to simulation
data. Alternately,
scaling may be accomplished in real time by positioning the avatar's head,
hands, and feet in the
correct location to allow kinematics software to solve for the other joints.
In addition, scaling
may be accomplished in post processing, as opposed to in real time.
[0012] According to embodiments of the present invention, the virtual reality
simulator can
include interactions with the simulated environment. In one such example, the
virtual reality
simulator can include collision detection software to provide feedback within
the simulation. If
a user sticks his or her hand where the simulation indicates that a wall
should be, for example a
virtual collision is detected. The hand motion is set to either stop on the
collision or allowed to
disappear from the view of the user (because it is, after all, behind the
wall), and the panel of the
wall can be set to change color (or some other behavior) to provide feedback
and indicate that a
collision has occurred. In a preferred configuration, the wall turns red;
similarly, if a knee
"collides" with a toolbox in the simulation, the toolbox turns red. In
exemplary configuration,
the collision triggers a sound; the sound can be a directional sound
indicating a direction of the
collision with respect to the user or observer. Various types of sounds can
further provide
information regarding the collision, such as, severity or the objects involved
in the collision. For
example, a user hitting the user's head on part of an aircraft can result in a
different sound than a
Page 5
CA 02662318 2009-04-09
object colliding with the floor. In addition, the simulation can alter its
behavior based on
detected collisions, by opening a door or panel, for example.
[0013] In addition, embodiments of the present invention can include a
portable motion
capture system for capturing tasks in the field. This system includes motion
tracking markers
(perhaps on a suit, body, or other apparel), a plurality of cameras installed
on a tripod or clamped
on a rigid structure so that cameras can track the movements of a user wearing
the motion
capture markers, and a computer to record the images from the camera. The
portable motion
capture system allows for a remote procedure, such as a field maintenance
operation, be
recorded. Because of the incorporated nature of the virtual reality simulator
and the motion
capture system provided by the embodiments of the present invention, the data
from a field
maintenance operation can later be studied in the virtual reality simulator
for interactions with a
new design or for real-time evaluation of, for example, an existing design or
a sequence of
operations on an existing design. Moreover, according to embodiments of the
present invention,
a portable motion capture system can be utilized for real-time design
evaluation in the field, for
presentation of a design, and for training at a remote location. Evaluation of
a design can
include, for example, evaluating a design with respect to an environment, an
analysis of the
ergonomics of the design, a study of tasks associated with the design, and
other considerations as
understood by those skilled in the art.
[0014] Furthermore, embodiments of the present invention include methods of
validating a
simulation with real-world video using immersive technology. For example,
according to an
embodiment of such a method, a spherical camera captures real-world video, or
a real-world still
photograph, at a remote location. Later, the video or photograph is rendered
in a head mounted
display. A motion capture system collects the user's head rotation
information, which is used to
Page 6
CA 02662318 2009-04-09
control the pan, tilt, and zoom of the video. Then the user can switch between
displaying the
real-world video and the simulation as a way of validating the simulation. In
addition, the video
can be displayed on a desktop or CAVE. As an example, using a spherical camera
to capture the
real-world images from the deck of an aircraft carrier can be used to validate
a simulation of that
environment.
[0015] Embodiments of the present invention include, for example, systems
and associated
methods of providing an immersive environment with multiple simultaneous
users, with external
observers, with real-time interactions and scaling, and with the ability to
switch between a
simulation and a telepresence view, as will be understood by those skilled in
the art.
Embodiments of the present invention provide improved approaches to evaluate
designs without
having to build prototypes and to train personnel without the need for
prototypes or on location
travel.
BRIEF DESCRIPTION OF DRAWINGS
[0016] So that the manner in which the features and benefits of the
invention, as well as
others which will become apparent, may be understood in more detail, a more
particular
description of the invention briefly summarized above may be had by reference
to the
embodiments thereof which are illustrated in the appended drawings, which form
a part of this
specification. It is also to be noted, however, that the drawings illustrate
only various
embodiments of the invention and are therefore not to be considered limiting
of the invention's
scope as it may include other effective embodiments as well.
[0017] Figure 1 is a schematic diagram of a system to provide a
collaborative immersive
environment for the evaluation of an engineering design according to an
embodiment of the
present invention;
Page 7
CA 02662318 2009-04-09
[0018] Figure 2 is an environmental view illustrating four users wearing
motion capture
equipment interacting with a virtual reality simulation, according to another
embodiment of the
present invention;
[0019] Figure 3 is an environmental view of a motion capture glove according
to an
embodiment of the present invention;
[0020] Figure 4 is a perspective view of a Hergo Easy Mount Mobile Computer
Cart - Star
Base according to an embodiment of the present invention;
[0021] Figure 5 is a perspective view of a head-mounted display according to
an embodiment
of the present invention;
[0022] Figure 6 is a perspective view of a head tracker to be used in
conjunction with the
CAVE according to an embodiment of the present invention;
[0023] Figure 7 is an environmental view of wand hardware to be used in
conjunction with
the CAVE according to an embodiment of the present invention;
[0024] Figure 8A is a perspective view of a spherical camera according to an
embodiment of
the present invention;
[0025] Figure 8B is a perspective view of motion capture cameras according to
an
embodiment of the present invention;
[0026] Figure 9 is an perspective view of avatars within a virtual reality
simulation according
to an embodiment of the present invention;
[0027] Figure 10 is a schematic block diagram of a collaborative
visualization system
according to an embodiment of the present invention;
Page 8
CA 02662318 2009-04-09
[0028] Figure 11 is a schematic flow diagram of a method to provide a
collaborative
immersive environment for the evaluation of an engineering design according to
an embodiment
of the present invention; and
[0029] Figure 12 is a schematic flow diagram of a method of validating a
simulation with
real-world video using immersive technology according to an embodiment of the
present
invention.
DETAILED DESCRIPTION OF INVENTION
[0030] The present invention will now be described more fully hereinafter
with reference to
the accompanying drawings, which illustrate embodiments of the invention. This
invention may,
however, be embodied in many different forms and should not be construed as
limited to the
illustrated embodiments set forth herein; rather, these embodiments are
provided so that this
disclosure will be thorough and complete, and will fully convey the scope of
the invention to
those skilled in the art. Like numbers refer to like elements throughout.
[0031] As illustrated in Figures 1 and 10, embodiments of the present
invention include an
collaborative visualization system 20 which integrates motion capture and
virtual reality
technologies, along with kinematics and CAD, for the purpose of, amongst
others, evaluating a
design, virtual training on a task, and validating a simulation with a real-
world video. In
addition, embodiments of the present invention include, for example, immersive
observation
environments as well.
[0032] Embodiments of the present invention include, for example, a virtual
reality simulator
58 to create the virtual reality environment. The virtual reality simulator
can receive data from a
CAD program and create a virtual reality simulation of the design projected to
a user via a head
Page 9
CA 02662318 2014-03-13
mounted display 40. as illustrated, for example, in Figures 1. 2, and 5. In
addition, the
simulation may be displayed via a desktop or a CAVE 44. The virtual reality
simulation is three-
dimensional (3D) and allows a user to inspect, evaluate, and interact with the
CAD design. The
simulation environment is labeled immersive because the simulation is 3D and
full-scale, and the
user's view can rotate throughout the simulation so that and the user becomes
immersed in the
simulation.
[0033] Embodiments provide for evaluation of designs for aircraft, space
systems, spacecraft,
ships, and missile systems, which often utilize an extensive evaluation
process traditionally
requiring, for example, expensive mock-ups and prototypes. The extensive
evaluation process
can advantageously include ergonomic analysis and task analysis for operation
and maintenance
tasks, as understood by those skilled in the art.
[0034] According to an exemplary embodiment of the present invention, the
virtual reality
software used includes ENVISION (D5) from DELMIA. As understood by those
skilled in the
art, this software provides a physics-based, 3D environment specifically for
designing, verifying,
and rapid prototyping of concept designs involving structures, mechanical
systems, and humans.
As understood by those skilled in the art, the software enhances system and
subsystem level
models with physics-based motion, virtual reality immersion, and ergonomic
evaluation
capabilities for highly accurate 3D simulation, analysis, and visualization.
In addition, according
to an exemplary embodiment of the present invention, other software components
used to create
TM
the virtual reality environment include: PolyWorks from InnovMetric Software
Inc, a software
TM
tool used to convert point cloud data to polygons; NuGraf from Okino Computer
Graphics, a
TM TM
polygon conversion and reduction tool; Deep Exploration or Deep Server from
Right
TM
Hemisphere, a polygon conversion and reduction tool; and MS Visual Studio from
Microsoft, a
Page 10
CA 02662318 2014-03-13
code development suite. According to an exemplary embodiment of the present
invention, the
TM
hardware supporting this software can includes: four Dell Precision
Workstations 670, 16x
TM
DVD-ROM, 48/32 CDRW, Dual 3.0 Ghz Xeon with 2 MB L2 Cache, 800 FSB 4 GB RAM,
TM
nVidia Quadro FX3400 256 MB, 136 GB HD. As understood by those skilled in the
art, the
virtual reality simulator can include a computer program product, stored in
one or more tangible
computer readable media and readable by a computer so that the computer
program product
operates to perform the various instructions when read by the computer as
described herein.
[0035] As illustrated in Figure 1, 2, and 3, according to an embodiment of
the present
invention, the motion capture system 30 includes, for example, users wearing
bodysuits 32,
gloves 34, and headgear 36 with markers 52 at known locations on the suit,
such as the knee, the
wrist, and the top of the shoulders. The motion capture system can further
include real objects,
or props, also having markers to be represented in the virtual reality
simulator. Cameras 54, as
illustrated in Figure 8B, then digitally record the locations of the markers
as the users move
around and interact in the simulation, capturing a set of data. This motion
capture data can then
made available, in real time, to the virtual reality simulator 58 so that the
users and their
movements are modeled as avatars 56 within the simulation and so that feedback
is provided to
the users. An avatar is an electronic image that represents and is manipulated
by, or driven by a
computer user, as in a computer game or other virtual reality setting,
typically being an
incarnation in human form.
[0036] According to an exemplary embodiment of the present invention, the
motion capture
TM TM
system 30 includes up to twenty-four cameras 54 (e.g., 12 Eagle-i and 12 Hawk-
i from Motion
Analysis Corporation, as shown in Figure 8B), for example, mounted on a truss
38 (e.g., LxWxH
is 20.x15'x10') used to track up to six concurrent users wearing head-mounted
displays 40,
Page 11
CA 02662318 2014-03-13
TM
gloves 34 (e.g.. TALON Gloves from Motion Analysis Corporation), and body
suits 32.
According to an embodiment of the present invention, the motion capture system
30 uses
TM
software from Motion Analysis Corporation called EVaRT V5Ø4 and includes the
following,
TM
TM
plug-ins: Animation Plugins, RT2 Animation Plugins, Calcium 4, Talon Streaming
4, Talon
TM
Viewer 4, and EVaRT5. As understood by those skilled in the art, this software
allows templates
and props to be created and customized for tracking everything from physical
mockups to full
TM
body person. In addition, VRSim's SimI0 module can be used to multicast the
data collected by
TM
EVaRT to be used by the simulation engine software ENVISION D5. As understood
by those
skilled in the art, embodiments can, for example, incorporate any number of
cameras.
[0037]
According to an embodiment of the present invention, the virtual reality
simulator can,
for example, scale each avatar in real time. That is, a 5' 4" user within the
simulation can be
scaled in real-time to be a 6' 2" avatar within the simulation. The images
rendered in the 5' 4"
user's head mounted display will correspond to the perspective expected by
someone 6' 2". An
observer of the simulation will see a 6' 2" avatar, and the avatar's posture
will match that of the
user. Scaling may be accomplished by applying a ratio to each aspect of the
data in the
translation from motion capture data to simulation data. Alternately, scaling
may be
accomplished by positioning the avatar's head, hands, and feet in the correct
location to allow
kinematics software to solve for the other joints. Figure 9 illustrates 4
avatars scaled to different
sizes, all driven from the same user; hence, 'all have the same posture. Note
also that scaling can
be accomplished in post processing, as opposed to in real time. That is, for a
remote training
example, a user, i.e., a trainee, of a first size, e.g, 6' 2", can be
displayed an avatar of the first
size, e.g, 6' 2", responsive to motion capture data from a user, i.e., a
trainer, of a second size
different than the first size, e.g, 5' 4".
Page 12
CA 02662318 2014-03-13
[0038] As illustrated in Figures 1, 2. and 5, according to embodiments of
the present
invention, a user experiences the virtual reality environment visually through
a head-mounted
display 40, and each of the head mounted displays can have a different
perspective of the virtual
reality simulation. The head-mounted display 40 can include a stereo display
helmet worn by
users for immersive visualization of full-scale data. As understood by those
skilled in the art,
TM
one type of head-mounted display 40, the VR1280 from Virtual Research, has the
ability to
display at 1280 x 1204 at 60 Hz in mono or stereo. As understood by those
skilled in the art, a
head mounted displays can include separate left-eye and right-eye displays
with different images
so that a user views an image in the head mounted display stereoscopically.
[0039] In an exemplary embodiment of the present invention, the software
used to display the
TM TM
3D scene can be based on OpenSceneGraph 3D graphics toolkit. MiniViz from
VRSim is a
TM
viewer that allows the user to view a running ENVISION (D5) simulation. The
viewer loads
TM
models in the environment and then references the ENVISION (D5) simulation for
the positions
of the models and tracked viewpoints.
[0040] Embodiments of the present invention provide computer workstations
to support the
head-mounted displays. In an exemplary embodiment of the present invention,
the hardware
TM
used to support four head-mounted displays includes: four Dell Precision
Workstation 670, 16x
TM
DVD-ROM 48/32 CDRW, Dual 3.0 Ghz Xeon with 2 MB L2 Cache, 800 FSB, 4 GB RAM,
TM
nVidia Quadro FX3400 256 MB, 136 GB HD. For convenience and as understood by
those
TM TM
skilled in the art, a Hergo Easy Mount Mobile Computer Cart - Star Base 42, as
illustrated in
Figure 4, can be used to mount each set of equipment, including the computer,
keyboard, mouse,
TM
head-mounted diS-play 40, Talon Gloves 34, and a flat panel monitor from
Hergo, according an
embodiment of the present invention.
Page 13
CA 02662318 2009-04-09
[0041] According to embodiments of the present invention, the virtual
reality simulator can
include, for example, interactions with the simulated environment. In one such
example, the
virtual reality simulator can include collision detection software to provide
feedback within the
simulation. If a user sticks a hand where the simulation indicates that a wall
should be, a virtual
collision is detected. The hand can be stopped at the time of the collision or
be allowed to
disappear from the view of the user and the panel of the wall change color (or
some other
behavior) to provide feedback and indicate that a collision has occurred. In a
preferred
configuration, the wall turns red. Similarly, if a knee "collides" with a bomb
in the simulation,
the bomb turns red. In an exemplary embodiment, an appearance of the object is
altered in
response to the collision. In exemplary configuration, the collision triggers
a sound; the sound
can be a directional sound indicating a direction of the collision with
respect to the user or
observer. Various types of sounds can further provide information regarding
the collision, such
as, severity or the objects involved in the collision. For example, a user
hitting the user's head on
part of an aircraft can result in a different sound than a object colliding
with the floor. In
addition, the simulation can alter its behavior based on detected collisions,
by opening a door or
panel, for example. In addition, this collision detection feature can be used
to facilitate the
grabbing of a virtual object within the simulation, permitting the object to
be moved or otherwise
manipulated in the virtual environment. Collisions can also occur between
avatars and between
simulated objects, such as, between a simulated hand tool and simulated wall.
[0042] Embodiments of the present invention can also include, for example,
an immersive
observation environment as well. The observation environment allows designers
to observe and
interact with the virtual reality simulation including the avatars 56 (see
Figure 9), which can be
driven in real-time by motion capture data from the users. As illustrated in
Figure 1, the
Page 14
CA 02662318 2014-03-13
observation environments can include the CAVE (Cave Automatic Virtual
Environment) 44, a
reconfigurable, e.g., 8-foot-high by 10-foot-wide by 10-foot-long room with
displays on three
walls and the floor, where one or more observers view a common environment in
an immersive
way, including stereoscopic and full-scale images. As understood by those
skilled in the art, the
CAVE 44 from Mechdyne displays a resolution of 1280 x 1024 at 96 Hz.
Therefore, designers
and other observers can view in real-time detailed interaction of the users
within the simulation.
In an exemplary configuration, one wall of the CAVE 44 may be used to display
all individual
viewpoints with the other two walls and the floor of the CAVE 44 being used to
display an
overall view. In this configuration, an observer immersed in the simulation
can view avatar 56
posture and the avatar view point simultaneously. Other available data, such
as temperature,
distance measurements, time, etc. whether visible or not, may be also be
displayed on the one
wall of the CAVE 44, according to an exemplary configuration.
[0043] In an exemplary embodiment of the present invention, the software
used to display the
TM
3D scene is based on OpenSceneGraph, a 3D graphics open source toolkit. Much
like the stand
alone viewer for the head-mounted display, the CAVE 44 can use the MiniViz
software program.
The projections of each screen, however, can be mapped using an Application
Programmer's
TM
Interface (API) called CAVELib from Mechdyne. Also running the CAVE 44 are
other
TM TM TM
programs, which can include: Vega, Vega Prime, and Ensight Gold, according to
an embodiment
of the present invention.
[0044] Embodiments of the present invention can also include head tracker
46 (see Figure 6)
and wand hardware 48 (see Figure 7), to be used in the CAVE 44 to allow the
observers to move
around easily within the simulation. AcCording to an exemplary embodiment of
the present
invention, the InterSense IS-900 uses a combination of inertial and ultrasonic
sensors to
Page 15
CA 02662318 2014-03-13
determine the positions of the head tracker and wand. The software running the
head tracker and
wand can include a program called Trackd 5.5 from VRCO. As understood by those
skilled in
the art, the Trackd application takes information from the head tracker and
wand and makes that
information available to either CAVELib or the ENVISION (D5) simulation. This
tracked
information is applied to correct the projections on the walls and the floor
of the CAVE 44.
[0045] Embodiments of the present invention can also include, for example,
an immersive
environment with the ability to switch between a simulation and a telepresence
view. The
telepresence view is available through the CAVE 44, a head mounted device 40,
or desktop
display. The content for the telepresence view are gathered via a spherical
camera 50 (see Figure
8A) at a remote location, such as, for example, the surface of an aircraft
carrier. According to an
embodiment of the present invention, the spherical camera 50 has a set of six
digital cameras
embedded into one device that captures 75% of a sphere. The dynamic scene can
be displayed in
the head mounted display 40 or the CAVE 44, where the user can be immersed in
real-world
video information to help validate simulations. As understood by those skilled
in the art, the
Ladybug2 is a 4.7 MegaPixel video capture device using six 1024x768 CCDs.
According to an
TM
exemplary embodiment of the present invention, the Ladybug2 from Point Grey
Research comes
with a software development environment Laybug2 SDK and a tool called
LadybugCap that
allows content to be captured; this software produces spherical avi files for
playback, as
TM
understood by those skilled in the art. In an exemplary embodiment, Ladybug3
can be used for
better resolution but a lower frame rate.
[0046] As understood by those skilled in the art, the spherical camera 50
produces data files
in 1GB increments, which may be 2 seconds or 2 minutes. So, 30 seconds of
video capture can
turn into. 15 files at 1GB each. These files require translation into a
viewable format. Other
Page 16
CA 02662318 2014-03-13
solutions use a video editor and paste the first viewable files together to
form a single video file,
in the process reducing the quality to produce a smaller, second level video
file. In contrast,
embodiments of the present invention read the first level video files into
buffers and provide
indexing, such as the first file, last, current, the file before the current
and the file after the
current. This allows the video group with many files to be played as if they
were a single video
file.
[0047] According to an embodiment of the present invention, real-world
objects can be
scanned to create models for the virtual reality simulator. In an exemplary
embodiment of the
TM
present invention, the Creatform Handy Scan EXAscan can be employed. In an
exemplary
TM TM
embodiment of the present invention, a Leica HDS 3000 can be employed. The
Leica HDS 3000
is a laser based device that scans equipment to quickly create high-fidelity
3D models where they
do not yet exist. As understood by those skilled in the art, in practice, the
true resolution of the
scanner is 1/4" of point accuracy for unmodeled data and 1/8" of point
accuracy for data
modeled based on multiple points from a point cloud. According to an exemplary
embodiment
of the present invention, the software used to capture the area, set the
resolution, and register the
point clouds together is Cyclone. Also used is PolyWorks to register the
clouds and produce
polygon geometry to be incorporated into the simulation. According to an
exemplary
TM
embodiment of the present invention, the hardware supporting the device can be
a Dell Precision
TM
M70 Laptop, 1.86GHz, 1GB RAM, and Nvidia Quadro FX Go 1400.
[0048] Embodiments of the present invention can include, for example, a
portable motion
capture system for capturing tasks in the field. According to an embodiment of
the portable
motion capture system, the system can include markers at predetermined
locations, a motion
capture suit, a plurality of cameras installed on a tripod so that cameras can
track the movements
Page 17
CA 02662318 2009-04-09
,
of a user wearing the motion capture suit, and a computer or other storage
medium to record the
images from the camera. According to another embodiment of the portable motion
capture
system for capturing tasks in the field, the system can include markers at
predetermined
locations, a motion capture suit, a plurality of cameras clamped on a rigid
structure so that
cameras can track the movements of a user wearing the motion capture suit, and
a computer or
other storage medium to record the images from the camera and to digitally
record locations of
the markers associated with the bodysuit, responsive to the recorded images
from the plurality of
cameras. In a training application, for example, motions of a remote trainer
at a first location can
be tracked and captured for training of a trainee at a second location, ever
later or in real time. In
addition, motions of one or more users can be tracked and captured for task
analysis, including,
for example, ergonomic and efficiency analysis. In another embodiment, for
example, a portable
motion capture system can be utilized for real-time design evaluation in the
field and for design
presentation or demonstration, including, for example, a new design.
[0049] The embodiments of the present invention include a method of evaluating
an
engineering design, as illustrated in Figure 11. The method includes
simulating a CAD design in
a virtual reality environment (step 80) and driving one or more avatars within
the virtual reality
simulation using motion capture data obtained from a user interacting with the
virtual reality
simulation (step 81). The method continues with displaying the virtual reality
simulation,
including the interactions of the one or more avatars, to multiple observers
in a common,
immersive environment in real-time so as to evaluate the CAD design (step 82)
to thereby verify
that tasks associated with a product built according to the CAD design can be
performed by a
predetermined range of user sizes.
Page 18
CA 02662318 2009-04-09
[0050] The embodiments of the present invention include a method of
validating a simulation
with real-world video using immersive technology, as illustrated in Figure 12.
The method
includes capturing real-world video by a spherical camera at a remote location
(step 90) and
rendering the video in a head mounted display (step 91). The method continues
with capturing
the head rotation information of a user by a motion capture system (step 92)
and controlling the
pan, tilt, and zoom of the video by the head rotation information of the user
(step 93). The
method also includes switching, under user control, between displaying the
real-world video and
the simulation (step 94).
[0051] The embodiments of the present invention also include a computer
program product,
stored on a tangible computer memory media, operable on a computer, the
computer program
product comprising a set of instructions that, when executed by the computer,
cause the
computer to perform various operations. The operations include, for example,
receiving CAD
data, generating video signals to simulate in virtual reality the design from
the CAD data,
providing for the tracking of multiple users interacting with each other and
the simulation,
providing for the tracking of objects interacting with the simulation,
generating scaled avatars
within the simulation, generating video signals for the common immersive
environment, and
receiving user input to select between video, graphics, or both together.
[0052] Embodiments can also include a computer program product, being stored
in one or
more tangible computer readable media and readable by a computer so that the
computer
program product operates to perform instructions described herein when read by
the computer.
The instructions include recording at a first location full-body motion
capture data for one or
more trainers performing one or more tasks by a portable motion capture
system. The
instructions include animating one or more avatars within a virtual reality
simulation by a virtual
Page 19
CA 02662318 2009-04-09
reality simulator at a second location, responsive to recorded motion capture
data for the one or
more trainers at the first location so that each of the one or more trainers
corresponds to one of
the one or more avatars. The instructions include displaying the virtual
reality simulation,
including the one or more animated avatars, as a three-dimensional image that
appears to
surround one or more trainees to thereby define a common immersive environment
using one or
more head mounted displays so that the one or more trainees can analyze the
one or more tasks
performed. The instructions can also include obtaining motion capture data for
one or more
trainees interacting with the virtual reality simulation through a motion
capture system;
animating one or more avatars within a virtual reality simulation by a virtual
reality simulator in
real time, responsive to motion capture data for the one or more trainees at
the second location;
detecting a collision between an avatar animated by a trainee and a simulated
object in the virtual
reality simulation by the virtual reality simulator; and altering a color of
the simulated object in
the virtual reality simulation by the virtual reality simulator to provide
feedback for the detected
collision.
[0053]
The embodiments of the present invention can also include a system for
training at a
remote location, for example, tasks associated with operation or maintenance
of an aircraft.
Likewise, tasks can be associated with the operation or maintenance of a
design for an aircraft, a
space system, a spacecraft, a ship, or a missile system.
[0054] Embodiments of the present invention further include a method of
simulating a task.
The method includes recording full-body motion capture data for one or more
users performing
one or more tasks by a portable motion capture system. The method includes
animating one or
more avatars within a virtual reality simulation by a virtual reality
simulator responsive to
motion capture data for the one or more users so that each of the one or more
users corresponds
Page 20
CA 02662318 2009-04-09
to one of the one or more avatars. The method includes displaying the virtual
reality simulation,
including the one or more animated avatars, as a three-dimensional image using
one or more
head mounted displays so that each of the one or more head mounted displays
can provide a
different perspective of the virtual reality simulation.
[0055] The system includes a portable motion capture system 30, 42 at a
first location
positioned to track the movements of one or more users, e.g., trainers, and to
record full-body
motion capture data for one or more users, e.g., trainers, performing one or
more tasks. The
system can include a virtual reality simulator 58 being positioned to receive
the recorded motion
capture data from a first location and capable of animating one or more
avatars 56 within a three-
dimensional virtual reality simulation at a second different location,
responsive to recorded
motion capture data. The system can include an immersive observation system to
display the
virtual reality simulation, including the one or more animated avatars 56, as
a three-dimensional
image that appears to surround one or more trainees to thereby define a common
immersive
environment 20 using one or more head mounted displays 40 so that each of the
one or more
head mounted displays 40 can have a different perspective of the virtual
reality simulation and so
that the one or more trainees can analyze the one or more tasks performed.
[0056] It is important to note that while embodiments of the present
invention have been
described in the context of a fully functional system, those skilled in the
art will appreciate that
the mechanism of at least portions of the present invention and/or aspects
thereof are capable of
being distributed in the form of a computer readable medium of instructions in
a variety of forms
for execution on a processor, processors, or the like, and that the present
invention applies
equally regardless of the particular type of signal bearing media used to
actually carry out the
distribution. Examples of computer readable media include but are not limited
to: nonvolatile,
Page 21
CA 02662318 2014-03-13
hard-coded type media such as read only memories (ROMs). CD-ROMs, and DVD-
ROIVIs, or
erasable, electrically programmable read only memories (EEPRON1s), recordable
type media
such as floppy disks, hard disk drives, CD-R/RWs, DVD-RAMs. DVD-R/RWs,
DVD+R/RWs,
flash drives, and other newer types of memories, and transmission type media
such as digital and
analog communication links. For example, such media can include both operating
instructions
and operations instructions related to the design and evaluation program
product and the method
steps, described above.
[0057]
In the drawings and specification, there have been disclosed a typical
preferred
embodiment of the invention, and although specific terms are employed, the
terms are used in a
descriptive sense only and not for purposes of limitation. The invention has
been described in
considerable detail with specific reference to these illustrated embodiments.
It will be apparent,
however, that various modifications and changes can be made within the scope
of the
invention as described in the foregoing specification and as defined in the
attached claims.
Page 22
