Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
WO 2022/045898
PCT/NZ2021/050044
MOTION CAPTURE CALIBRATION USING DRONES
Cross-reference to Related Applications
111 This application claims priority from U.S. Provisional
Patent Application No.
63/072,085, entitled "MOTION CAPTURE CALIBRATION USING DRONES WITH
MULTIPLE CAMERAS," filed August 28, 2020; U.S. Provisional Patent Application
No. 63/072,088, entitled "MOTION CAPTURE CALIBRATION USING DRONES,"
filed August 28, 2020; and U.S. Provisional Patent Application No. 63/072,092,
entitled
"MOTION CAPTURE CALIBRATION USING FIXED CAMERAS AND DRONES,"
filed August 28, 2020, which are hereby incorporated by reference as if set
forth in full in
this application for all purposes. This application is related to U.S. Utility
Patent
Application No. 17/120,020, entitled "MOTION CAPTURE CALIBRATION USING
DRONES WITH MULTIPLE CAMERAS," filed December 11, 2020; U.S. Utility Patent
Application No. 17/120,024, entitled "MOTION CAPTURE CALIBRATION USING
DRONE", filed December 11, 2020; and U.S. Utility Patent Application No.
17/120,031,
entitled "MOTION CAPTURE CALIBRATION USING CAMERAS AND DRONES,"
filed December 11, 2020, which are hereby incorporated by reference as if set
forth in
full in this application for all purposes.
Background
121 Many visual productions (e.g., movies, videos, clips, and
recorded visual media)
include combinations of real and digital images to create animation and
special effects
that form an illusion of being integrated with live action. For example, a
visual
production may include a live actor in a location shoot appearing in a scene
with a
computer-generated ("CG," "virtual," or "digital") character. It is desirable
to produce
seemingly realistic visual productions by compositing CG items with the live
action
1
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
items. Often, several types of cameras are used on a set, where each camera
provides
different data, such as images of the live action scene, depth information,
tracking of
markers in a live action scene, etc. It is necessary to calibrate the various
camera data in
real time to accurately composite the live action elements with CG images and
produce a
realistic looking visual production.
131 It is an object of at least preferred embodiments to
address at least some of the
aforementioned requirements. An additional or alternative object is to at
least provide the
public with a useful choice.
Summary
[4] Embodiments generally relate to the calibration of cameras
in a live action
scene using drones. Embodiments provide for automated calibration of cameras
in a live
action scene using drones and reference points in images captured by the
cameras
associated with the drones. In various embodiments, a method configures a
plurality of
reference cameras to observe at least one portion of the live action scene.
The method
further configures one or more moving cameras having unconstrained motion to
observe
one or more moving objects in the live action scene and to observe at least
three known
reference points associated with the plurality of reference cameras. The
method further
receives reference point data in association with the one or more moving
cameras, where
the reference point data is based on the at least three known reference
points. The
method further computes a location and an orientation of each moving camera of
the one
or more moving cameras based on one or more of the reference point data and
one or
more locations of one or more reference cameras of the plurality of reference
cameras.
151 In an embodiment, each moving camera is mounted on a mobile
apparatus.
161 In an embodiment, at least one moving object of the one or
more moving
objects is a person.
2
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
171 The term 'comprising' as used in this specification means
'consisting at least in
part of'. When interpreting each statement in this specification that includes
the term
'comprising', features other than that or those prefaced by the term may also
be present.
Related terms such as 'comprise' and 'comprises' are to be interpreted in the
same
manner.
[8] In this specification where reference has been made to
patent specifications,
other external documents, or other sources of information, this is generally
for the
purpose of providing a context for discussing the features of the invention.
Unless
specifically stated otherwise, reference to such external documents or such
sources of
information is not to be construed as an admission that such documents or such
sources
of information, in any jurisdiction, are prior art or form part of the common
general
knowledge in the art.
Brief Description of the Drawings
191 FIG. 1 is a top-view block diagram of an example
environment for calibrating
cameras in a live action scene, which may be used for embodiments described
herein.
[10] FIG. 2 is a side-view block diagram of an example environment for
calibrating
cameras in a live action scene, which may be used for embodiments described
herein.
[11] FIG. 3 is an example flow diagram for calibrating cameras in a live
action scene
using drones, according to some embodiments.
[12] FIG. 4 is a block diagram of an example environment for calibrating
cameras in
a live action scene, according to some embodiments.
[13] FIG. 5 is a block diagram of an example scenario including a reference
point
captured by cameras in a live action scene, according to some embodiments.
3
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[14] FIG. 6 is a block diagram of a group of reference points in a live
action scene,
where the reference points are arranged in a straight line, according to some
embodiments.
[15] FIG. 7 is a block diagram of an example scenario including reference
points in
images captured by cameras in a live action scene, according to some
embodiments.
[16] FIG. 8 is an example flow diagram for calibrating cameras in a live
action scene
using drones, according to some implementations.
[17] FIG. 9 is a block diagram of example cameras coupled to a drone,
according to
some embodiments.
[18] FIG. 10 is an example flow diagram for calibrating cameras in a live
action
scene using drones, according to some implementations.
[19] FIG. 11 is a block diagram of an example computer system, which may be
used
for embodiments described herein.
[20] FIG. 12 is a block diagram of an example visual content generation
system,
which may be used to generate imagery in the form of still images and/or video
sequences of images, according to some embodiments.
[21] FIG. 13 is a block diagram of an example computer system, which may be
used
for embodiments described herein.
Detailed Description of Embodiments
[22] Embodiments facilitate the calibration of cameras in a live action
scene using
drones. In some embodiments, an automated system calibrates cameras in a live
action
scene using reference points in images captured by cameras associated with the
drones.
This calibration may be referred to as motion capture (MoCap) calibration.
4
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
Embodiments described herein enable the system to provide a calibrated
multiview vision
system for tracking reference points, which include drones and may include
active and/or
passive reference markers.
[23] In various embodiments, an apparatus such as a drone includes at least
two
cameras, where one camera is configured to follow the action in the live
action scene,
which includes following the actors. The other camera is configured to view
reference
markers in the live action scene. As described in more detail herein, in
various
embodiments, a system configures multiple reference cameras to observe at
least one
portion of the live action scene. The system further configures at least a
first camera
coupled to a drone to observe one or more moving objects in the live action
scene. The
system further configures at least a second camera coupled to the drone to
observe at
least three known reference points located in the live action scene. The
system further
receives reference point data in association with at least the second camera,
where the
reference point data is based on the three known reference points. The system
further
computes a location and an orientation of the first camera and the second
camera based
on the reference point data.
[24] In various embodiments, a system uses reference point data in
association with
moving cameras on drones and locations of one or more reference cameras in
order to
compute the locations of the moving cameras on the drones. As described in
more detail
herein, in various embodiments, a system configures multiple reference cameras
to
observe at least one portion of the live action scene. The system further
configures one
or more moving cameras having unconstrained motion to observe one or more
moving
objects in the live action scene and to observe at least three known reference
points
associated with the reference cameras. The system further receives reference
point data
in association with the one or more moving cameras, where the reference point
data is
based on at least the three known reference points. The system further
computes a
location and an orientation of each moving camera of the one or more moving
cameras
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
based on one or more of the reference point data and one or more locations of
one or
more reference cameras.
[25] In various embodiments, a system uses reference point data in
association with
at least three known reference points and the one or more reference points
associated with
the one or more moving cameras in order to compute the locations of the moving
cameras
on the drones. As described in more detail herein, in various embodiments, a
system
configures multiple reference cameras to observe at least three known
reference points
located in the live action scene and to observe one or more reference points
associated
with one or more moving cameras having unconstrained motion. The system
further
configures the one or more moving cameras to observe one or more moving
objects in the
live action scene. The system further receives reference point data in
association with
one or more reference cameras, where the reference point data is based on at
least the
three known reference points and the one or more reference points associated
with the
one or more moving cameras. The system further computes a location and an
orientation
of each moving camera based on one or more of the reference point data and one
or more
locations of one or more reference cameras.
[26] FIG. 1 is a top-view block diagram of an example environment 100 for
calibrating cameras in a live action scene, which may be used for embodiments
described
herein. Shown are a system 102, a network 104, and cameras 112, 114, 116, and
118. In
various embodiments, cameras 112, 114, 116, and 118 capture video or images of
objects
such as person 130 in their fields of view (indicated by dotted lines) of
environment 100.
[27] In various embodiments, cameras 112, 114, 116, and 118 are in known
locations
and/or positions. In various embodiments, the location and/or position of the
given
camera is ascertained and/or predetermined. In various embodiments, cameras
112, 114,
116, and 118 may also be referred to as reference cameras 112, 114, 116, and
118.
[28] As described in more detail herein, in various embodiments, one or
more
6
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
reference points are attached to at least some of the cameras in environment
100. For
example, reference points 132, 134, 136, and 138 are attached to respective
reference
cameras 112, 114, 116, and 118. Reference points 132- 138 enable reference
cameras
112 - 118 to identify and locate each other via their respective attached
reference points
132 - 138. In various embodiments, as the positions and/or locations of
reference
cameras 112 - 118 are ascertained and/or predetermined, reference points 132 -
138 being
attached to respective reference cameras 112 - 118 may also be referred to as
known or
predetermined reference points.
[29] In various embodiments, mobile cameras 122 and 124 may also identify
and
locate cameras 112 - 118 via attached reference points 132 - 138 when in the
respective
fields of view of cameras 122 and 124. In various embodiments, as the system
ascertains
the positions and/or locations of one or more given mobile cameras (e.g.,
mobile cameras
122, 124, etc.), such mobile cameras may also be referred to as reference
cameras. Also,
any reference points attached to such reference cameras may be also referred
to as known
or predetermined reference points. In other words, there may be reference
points on
mobile cameras. This enables a given mobile camera to determine its location
and/or
position based on known reference points or a series or chain of known
reference points
attached to stationary reference cameras, a moving system of reference cameras
(e.g.,
cameras on a vehicle, on a train car on a track, etc.), and/or independently
moving
cameras (e.g., mobile cameras 122, 124, etc.).
[30] In various embodiments, the system computes the location of each
moving
camera based at least in part on one or more of global positioning system
(GPS) data,
position sensor data, and inertial guide sensor data, or any combination
thereof. The
system may use these techniques to increase the precision of computing
locations and
positions of cameras in combination with other embodiments described herein.
In
various embodiments, each camera in environment 100 may also use GPS
techniques to
supplement the system in determining the location and/or position of each
camera. In
7
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
some embodiments, the system computes the location of one or more mobile or
moving
cameras on a mobile apparatus such as a drone based at least in part on one or
more
global positing system (GPS) techniques. In various embodiments, a known
location and
position/orientation of a given object may be based on a predefined global
coordinate
system, and known relative to the given object's location and position
relative to another
object. In various embodiments, each camera in environment 100 may also use a
combination of positioning data from position sensors or encoders (e.g.,
motion encoders,
rotation encoders, etc.) to supplement the system in determining the location
and/or
position of each camera. In another example, in various embodiments, each
camera in
environment 100 may also use a combination of inertial guide sensors (e.g.,
altimeters,
proximity sensors, acceleration, etc.) to supplement the system in determining
the
location and/or position of each camera.
[31] As indicated above, in various embodiments, the reference cameras may
be
mounted on one or more rigid structures. For example, in some embodiments,
cameras
112 - 118 may each be attached to tripods standing on the ground (shown in
FIG. 2). In
some embodiments, cameras 112 - 118 may each be attached to tripods standing
on one
or more stationary platforms or levels. Such platforms or levels may be
different from
each other, yet stationary relative to the ground.
[32] In various embodiments, cameras 112 - 118 may be fixed relative to
each other.
For example, in some embodiments, cameras 112 - 118 may all be attached to the
same
single rigid frame such as a cross-braced frame, truss, etc. As such cameras
112 - 118
remain fixed relative to each other. If the single rigid frame is on the
ground or on a
stationary platform, the group of reference cameras may remain stationary.
[33] In various embodiments, if the single rigid frame is moving (e.g., not
on the
ground or on a stationary platform), the reference cameras 112 - 118 may be
attached to
the rigid frame move together. For example, in some scenarios, cameras 112 -
118 may
be on a frame that is floating on water (e.g., ocean, lake,), where the
locations and/or
8
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
positions of cameras 112 - 118 remain known relative to each other even if the
frame has
movement from floating on the water. In some scenarios, cameras 112 - 118 may
be on a
frame that is being carried in the air (e.g., by a drone, group of drones,
etc.). Similarly,
the locations and/or positions of cameras 112 - 118 remain known relative to
each other
even if the frame has movement from hovering in the air. As such, the
reference points
attached to cameras 112 - 118 are known reference points.
[34] Also shown are cameras 122 and 124. In various embodiments, cameras
122
and 124 are mobile. As such, cameras 122 and 124 may also be referred to as
mobile
cameras or moving cameras. As such, the terms mobile cameras and moving
cameras
may be used interchangeably. In various embodiments, cameras 122 and 124 may
be
attached to respective drones 126 and 128. In various embodiments, cameras 122
and
124 may be mounted to remote controlled heads and/or gimbaled to facilitate in
following the action of a scene. For example, cameras 122 and 124 may zoom,
pan, tilt,
etc. to follow a hero actor in environment 100.
[35] As indicated above, in various embodiments, one or more reference
points are
attached to at least some of the cameras in environment 100. For example,
reference
points 132, 134, 136, and 138 are attached to respective reference cameras
112, 114, 116,
and 118. Reference points 132 - 138 enable reference cameras 112 - 118 to
identify and
locate each other via their respective attached reference points 132 - 138.
Mobile
cameras 122 and 124 may also identify and locate cameras 112 - 118 via
attached
reference points 132 - 138 when in the respective fields of view of cameras
122 and 124.
[36] For the purposes of computing the location of a given camera such as
camera
112, system 102 may render the location of the given camera the same as the
location
reference point such as reference point 132 that is attached to camera 112.
This common
location may apply to any camera-reference point pairing or association.
[37] As described in more detail below, system 102 receives videos
including images
9
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
from multiple cameras such as cameras 112 - 118. In various embodiments,
system 102
utilizes cameras 112 - 118 to capture images of known reference points in the
live action
scene or set for image. Cameras 112 - 118 provide reference point data for
system 102 to
compute the locations of cameras 112 - 118. In various example embodiments,
reference
points may be also referred to as reference markers. Embodiments described
herein
calibrate cameras 112 - 118, which improve the accuracy of system 102 locating
and
tracking reference points.
[38] In various embodiments, reference point 140 is attached to person 130.
Reference point 140 enables any of reference cameras 112 - 118 and mobile
cameras 122
and 124 to identify and locate person 130 via attached reference point 140
when in the
respective fields of view of cameras 112 - 124. As described in more detail
herein, in
various embodiments, some references points such as reference point 140 may be
continuously in the fields of view of cameras 122 and 124, which are
configured to
follow one or more reference points in the scene action, which may involve
person 130.
[39] In various embodiments, reference points 142 and 144 are attached to
respective
mobile cameras 122 and 124. Also, reference points 146 and 148 are attached to
respective drones 126 and 128. Reference points 122 and 124 enable mobile
cameras
122 and 124 to identify and locate each other based on their respective
attached reference
points 142 and 144 when in the respective fields of view of cameras 122 and
124.
[40] Similarly, reference point 146 enables mobile camera 124 to identify
and locate
drone 126 based on reference point 146 when in the respective field of view of
camera
124. Also, reference point 148 enables mobile camera 122 to identify and
locate drone
128 based on reference point 148 when in the respective field of view of
camera 122.
[41] In various embodiments, cameras 112- 124 may be hidden or camouflaged
such
that these and other cameras do not capture images that visibly show these
cameras. As
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
such, system 102 locates and calibrates these cameras based on the reference
points
attached to them.
[42] As indicated herein, in various embodiments, cameras 112 - 118 may be
stationary or fixed, depending on the particular implementation. Cameras 112 -
118 are
also orientated in different directions and have broad overlapping fields of
view to
capture video or images of much of environment 100. Cameras 112 - 118 capture
various
reference points in their fields of view, such as those reference points
described in
connection with FIG. 1. The particular distance between cameras 112 - 118 and
their
overall coverage of the set may vary, and will depend on the particular
implementation.
[43] FIG. 2 is a side-view block diagram of example environment 100 of FIG.
1 for
calibrating cameras in a live action scene, which may be used for embodiments
described
herein. In various embodiments, environment 100 may have multiple levels or
layers of
cameras for capturing different aspects of environment 100. For example, in
various
embodiments, reference cameras 112 - 118 may operate on a first level or
layer. In this
context, two or more cameras operating at the same level or layer may mean
operating at
the same height (e.g., 4 feet above ground, 5 feet above ground, etc.) or
operating in the
same height range (e.g., between 1 foot above ground to 8 feet above ground,
etc.). In
various embodiments, the positions and/or locations of reference cameras 112-
118 are
known relative to each other whether the reference cameras remain stationary
or move
together as a unit. The particular levels, layers, and/or ranges may vary,
depending on
the particular implementation.
[44] In various embodiments, mobile cameras 122 and 124 being mobile may
each
operate in their own separate levels or layers and/or share levels or layers
throughout
environment 700, depending on the particular scene and action in the scene
that either
cameras 122 and 124 are capturing. For example, in various embodiments, mobile
cameras 122 and 124 may operate at the same substantial layer or level with
each other,
and either may independently move to another layer or level. In various
embodiments,
11
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
any one or more of mobile cameras 122 and 124 may operate at the same
substantial
layer as other cameras such as cameras 112 - 118.
[45] Referring to both FIGS. 1 and 2, in various embodiments, cameras 112 -
118
may capture various combinations of reference points for calibration purposes.
In
various embodiments, reference points used for calibration may be implemented
in
accordance with embodiments described herein in association with a group 600
of
reference points 602, 604, and 606 of FIG. 6.
[46] Cameras 112- 118 may also capture any combination of references points
132 -
138 associated with respective cameras 112 - 118, which may also include other
known
reference points in environment 100. For ease of illustration, a set of 4
reference cameras
112 - 118 are shown, there may be any number cameras present in environment
100 with
respective reference points attached. As such, any one or more cameras in
environment
100 may compute its own location in environment 100 based on a captured set of
known
reference points in environment 100. System 102 may then calibrate the one or
more
cameras based on their respective reference points captured in one or more
images. Once
calibrated, each camera accurately locates the positions of reference points
in their fields
of view.
[47] As indicated herein, in various embodiments, the field view of a given
reference
camera is generally wide. The fields of view may also be adjustable and
configured with
a wider or narrower field of view. The particular field of view may vary, and
will depend
on the particular implementation. While the field of view of a given mobile
camera is
generally narrower than that of a reference camera, the field of view of a
given mobile
camera may also be adjustable and configured with a wider or narrower field of
view,
depending on the particular implementation. As indicated herein, mobile
cameras are
configured to follow action in environment 100, which may involve following
one or
more actors (hero actors, etc.). Also, any given cameras such as cameras 122
and 124
may split off to follow different moving objects (e.g., actors, hero actors,
vehicles,
12
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
animals, etc.).
[48] In various embodiments, two or more mobile cameras such as cameras 122
and
124 may follow a given moving objects such as hero actors. While all reference
and
mobile cameras are capable of zooming and panning, cameras 112 and 118
generally
maintain constant configurations over multiple scenes, while cameras 122 and
124
generally often change configurations, including zooming, panning, etc., in
order to
closely follow and capture details, reference points, etc. associated with
target moving
objects such as hero actors.
[49] In various embodiments, having more cameras capturing more reference
points
optimizes the computing of the location and position or orientation of cameras
112 - 118,
as more data is available to system 102. In various embodiments, system 102
computes
the locations and positions or orientation of cameras 112 - 118 based on their
respective
reference points 132 - 138. In various embodiments the position of a given
object (e.g.,
camera, reference marker, etc.) may includes its orientation relative to other
objects in the
environment.
[50] As described in other example embodiments described herein, each
camera of
cameras 112 - 118 captures at least one image of a set of known reference
points. As
indicated above and described in more detail below, wand 600 of FIG. 6 may be
used to
implement such a set of reference points. For example, before calibration, a
person may
enter the live action set and place the set of references points in a location
that is in the
field of view of cameras 112 - 118. In various embodiments, the set of
reference points
remain in a predetermined or known position relative to the reference cameras
throughout
the calibration. Cameras 112 - 118 then each capture video or one or more
images of the
references points. System 102 then performs the calibration of cameras 112 -
118 by
computing an aspect ratio between each pair of the reference points, and
computes the
location and orientation of cameras 112 - 118 based on the aspect ratios. The
computed
positions include the absolute location coordinates of cameras 112 - 118 in
the physical
13
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
space of the live action scene or set. System 102 computes the correct
location in space,
the correct scale, and the correct alignment.
[51] In various embodiments, cameras 112 - 118 are positioned at four
corners or
positions in environment 100. In some embodiments, the coordinates of a given
camera
may be associated with and calibrated to be at the optical center of the lens
of the given
camera. The actual part of the given camera associated with a coordinate may
vary, and
will depend on the particular implementation. Cameras 122 and 124 being mobile
may
be located at or may relocated to any particular location in environment 100.
Also, the
particular coordinate system (e.g., Cartesian, polar, etc.) that system 102
uses in
computations may vary, and will depend on the particular implementation.
[52] In some embodiments, system 102 may calibrate cameras in a particular
order.
For example, system 102 may first calibrate two cameras such as cameras 112
and 114
having good angles and overlap in their fields of view. System 102 may compute
the
relative locations and on of the cameras from one to the other.
System 102 may
then calibrate other cameras such as cameras 116 and 118 in turn. In some
embodiments,
system 102 may start with any given pair and continue calibrating cameras pair-
by-pair.
This technique is beneficial in that any one or more cameras can be added to
the overall
group of cameras on the live action set. Such added cameras may be
subsequently
calibrated based on the calibration of existing cameras.
[53] Embodiments described herein provide various benefits. For example, if
cameras need to be recalibrated often, system 102 can quickly calibrate any
already
calibrated camera or newly added or moved camera to be calibrated based on
existing
calibrated cameras. This saves valuable set up time for filming on the live
action film set
or stage.
[54] In various embodiments, in addition to system 102 calibrating cameras
112 -
118 based on a particular set of reference points, system 102 may also
calibrate cameras
14
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
112 - 118 based on other known reference points such as reference points 132 -
138 that
are attached to cameras 112 - 118. For example, if system 102 has computed
relative
locations of reference points 136 and 138, and one or more other known
reference points
in environment 100, system 102 may calibrate cameras 112 and 114 based on
those
reference points captured by cameras 112 and 114 using associated aspect
ratios.
[55] In some embodiments system 102 may also utilize one or more inertial
measurement unit (IMU) sensors in each camera to estimate a location and
orientation of
each camera to supplement the calibration information. IMU sensors may include
magnetometers, accelerometers, etc. The associated IMU measurements in
combination
with associated aspect ratio measurements helps system 102 to compute accurate
orientation of cameras 112 - 118.
[56] These additional techniques are beneficial in optimizing the
calibration of
cameras 112 - 118. By utilizing different calibration techniques, system 102
accurately
calibrates the location and orientation of different cameras despite potential
occlusion of
reference points and varying lighting conditions.
[57] Embodiments described herein provide various benefits. For example,
embodiments enable stage sets for motion picture productions to use fewer
cameras (e.g.,
8 cameras instead of 60 cameras), because the mobile cameras such as mobile
cameras
122 and 124 are able to capture video including images at multiple locations.
Mobile
cameras 122 and 124 are able to follow target objects such as hero actors
while avoiding
occlusions. As described in more detail herein, system 102 computes the
locations of
mobile cameras 122 and 124 based the locations of reference cameras 112 - 118.
As
such, even if mobile cameras 122 and 124 are often moving and changing
locations,
system 102 continually computes the locations and orientation of cameras 122
and 124.
As such, fewer cameras are needed_ Fewer cameras allow for smaller film crews,
which
results in substantial cost reductions. Fewer cameras also allow for less
hardware gear,
which substantially reduces costs and set up times.
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[58] FIG. 3 is an example flow diagram for calibrating cameras in a live
action scene
using drones, according to some embodiments. Referring to both FIGS. 1, 2, and
3, a
method is initiated at block 302, where a system such as system 102 configures
multiple
reference cameras such as cameras 112 - 118 to observe at least one portion of
a live
action scene such as environment 100 of FIG. 1. In various embodiments, each
reference
camera observes a portion or portions of the live action scene in that each
reference
camera captures and stores images, series of images, and/or videos of the live
action
scene, including capturing and storing reference markers in such images and/or
videos.
As described in more detail herein, in various embodiments, reference cameras
112 - 118
are configuring to observe at least three known reference points located in
the live action
scene and to observe one or more reference points associated with one or more
mobile
cameras having unconstrained motion. As indicated herein, in various
embodiments,
each reference camera of the set of reference cameras 112 - 118 is at a known
location
and position relative to one or more other reference cameras in the live
action scene.
Also, in various embodiments, each reference camera 112 - 118 is mounted on
one or
more rigid structures, such as tripods. As indicated herein, cameras 112 - 118
are located
at various points of environment 100 with wide fields of view in order capture
various
different perspectives of environment 100.
[59] At block 304, system 102 configures one or more moving or mobile
cameras
such as mobile cameras 122 and 124 to observe one or more mobile objects or
moving
objects in one or more portions of the live action scene that one or more of
reference
cameras 112 - 118 also observe. In various embodiments, each mobile camera
observes a
portion or portions of the live action scene in that each mobile camera
captures and stores
images, series of images, and/or videos of the live action scene, including
capturing and
storing reference markers in such images and/or videos. In various
embodiments, mobile
cameras 122 and 124 have unconstrained motion and are configured to observe
one or
more moving objects in the live action scene and to observe at least three
known
reference points associated with the reference cameras. In various
embodiments, the
16
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
mobile objects are moving objects in that they may move from one location on
in the live
action scene to another location in the live action scene. For example, a
given moving
object may be hero actor, vehicle transporting a hero actor, etc., which may
from one
location to another in the live action scene. As indicated herein, in various
embodiments,
each mobile camera is mobile in that each mobile camera follows action in the
live action
scene. In various embodiments, a given mobile camera may be associated with
and
mounted on a mobile apparatus. For example, in various embodiments, each
mobile
camera may be associated with and mounted on a mobile vehicle such as a drone,
as
shown in FIGS. 1 and 2, where mobile cameras 122 and 124 are attached to
respective
drones 126 and 128. The particular type of mobile apparatus may vary,
depending on the
particular implementation. For example, in various embodiments, mobile cameras
122
and 124 and/or other mobile cameras may be attached to moving platforms on a
rail or
wheels. In various embodiments, each mobile camera may be associated with and
mounted on a mobile vehicle such as an automobile, a train car on a rail, etc.
In various
embodiments, each mobile camera may be associated with and mounted on a camera
stabilizer mount, a boom, etc. and hand-carried. The particular means of
movement may
vary and will depend on the particular implementation.
[60] As indicated herein, in various embodiments, at least one moving
object of the
moving objects in the live action scene is a person such as person 130 shown
in FIGS. 1
and 2, which may be a hero actor, for example. In various embodiments, mobile
cameras
122 and 124 are configured to capture objects in environment 100 such as
person 120.
Mobile cameras 122 and 124 are configured to capture details of person 130,
including
any one or more reference points such as reference point 140. Mobile cameras
122 and
124 are configured to self-adjust including zoom, pan, etc. in order to
capture such
details.
[61] At block 306, system 102 obtains or receives reference point data in
association
with one or more mobile cameras such as mobile cameras 122 and 124. In various
17
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
embodiments, the reference point data may be in association the one or more
reference
cameras such as reference cameras 112 - 118. In various embodiments, the
reference
point data is based on one or more reference points located in the live action
scene, where
the one or more reference points are captured by mobile cameras. In various
embodiments, the reference point data may be based on reference points coupled
to one
or more reference cameras. In various embodiments, the reference point data is
based on
at least three known reference points in the live action scene and one or more
reference
points associated with one or more mobile cameras. For example, in various
embodiments, the reference point data may be based on reference points coupled
to
mobile cameras 122 and 124. As indicated above, in various embodiments, at
least a
portion or some of the one or more known reference points such as reference
points 132 -
138 are coupled to at least a portion of the one or more reference cameras 112
- 118.
[62] In various embodiments, the reference point data is based on at least
three
known reference points. Such the three known reference points may include, for
example, one or more of reference points 132 - 138 attached to respective
reference
cameras 112 - 118 of FIG. 1, any one or more other known reference points
located in the
live action scene (such as any one or more reference points 602, 604, and 606
of wand
600 of FIG. 6), and any combination thereof In various embodiments, system 102
computes a location and an orientation of at least camera 902 and/or 904 based
on the
reference point data or combination thereof. Also, in various embodiments, at
least a
portion (e.g., a subset) of the reference points in the live action scene are
coupled to the
one or more moving objects, such as person 130. Further example embodiments
directed
to the reference point data are described in detail herein.
[63] At block 308, system 102 computes a location and an orientation of
each mobile
camera based on one or more of the reference point data and the locations of
one or more
of reference cameras 112 - 118. In various embodiments, system 102 computes
the
location and orientation of the mobile cameras in real time as the mobile
cameras are
18
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
capturing footage of the live action scene. Note that As described in more
detail herein,
once system 102 computes the locations and orientation of one or more of
reference
cameras 112 - 118, system 102 may compute the locations of mobile cameras 122
and
124 based on known reference points captured by mobile cameras 122 and 124. As
described in more detail herein, system 102 computes the locations and
orientation of
mobile cameras 122 and 124 in a similar manner to computing the locations and
orientation of reference cameras 112 - 118. In various embodiments, the
reference point
data may include the locations of one or more of reference points 132 - 138.
In various
embodiments, system 102 computes the location and an orientation of each
mobile
camera based the reference point data and/or one or more locations of one or
more
reference cameras 112 - 118.
[64] Although the steps, operations, or computations may be presented in a
specific
order, the order may be changed in particular implementations. Other orderings
of the
steps are possible, depending on the particular implementation. In some
particular
implementations, multiple steps shown as sequential in this specification may
be
performed at the same time. Also, some implementations may not have all of the
steps
shown and/or may have other steps instead of, or in addition to, those shown
herein.
[65] In a particular example embodiment, system 102 computes the locations
and
orientation of any one or more of reference cameras 112 - 118 based on
reference point
data in association with reference cameras 112 - 118 and based on any one or
more
techniques described herein. System 102 then computes the locations of mobile
camera
122 and/or mobile camera 124 based on reference point data in association with
respective mobile camera 122 and/or mobile camera 124 and based on the
locations of
one or more of reference cameras 112 - 118. In other words, system 102
computes the
locations of one or more of mobile cameras 122 and 124 based on the locations
of one or
more reference cameras 112 - 118.
[66] In some scenarios, the locations of one or more drones with cameras
may be
19
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
known at one point in time. In a subsequent point in time, however, a given
drone may
fly to a new location where its location becomes unknown or uncertain. This
may occur,
for example, if the drone flies into a new area (e.g., from outside to the
inside of a cave,
indoor area, new indoor area, etc.). Consequently, the location of the drone
relative to
one or more associated known referent points (e.g., outside the cave) becomes
unknown
or uncertain. In this scenario, in various embodiments, a camera attached to
the drone
may capture new known reference points in the new area (e.g., in the cave). In
various
embodiments, the camera observes/captures at least three known reference
points
associated with the new known reference points. System 102 may then process
new
reference point data based on those new known referenced points captured by
the camera.
In various embodiments, the reference point data is based on at least three
known
reference points. System 102 processes the reference point data to
recalibrate, including
computing the location and orientation of the camera on the drone relative to
the newly
associated reference points, according to embodiments described herein (e.g.,
based on
reference point data and locations of the known reference points.
[67] FIG. 4 is a block diagram of an example environment 400 for
calibrating
cameras in a live action scene, according to some embodiments. Shown are
system 102,
cameras 402, 404, 406, and 408, and reference point 410. Any one or more of
cameras
402 - 408 may be used to represent any one or more of cameras 112 - 118 in
embodiments described herein.
[68] As described in more detail below, system 102 receives videos
including images
from multiple cameras such as cameras 402 - 408. As described in more detail
herein,
system 102 utilizes cameras 402 - 408 to locate and track the reference points
such as
reference markers on the live action scene or set. In various example
embodiments,
reference points may be also referred to as reference markers. Embodiments
described
herein calibrate cameras 402 - 408, which improve the accuracy of system 102
locating
and tracking reference points.
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[69] Each of cameras 402 - 408 has a field of view (indicated by dotted
lines) that
enables each camera to capture video and/or images of objects in a live action
scene. In
various embodiments, cameras 402 - 408 are stationary at the point of their
calibration
until they need to be moved for subsequent scene changes. Cameras 402 - 408
may be
attached to tripods or other camera stabilizing equipment. In various
embodiments, the
positions of and orientations of cameras 402 - 408 may vary, and will depend
on the
particular implementation.
[70] In various embodiments, if a particular camera is moved (e.g., used in
another
location of the set, used in another set, etc.), that camera may then
recapture reference
point 410 and/or capture and collect other reference points. System 102 may
then
recalculate the new position of the camera.
[71] Cameras 402 - 408 may be any suitable cameras, including cameras
dedicated to
tracking reference points (e.g., active reference markers, passive reference
markers, etc.).
Such cameras may also include infrared cameras and other digital cameras. In
some
embodiments where a reference point is an active reference marker, the
reference marker
emits an infrared light. At least some cameras may have a narrow-pass filter
to detect
and capture the infrared light, which system 102 analyzes to compute the
location of the
active reference marker. Such as an active reference marker may be used to
implement
any one or of reference points described herein.
[72] In various embodiments, objects may include scene props and actors,
and these
objects may have reference points such as reference point 112 attached to them
for
tracking live action tracking purposes. In various embodiments, the reference
points may
be any type of reference or position that system 102 identifies using any
suitable
approach and techniques. Such techniques may vary and the particular
techniques used
will depend on the particular implementation. For example, system 102 may use
techniques involving image recognition, pattern recognition, reference
markers, radio-
frequency identification (RFID), wireless beacons, etc.
21
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[73] As described in more detail herein, system 102 causes cameras 402 -
408 to
project respective rays 412, 414, 416, and 418 into the space and through
reference point
410. For ease of illustration, as indicated above, one reference point 410 is
shown for the
calibration of cameras 104 - 110. There may any number of reference points
used for the
calibration of cameras 104 - 110. The particular number of reference points in
a given
live action scene may vary and will depend on the implementation. For example,
there
may be tens or hundreds of reference points on a given live action scene. In
some
embodiments, system 102 may cause cameras 402 ¨ 408 to also project other
respective
rays into the space and through other reference points.
[74] In various embodiments, the reference point data is based on at least
three
reference points in the live action scene. In various embodiments, the three
reference
points are known relative to each other. In some embodiments, the three
reference points
may be stationary. In various embodiments, the three reference points are
arranged in a
predetermined pattern. Example embodiments directed to the calibration of
cameras
using multiple reference points arranged in a predetermined pattern are
described below
in connection with FIGS. 6 and 7.
[75] In various embodiments, system 102 associates each reference point in
a given
image with a ray from each camera of a set of different cameras that capture
such
reference points in their respective image(s). System 102 searches for and
identifies
intersections of rays 412 - 418 to identify particular reference points. In
various
embodiments, system 102 analyzes information associated with each intersection
to
identify the respective reference point, respective rays that intersect the
reference point,
and respective cameras associated with such rays.
[76] Rays 412 - 418 may also be referred to as epipolar lines 412 -418.
Each
epipolar line 412 - 418 is a straight line of intersection in an epipolar
plane, where each
epipolar line 412 - 418 represents a different point of view of a respective
camera. In
various scenarios, there may be tens of cameras that capture tens or hundreds
of reference
22
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
points. In various scenarios, system 102 may perform thousands or millions of
calculations to analyze different intersections associated with different
references points
in a live action scene.
[77] As system 102 locates the different reference points such as reference
point 410
based on the epipolar lines 412 - 418, system 102 computes or solves for the
3D
coordinates and orientation of each of cameras 402 - 408. Such epipolar
geometry
describes the relationships between different cameras 104 - 110, including
their
respective points of view.
[78] For ease of illustration, one system 102 and four cameras 402 - 408
are shown.
System 102 may represent multiple systems, and cameras 402 - 408 may represent
any
number of cameras. In other implementations, environment 100 may not have all
of the
components shown and/or may have other elements including other types of
elements
instead of, or in addition to, those shown herein.
[79] While system 102 performs embodiments described herein, in other
embodiments, any suitable component or combination of components associated
with
system 102 or any suitable processor or processors associated with system 102
may
facilitate performing the embodiments described herein. Various example
embodiments
directed to environment 100 for calibrating cameras 402 - 408 are described in
more
detail herein.
[80] In various embodiments, the images are taken by the cameras within a
predetermined time frame. For example, in some embodiments, the predetermined
time
frame may be a predetermined number of hours (e.g., 1 hour, 10 hours, 24
hours, etc.), or
predetermined number of days (e.g., 1 day, 7 days, 365 days, etc.). In some
embodiments, the predetermined time frame may be a based on a predetermined
condition. For example, a condition may be that the cameras being calibrated
have not
moved (e.g., changed location and orientation) since the beginning of the
calibration
23
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
process. For example, as long as the cameras have not moved, the cameras may
continue
to take images to be used for calibration. If and when a given camera moves,
the cameras
may continue to captures images, but system 102 will use such images in a new
calibration based on the new or current positions of the cameras.
[81] In some embodiments, system 102 performs embodiments described herein
in
real time. In some embodiments, system 102 need not perform some steps
associated
with embodiments described herein at the same time as the images are captured.
This is
because there may be some delay from the processing and workflow steps before
calibration is completed.
[82] FIG. 5 is a block diagram of an example scenario 500 including a
reference
point captured by cameras in a live action scene, according to some
embodiments.
Shown are cameras 402, 404, 406, and 408, each of which are capturing
respective
images 502, 504, 506, and 508 of reference point 410. While one reference
point 410 is
shown, the number of reference points captured by a given camera may vary, and
the
number will depend on the particular implementation.
[83] As shown, images 502 - 508 show reference point 410 in a different
location in
the different image frames depending on the relative location of reference
point 410 to
the respective camera in the physical live action scene. In various
embodiments, system
102 sends images 502 - 508 to a performance capture system, which may be
remote to
system 102 or integrated with system 102.
[84] In various embodiments, cameras 402 - 408 have a known projection
matrix for
mapping reference points in three-dimensions (3D) to two-dimensional (2D)
points in an
image. In various embodiments, system 102 identifies reference point 410 in 2D
in an
image frame from 3D in the live action scene. System 102 then causes each
camera to
project a ray into the space and through reference point 410 and/or other
references points
in the image. As such, all the cameras see the same reference point 410 in a
different
24
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
place in their respective 2D image frame. As shown, cameras 402 - 408 see the
same
reference point 410 but in different positions in their respective image
frame. The rays
projected by the different cameras 402 - 408 intersect at reference point 410
in the 3D
space, and system 102 computes these intersections.
[85] As indicated above, while some embodiments are described herein in the
context of a single reference point, these embodiments and others also apply
to multiple
reference points. For example, in various embodiments, each camera may capture
3
reference points attached to a wand. System 102 may analyze each reference
point
individually and together as a group, including their relative positions from
each other.
Further examples of such embodiments are described in more detail herein.
[86] FIG. 6 is a block diagram of a group 600 of reference points 602, 604,
and 606
in a live action scene, where reference points 602, 604, and 606 are arranged
in a straight
line, according to some embodiments. As shown, group 600 includes reference
points
602, 604, and 606. In various embodiments, reference points 602, 604, and 606
form a
straight line.
[87] In various embodiments, reference points 602, 604, and 606 are
attached to a
rigid form. For example, in the example embodiment shown, reference points
602, 604,
and 606 are attached to respective rigid arms 608 and 610, which form a
straight line of a
wand. As such, group 600 of reference points may also be referred to as wand
600.
While three reference points 602, 604, and 606 are shown, the number of
reference points
on wand 600 may vary, and the number will depend on the particular
implementation.
For example, there may be 4 references points or 5 reference points, etc.,
attached to
wand 600.
[88] In various embodiments, reference points 602, 604, and 606 of wand 600
are
known or predetermined and their distances from each other are invariant or
predetermined/known and/or do not change. In other words, the absolute length
of wand
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
600 is known, including distances D1 and D2. In the example shown, in various
embodiments, reference points 602, 604, and 606 of wand 600 are equidistant,
were the
distance D1 between reference point 602 and reference point 604 is
substantially equal to
the distance D2 between reference point 604 and reference point 606. In
various
embodiments, the distances between reference points 602, 604, and 606 need not
be
equidistant. For example, the distance D1 and the distance D2 may be different
as long
as the ratio between distances D1 and D2 are known or can be determined using
the
known length of wand 600.
[89] In some embodiments, system 102 collects thousands of frames from
cameras
612 and 614 for one calibration of these cameras. In some embodiments, system
102
may analyze the reference points of wand 600 at different locations and
orientations in
the live action scene in order to optimize calibration measurements. In
various
embodiments, by having at least three reference markers 602 - 606, system 102
accurately computes the orientation of wand 600 regardless of its relative
orientation to a
given camera.
[90] In various embodiments, system 102 computes the location and
orientation of
cameras 612 and 614 based the reference point data and on one or more locating
techniques such as triangulation, trilateration, etc. In various embodiments
where system
applies a triangulation technique, system 102 locates the reference points in
one or more
images. System 102 then computes an aspect ratio of multiple reference points
in the one
or more images. In embodiments where system 102 analyzes a group of 3
reference
points on a wand. Example embodiments directed to a wand with reference points
are
described below in connection with FIG. 4. In various embodiments, system 102
computes the aspect ratio of the three reference points in the one or more
images. System
102 then triangulates each camera based on the aspect ratio.
[91] FIG. 7 is a block diagram of an example scenario 700 including
reference points
in images captured by cameras in a live action scene, according to some
embodiments.
26
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
Shown are cameras 612 and 614, each of which are capturing respective images
702 and
704 of reference points 602, 604, and 606.
[92] In this example embodiment, while distances D1 and D2 are equidistant
in the
3D space, distances D1 and D2 form an aspect ratio in a 2D image, where
distances D1
may differ from distance D2 in the 2D image depending on the point of view of
a given
camera. For example, images 702 and 704 show reference points 602 - 606 in a
different
location in the different image frames depending on the relative location of
reference 602
- 606 to the respective camera 612 or 614 in the physical live action scene.
As shown,
comparing images 702 and 704, the reference points 602 - 606 are farther apart
from each
other in image 702 compared to their relative locations in image 704, where
there may be
some foreshortening due to the camera angle. Also, the position of the group
of reference
points 602 - 606 are positioned more on the right portion of image 702, and
are
positioned more in the center portion of image 704.
[93] In various embodiments, system 102 computes the distance between each
pairing of reference points 602 - 606, including all combinations. In some
embodiments,
system 102 generates a graph of the distance between each reference point to
every other
reference point of wand 600. System 102 computes or ascertains the location of
each of
the reference points of wand 600 and the orientation of the reference points
of wand 600.
Based on the location and orientation of reference points 602 - 606, system
102 computes
the location and orientation of cameras 612 and 614 and any other cameras
capturing
images of reference points 602 - 606.
[94] In various embodiments, system 102 sends images 702 and 704 to a
performance capture system, which may be remote to system 102 or integrated
with
system 102. In various embodiments, system 102 computes or ascertains the
location and
orientation of each camera (e.g., camera 612, camera 614, etc.) based on the
aspect ratio
of distances D1 and D2.
27
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[95] While the reference points of group 600 are shown to be arranged in a
straight
line, the particular arrangement and relative positions of the reference
points may vary
and will depend on the particular implementation. For example, a given group
of
reference points used for calibration of cameras may form a cluster of
reference points,
where the reference points are attached to a rigid form having a three-
dimensional shape.
As such, the reference points may form a three-dimensional pattern.
[96] FIG. 8 is an example flow diagram for calibrating cameras in a live
action scene
using drones, according to some implementations. Referring to both FIGS. 1 and
8, a
method is initiated at block 802, where a system such as system 102 configures
multiple
reference cameras to observe at least three known reference points located in
the live
action scene and to observe one or more reference points associated with one
or more
moving cameras having unconstrained motion.
[97] At block 804, system 102 configures the one or more moving cameras to
observe one or more moving objects in the live action scene.
198] At block 806, system 102 receives reference point data in
association with one
or more of the reference cameras. In various embodiments, the reference point
data is
based on at least three known reference points located the live action scene
and based on
the one or more reference points associated with the one or more moving
cameras.
[99] At block 808, system 102 computes a location and an
orientation of each of the
moving cameras based on one or more of the reference point data and one or
more
locations of one or more of the reference cameras.
1100] In various embodiments, reference cameras may visually
detect a reference
point on a drone and/or a reference point on a camera mounted on the drone
that is
hovering in a particular location. In accordance with various embodiments
described
herein, system 102 computes the locations and orientation or position of the
reference
cameras in the live action scene and their locations and orientation relative
to each other.
28
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
In various embodiments, the reference cameras are stereoscopic. Based on
associated
reference point data, system 102 may compute the location of the mobile camera
on the
drone relative to the known locations of the reference cameras. System 102 may
direct
the drone where to go and when. Because system 102 may compute the location of
the
mobile camera, the mobile camera can zoom in on a given object without concern
for
losing track of other cameras. This is because system 102 continues to track
the mobile
camera using the reference cameras.
[101] In some embodiments, system 102 may compute the location of a given
drone
or estimate the location of the drone based on the previously known location
and
subsequent movement from that location (e.g., distance and direction traveled,
etc.).
System 102 may also obtain any new reference point data based on any new
reference
points captured by the drone and use such reference point data to refine
estimations of the
location of the drone.
[102] Although the steps, operations, or computations may be presented in a
specific
order, the order may be changed in particular implementations. Other orderings
of the
steps are possible, depending on the particular implementation. In some
particular
implementations, multiple steps shown as sequential in this specification may
be
performed at the same time. Also, some implementations may not have all of the
steps
shown and/or may have other steps instead of, or in addition to, those shown
herein.
[103] FIG. 9 is a block diagram of example assembly 900, which includes
cameras
902 and 904 coupled to a mobile apparatus such as a drone 906, according to
some
embodiments. Shown is a reference point 907 coupled to camera 902, a reference
point
908 coupled to camera 904, and a reference point 909 coupled to drone 906.
While some
embodiments are described in the context of a drone such as drone 906, these
embodiments and other also apply to other mobile apparatuses (e.g., vehicles,
drones,
etc.) that carry cameras to follow action of the live action scene. The
apparatus or
assembly 900 applies to other embodiments described herein.
29
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[104] In various embodiments, by having at least two mobile cameras coupled
to a
single drone, one camera such as camera 902 may be configured to capture one
or more
reference points attached to a moving object such as a hero actor. In other
words, camera
902 is configured to follow the action of a scene, including being configured
to observe
one or more moving objects (e.g., actors, mobile vehicles, etc.) in the live
action scene.
Camera 902 may also be referred to as capture camera 902. The other camera
such as
camera 904 may be configured to observe at least three known reference points
located in
the live action scene. Camera 904 may also be referred to as calibration
camera 904. In
various embodiments, camera 904 may be configured to capture one or more
reference
points attached to another camera such as reference points attached to any one
or more of
the cameras of FIGS. 1 and 2. Such cameras may include mobile cameras attached
to
other drones and/or reference cameras (e.g., on the ground, on a platform,
etc.). Camera
904 may be configured to also capture other reference points in the live
action scene,
such as reference points of wand 600 of FIG. 6, etc.
[105] In various embodiments, each camera 902 and 904 may be independently
configured, such that cameras 902 and 904 operate independently from each
other. In
various embodiments, cameras 902 and 904 different fields of view. For
example,
camera 902 may be configured with a narrow field of view to focus on details
of moving
objects such as a person or hero actor. Also, camera 904 may be configured
with a wide
field of view to capture more reference points such as a reference point
associated with
another mobile camera on another drone, as well as other reference points in
the live
action scene.
[106] FIG. 10 is an example flow diagram for calibrating cameras in a live
action
scene using drones, according to some implementations. Referring to both FIGS.
1, 9,
and 10, a method is initiated at block 1002, where a system such as system 102
configures a first camera such as capture camera 902 to observe one or more
moving
objects in a live action scene.
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[107] At block 1004, system 102 configures a second camera such as
calibration
camera 904 to observe at least three known reference points located in the
live action
scene.
[108] At block 1006, system 102 receives reference point data in
association with the
second camera. In various embodiments, the reference point data is based on at
least
three known reference points. For example, the three known reference points
may
include one or more reference points 602, 604, and 606 of wand 600 of FIG. 6,
one or
more of reference points 132, 134, 136, and 138 attached to respective
reference cameras
112, 114, 116, and 118 of FIG. 1, one or more other known reference points
located in
the live action scene, and any combination thereof.
[109] At block 1008, system 102 computes a location and an orientation of
the first
camera and/or the second camera based on the reference point data.
[110] Although the steps, operations, or computations may be presented in a
specific
order, the order may be changed in particular implementations. Other orderings
of the
steps are possible, depending on the particular implementation. In some
particular
implementations, multiple steps shown as sequential in this specification may
be
performed at the same time. Also, some implementations may not have all of the
steps
shown and/or may have other steps instead of, or in addition to, those shown
herein.
[111] FIG. 11 is a block diagram of an example computer system 1100, which
may be
used for embodiments described herein. Computer system 1100 is merely
illustrative and
not intended to limit the scope of the claims. One of ordinary skill in the
art would
recognize other variations, modifications, and alternatives. For example,
computer
system 1100 may be implemented in a distributed client-server configuration
having one
or more client devices in communication with one or more server systems.
[112] In one exemplary implementation, computer system 1100 includes a
display
device such as a monitor 1110, computer 1120, a data entry interface 1130 such
as a
31
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
keyboard, touch device, and the like, a user input device 1140, a network
communication
interface 1150, and the like. User input device 1140 is typically embodied as
a computer
mouse, a trackball, a track pad, wireless remote, tablet, touch screen, and
the like.
Moreover, user input device 1140 typically allows a user to select and operate
objects,
icons, text, characters, and the like that appear, for example, on the monitor
1110.
[113] Network interface 1150 typically includes an Ethernet card, a modem
(telephone, satellite, cable, ISDN), (asynchronous) digital subscriber line
(DSL) unit, and
the like. Further, network interface 1150 may be physically integrated on the
motherboard of computer 1120, may be a software program, such as soft DSL, or
the
like.
[114] Computer system 1100 may also include software that enables
communications
over communication network 1152 such as the HTTP, TCP/IP, RTP/RTSP, protocols,
wireless application protocol (WAP), IEEE 902.11 protocols, and the like. In
addition to
and/or alternatively, other communications software and transfer protocols may
also be
used, for example IPX, UDP or the like. Communication network 1152 may include
a
local area network, a wide area network, a wireless network, an Intranet, the
Internet, a
private network, a public network, a switched network, or any other suitable
communication network, such as for example Cloud networks. Communication
network
1152 may include many interconnected computer systems and any suitable
communication links such as hardwire links, optical links, satellite or other
wireless
communications links such as BLUETOOTHTm, WIFI, wave propagation links, or any
other suitable mechanisms for communication of information. For example,
communication network 1152 may communicate to one or more mobile wireless
devices
1156A-N, such as mobile phones, tablets, and the like, via a base station such
as wireless
transceiver 1154.
[115] Computer 1120 typically includes familiar computer components such as
a
processor 1160, and memory storage devices, such as a memory 1170, e.g.,
random
32
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
access memory (RAM), storage media 1180, and system bus 1190 interconnecting
the
above components. In one embodiment, computer 1120 is a PC compatible computer
having multiple microprocessors, graphics processing units (GPU), and the
like. While a
computer is shown, it will be readily apparent to one of ordinary skill in the
art that many
other hardware and software configurations are suitable for use with the
present
invention. Memory 1170 and storage media 1180 are examples of tangible non-
transitory
computer readable media for storage of data, audio/video files, computer
programs, and
the like. Other types of tangible media include disk drives, solid-state
drives, floppy
disks, optical storage media and bar codes, semiconductor memories such as
flash drives,
flash memories, random-access or read-only types of memories, battery-backed
volatile
memories, networked storage devices, Cloud storage, and the like.
[116] FIG. 12 is a block diagram of an example visual content generation
system
1200, which may be used to generate imagery in the form of still images and/or
video
sequences of images, according to some embodiments. The visual content
generation
system 1200 might generate imagery of live action scenes, computer generated
scenes, or
a combination thereof In a practical system, users are provided with tools
that allow
them to specify, at high levels and low levels where necessary, what is to go
into that
imagery. For example, a user might be an animation artist and might use the
visual
content generation system 1200 to capture interaction between two human actors
performing live on a sound stage and replace one of the human actors with a
computer-
generated anthropomorphic non-human being that behaves in ways that mimic the
replaced human actor's movements and mannerisms, and then add in a third
computer-
generated character and background scene elements that are computer-generated,
all in
order to tell a desired story or generate desired imagery.
[117] Still images that are output by the visual content generation system
1200 might
be represented in computer memory as pixel arrays, such as a two-dimensional
array of
pixel color values, each associated with a pixel having a position in a two-
dimensional
33
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
image array. Pixel color values might be represented by three or more (or
fewer) color
values per pixel, such as a red value, a green value, and a blue value (e.g.,
in RGB
format). Dimensions of such a two-dimensional array of pixel color values
might
correspond to a preferred and/or standard display scheme, such as 1920 pixel
columns by
1280 pixel rows. Images might or might not be stored in a compressed format,
but either
way, a desired image may be represented as a two-dimensional array of pixel
color
values. In another variation, images are represented by a pair of stereo
images for three-
dimensional presentations and in other variations, some or all of an image
output might
represent three-dimensional imagery instead of just two-dimensional views.
[118] A stored video sequence might include a plurality of images such as
the still
images described above, but where each image of the plurality of images has a
place in a
timing sequence, and the stored video sequence is arranged so that when each
image is
displayed in order, at a time indicated by the timing sequence, the display
presents what
appears to be moving and/or changing imagery. In one representation, each
image of the
plurality of images is a video frame having a specified frame number that
corresponds to
an amount of time that would elapse from when a video sequence begins playing
until
that specified frame is displayed. A frame rate might be used to describe how
many
frames of the stored video sequence are displayed per unit time. Example video
sequences might include 24 frames per second (24 FPS), 50 FPS, 80 FPS, or
other frame
rates. In some embodiments, frames are interlaced or otherwise presented for
display, but
for the purpose of clarity of description, in some examples, it is assumed
that a video
frame has one specified display time and it should be understood that other
variations are
possible.
[119] One method of creating a video sequence is to simply use a video
camera to
record a live action scene, i.e., events that physically occur and can be
recorded by a
video camera. The events being recorded can be events to be interpreted as
viewed (such
as seeing two human actors talk to each other) and/or can include events to be
interpreted
34
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
differently due to clever camera operations (such as moving actors about a
stage to make
one appear larger than the other despite the actors actually being of similar
build, or using
miniature objects with other miniature objects so as to be interpreted as a
scene
containing life-sized objects).
[120] Creating video sequences for story-telling or other purposes often
calls for
scenes that cannot be created with live actors, such as a talking tree, an
anthropomorphic
object, space battles, and the like. Such video sequences might be generated
computationally rather than capturing light from live scenes. In some
instances, an
entirety of a video sequence might be generated computationally, as in the
case of a
computer-animated feature film. In some video sequences, it is desirable to
have some
computer-generated imagery and some live action, perhaps with some careful
merging of
the two.
[121] While computer-generated imagery might be creatable by manually
specifying
each color value for each pixel in each frame, this is likely too tedious to
be practical. As
a result, a creator uses various tools to specify the imagery at a higher
level. As an
example, an artist might specify the positions in a scene space, such as a
three-
dimensional coordinate system, might specify positions of objects and/or
lighting, as well
as a camera viewpoint, and a camera view plane. Taking all of those as inputs,
a
rendering engine may compute each of the pixel values in each of the frames.
In another
example, an artist specifies position and movement of an articulated object
having some
specified texture rather than specifying the color of each pixel representing
that
articulated object in each frame.
[122] In a specific example, a rendering engine may perform ray tracing
where a pixel
color value is determined by computing which objects lie along a ray traced in
the scene
space from the camera viewpoint through a point or portion of the camera view
plane that
corresponds to that pixel. For example, a camera view plane may be represented
as a
rectangle having a position in the scene space that is divided into a grid
corresponding to
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
the pixels of the ultimate image to be generated. In this example, a ray
defined by the
camera viewpoint in the scene space and a given pixel in that grid first
intersects a solid,
opaque, blue object, and the given pixel is assigned the color blue. Of
course, for modern
computer-generated imagery, determining pixel colors, and thereby generating
imagery,
can be more complicated, as there are lighting issues, reflections,
interpolations, and
other considerations.
[123] In various embodiments, a live action capture system 1202 captures a
live scene
that plays out on a stage 1204. The live action capture system 1202 is
described herein in
greater detail, but might include computer processing capabilities, image
processing
capabilities, one or more processors, program code storage for storing program
instructions executable by the one or more processors, as well as user input
devices and
user output devices, not all of which are shown.
[124] In a specific live action capture system, cameras 1206(1) and 1206(2)
capture
the scene, while in some systems, there might be other sensor(s) 1208 that
capture
information from the live scene (e.g., infrared cameras, infrared sensors,
motion capture
("mo-cap") detectors, etc.). On the stage 1204, there might be human actors,
animal
actors, inanimate objects, background objects, and possibly an object such as
a green
screen 1210 that is designed to be captured in a live scene recording in such
a way that it
is easily overlaid with computer-generated imagery. The stage 1204 might also
contain
objects that serve as fiducials, such as fiducials 1212(1)-(3), that might be
used post-
capture to determine where an object was during capture. A live action scene
might be
illuminated by one or more lights, such as an overhead light 1214.
[125] During or following the capture of a live action scene, the live
action capture
system 1202 might output live action footage to a live action footage storage
1220. A
live action processing system 1222 might process live action footage to
generate data
about that live action footage and store that data into a live action metadata
storage 1224.
The live action processing system 1222 might include computer processing
capabilities,
36
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
image processing capabilities, one or more processors, program code storage
for storing
program instructions executable by the one or more processors, as well as user
input
devices and user output devices, not all of which are shown. The live action
processing
system 1222 might process live action footage to determine boundaries of
objects in a
frame or multiple frames, determine locations of objects in a live action
scene, where a
camera was relative to some action, distances between moving objects and
fiducials, etc.
Where elements are detected by sensor or other means, the metadata might
include
location, color, and intensity of the overhead light 1214, as that might be
useful in post-
processing to match computer-generated lighting on objects that are computer-
generated
and overlaid on the live action footage. The live action processing system
1222 might
operate autonomously, perhaps based on predetermined program instructions, to
generate
and output the live action metadata upon receiving and inputting the live
action footage.
The live action footage can be camera-captured data as well as data from other
sensors.
[126] An animation creation system 1230 is another part of the visual
content
generation system 1200. The animation creation system 1230 might include
computer
processing capabilities, image processing capabilities, one or more
processors, program
code storage for storing program instructions executable by the one or more
processors,
as well as user input devices and user output devices, not all of which are
shown. The
animation creation system 1230 might be used by animation artists, managers,
and others
to specify details, perhaps programmatically and/or interactively, of imagery
to be
generated. From user input and data from a database or other data source,
indicated as a
data store 1232, the animation creation system 1230 might generate and output
data
representing objects (e.g., a horse, a human, a ball, a teapot, a cloud, a
light source, a
texture, etc.) to an object storage 1234, generate and output data
representing a scene into
a scene description storage 1236, and/or generate and output data representing
animation
sequences to an animation sequence storage 1238.
[127] Scene data might indicate locations of objects and other visual
elements, values
37
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
of their parameters, lighting, camera location, camera view plane, and other
details that a
rendering engine 1250 might use to render CGI imagery. For example, scene data
might
include the locations of several articulated characters, background objects,
lighting, etc.
specified in a two-dimensional space, three-dimensional space, or other
dimensional
space (such as a 2.5-dimensional space, three-quarter dimensions, pseudo-3D
spaces,
etc.) along with locations of a camera viewpoint and view place from which to
render
imagery. For example, scene data might indicate that there is to be a red,
fuzzy, talking
dog in the right half of a video and a stationary tree in the left half of the
video, all
illuminated by a bright point light source that is above and behind the camera
viewpoint.
In some cases, the camera viewpoint is not explicit, but can be determined
from a
viewing frustum. In the case of imagery that is to be rendered to a
rectangular view, the
frustum would be a truncated pyramid. Other shapes for a rendered view are
possible and
the camera view plane could be different for different shapes.
[128] The animation creation system 1230 might be interactive, allowing a
user to
read in animation sequences, scene descriptions, object details, etc. and edit
those,
possibly returning them to storage to update or replace existing data. As an
example, an
operator might read in objects from object storage into a baking processor
that would
transform those objects into simpler forms and return those to the object
storage 1234 as
new or different objects. For example, an operator might read in an object
that has
dozens of specified parameters (movable joints, color options, textures,
etc.), select some
values for those parameters and then save a baked object that is a simplified
object with
now fixed values for those parameters.
[129] Rather than have to specify each detail of a scene, data from the
data store 1232
might be used to drive object presentation. For example, if an artist is
creating an
animation of a spaceship passing over the surface of the Earth, instead of
manually
drawing or specifying a coastline, the artist might specify that the animation
creation
system 1230 is to read data from the data store 1232 in a file containing
coordinates of
38
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
Earth coastlines and generate background elements of a scene using that
coastline data.
[130] Animation sequence data might be in the form of time series of data
for control
points of an object that has attributes that are controllable. For example, an
object might
be a humanoid character with limbs and joints that are movable in manners
similar to
typical human movements. An artist can specify an animation sequence at a high
level,
such as "the left hand moves from location (XI, Y1, Z I) to (X2, Y2, Z2) over
time Ii to
T2", at a lower level (e.g., "move the elbow joint 2.5 degrees per frame") or
even at a
very high level (e.g., "character A should move, consistent with the laws of
physics that
are given for this scene, from point PI to point P2 along a specified path").
[131] Animation sequences in an animated scene might be specified by what
happens
in a live action scene. An animation driver generator 1244 might read in live
action
metadata, such as data representing movements and positions of body parts of a
live actor
during a live action scene, and generate corresponding animation parameters to
be stored
in the animation sequence storage 1238 for use in animating a CGI object. This
can be
useful where a live action scene of a human actor is captured while wearing mo-
cap
fiducials (e.g., high-contrast markers outside actor clothing, high-visibility
paint on actor
skin, face, etc.) and the movement of those fiducials is determined by the
live action
processing system 1222. The animation driver generator 1244 might convert that
movement data into specifications of how joints of an articulated CGI
character are to
move over time.
[132] A rendering engine 1250 can read in animation sequences, scene
descriptions,
and object details, as well as rendering engine control inputs, such as a
resolution
selection and a set of rendering parameters. Resolution selection might be
useful for an
operator to control a trade-off between speed of rendering and clarity of
detail, as speed
might be more important than clarity for a movie maker to test a particular
interaction or
direction, while clarity might be more important than speed for a movie maker
to
generate data that will be used for final prints of feature films to be
distributed. The
39
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
rendering engine 1250 might include computer processing capabilities, image
processing
capabilities, one or more processors, program code storage for storing program
instructions executable by the one or more processors, as well as user input
devices and
user output devices, not all of which are shown.
[133] The visual content generation system 1200 can also include a merging
system
1260 (labeled "Live + CGI Merging System-) that merges live footage with
animated
content. The live footage might be obtained and input by reading from the live
action
footage storage 1220 to obtain live action footage, by reading from the live
action
metadata storage 1224 to obtain details such as presumed segmentation in
captured
images segmenting objects in a live action scene from their background
(perhaps aided
by the fact that the green screen 1210 was part of the live action scene), and
by obtaining
CGI imagery from the rendering engine 1250.
[134] A merging system 1260 might also read data from rule sets for
merging/combining storage 1262. A very simple example of a rule in a rule set
might be
"obtain a full image including a two-dimensional pixel array from live
footage, obtain a
full image including a two-dimensional pixel array from the rendering engine
1250, and
output an image where each pixel is a corresponding pixel from the rendering
engine
1250 when the corresponding pixel in the live footage is a specific color of
green,
otherwise output a pixel value from the corresponding pixel in the live
footage."
[135] The merging system 1260 might include computer processing
capabilities,
image processing capabilities, one or more processors, program code storage
for storing
program instructions executable by the one or more processors, as well as user
input
devices and user output devices, not all of which are shown. The merging
system 1260
might operate autonomously, following programming instructions, or might have
a user
interface or programmatic interface over which an operator can control a
merging
process. In some embodiments, an operator can specify parameter values to use
in a
merging process and/or might specify specific tweaks to be made to an output
of the
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
merging system 1260, such as modifying boundaries of segmented objects,
inserting
blurs to smooth out imperfections, or adding other effects. Based on its
inputs, the
merging system 1260 can output an image to be stored in a static image storage
1270
and/or a sequence of images in the form of video to be stored in an
animated/combined
video storage 1272.
[136] Thus, as described, the visual content generation system 1200 can be
used to
generate video that combines live action with computer-generated animation
using
various components and tools, some of which are described in more detail
herein. While
the visual content generation system 1200 might be useful for such
combinations, with
suitable settings, it can be used for outputting entirely live action footage
or entirely CGI
sequences. The code may also be provided and/or carried by a transitory
computer
readable medium, e.g., a transmission medium such as in the form of a signal
transmitted
over a network.
[137] According to one embodiment, the techniques described herein are
implemented
by one or more generalized computing systems programmed to perform the
techniques
pursuant to program instructions in firmware, memory, other storage, or a
combination.
Special-purpose computing devices may be used, such as desktop computer
systems,
portable computer systems, handheld devices, networking devices or any other
device
that incorporates hard-wired and/or program logic to implement the techniques.
[138] FIG. 13 is a block diagram of an example computer system 1300, which
may be
used for embodiments described herein. The computer system 1300 includes a bus
1302
or other communication mechanism for communicating information, and a
processor
1304 coupled with the bus 1302 for processing information. In some
embodiments, the
processor 1304 may be a general purpose microprocessor.
[139] The computer system 1300 also includes a main memory 1306, such as a
random
access memory (RAM) or other dynamic storage device, coupled to the bus 1302
for
41
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
storing information and instructions to be executed by the processor 1304. The
main
memory 1306 may also be used for storing temporary variables or other
intermediate
information during execution of instructions to be executed by the processor
1304. Such
instructions, when stored in non-transitory storage media accessible to the
processor
1304, render the computer system 1300 into a special-purpose machine that is
customized
to perform the operations specified in the instructions. In various
embodiments,
instructions may include memory-storing instructions, which when executed by
the one
or more processors cause the computer system to carry out embodiments
described
herein.
[140] The computer system 1300 further includes a read only memory (ROM)
1308 or
other static storage device coupled to the bus 1302 for storing static
information and
instructions for the processor 1304. A storage device 1310, such as a magnetic
disk or
optical disk, is provided and coupled to the bus 1302 for storing information
and
instructions.
[141] The computer system 1300 may be coupled via the bus 1302 to a display
1312,
such as a computer monitor, for displaying information to a computer user. An
input
device 1314, including alphanumeric and other keys, is coupled to the bus 1302
for
communicating information and command selections to the processor 1304.
Another
type of user input device is a cursor control 1316, such as a mouse, a
trackball, or cursor
direction keys for communicating direction information and command selections
to the
processor 1304 and for controlling cursor movement on the display 1312. This
input
device 1314 typically has two degrees of freedom in two axes, a first axis
(e.g., x) and a
second axis (e.g., y), that allows the input device 1314 to specify positions
in a plane.
[142] The computer system 1300 may implement the techniques described
herein
using customized hard-wired logic, one or more ASICs or FPGAs, firmware,
and/or
program logic, which, in combination with the computer system, causes or
programs the
computer system 1300 to be a special-purpose machine. According to one
embodiment,
42
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
the techniques herein are performed by the computer system 1300 in response to
the
processor 1304 executing one or more sequences of one or more instructions
contained in
the main memory 1306. Such instructions may be read into the main memory 1306
from
another storage medium, such as the storage device 1310. Execution of the
sequences of
instructions contained in the main memory 1306 causes the processor 1304 to
perform
the process steps described herein. In alternative embodiments, hard-wired
circuitry may
be used in place of or in combination with software instructions.
[143] The term "storage media" as used herein refers to any non-transitory
media that
store data and/or instructions that cause a machine to operate in a specific
fashion. Such
storage media may include non-volatile media and/or volatile media. Non-
volatile media
includes, for example, optical or magnetic disks, such as the storage device
1310.
Volatile media includes dynamic memory, such as the main memory 1306. Common
forms of storage media include, for example, a floppy disk, a flexible disk,
hard disk,
solid state drive, magnetic tape, or any other magnetic data storage medium, a
CD-ROM,
any other optical data storage medium, any physical medium with patterns of
holes, a
RAM, a PROM, an EPROM, a FLASH-EPROM, NVRAM, any other memory chip or
cartridge.
[144] Storage media is distinct from but may be used in conjunction with
transmission
media. Transmission media participates in transferring information between
storage
media. For example, transmission media includes coaxial cables, copper wire,
and fiber
optics, including the wires that include the bus 1302. Transmission media can
also take
the form of acoustic or light waves, such as those generated during radio wave
and
infrared data communications.
[145] Various forms of media may be involved in carrying one or more
sequences of
one or more instructions to the processor 1304 for execution. For example, the
instructions may initially be carried on a magnetic disk or solid-state drive
of a remote
computer. The remote computer can load the instructions into its dynamic
memory and
43
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
send the instructions over a network connection. A modem or network interface
local to
the computer system 1300 can receive the data. The bus 1302 carries the data
to the main
memory 1306, from which the processor 1304 retrieves and executes the
instructions.
The instructions received by the main memory 1306 may optionally be stored on
the
storage device 1310 either before or after execution by the processor 1304.
[146] The computer system 1300 also includes a communication interface 1318
coupled to the bus 1302. The communication interface 1318 provides a two-way
data
communication coupling to a network link 1320 that is connected to a local
network
1322. For example, the communication interface 1318 may be an integrated
services
digital network ("ISDN-) card, cable modem, satellite modem, or a modem to
provide a
data communication connection to a corresponding type of telephone line.
Wireless links
may also be implemented. In any such implementation, the communication
interface
1318 sends and receives electrical, electromagnetic, or optical signals that
carry digital
data streams representing various types of information.
[147] The network link 1320 typically provides data communication through
one or
more networks to other data devices. For example, the network link 1320 may
provide a
connection through a local network 1322 to a host computer 1324 or to data
equipment
operated by an Internet Service Provider ("ISP") 1326. The 1SP 1326 in turn
provides
data communication services through the world wide packet data communication
network
now commonly referred to as the "Internet" 1328. The local network 1322 and
the
Internet 1328 both use electrical, electromagnetic, or optical signals that
carry digital data
streams. The signals through the various networks and the signals on the
network link
1320 and through the communication interface 1318, which carry the digital
data to and
from the computer system 1300, are example forms of transmission media.
[148] The computer system 1300 can send messages and receive data,
including
program code, through the network(s), the network link 1320, and the
communication
interface 1318. In the Internet example, a server 1330 might transmit a
requested code
44
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
for an application program through the Internet 1328, the ISP 1326, the local
network
1322, and the communication interface 1318. The received code may be executed
by the
processor 1304 as it is received, and/or stored in the storage device 1310, or
other non-
volatile storage for later execution.
[149] Operations of processes described herein can be performed in any
suitable order
unless otherwise indicated herein or otherwise clearly contradicted by
context. Processes
described herein may be performed under the control of one or more computer
systems
(e.g., the computer system 1300) configured with executable instructions and
may be
implemented as code (e.g., executable instructions, one or more computer
programs, or
one or more applications) executing collectively on one or more processors, by
hardware,
or combinations thereof The code may be stored on a machine-readable or
computer-
readable storage medium, for example, in the form of a computer program
including a
plurality of machine-readable code or instructions executable by one or more
processors
of a computer or machine to carry out embodiments described herein. The
computer-
readable storage medium may be non-transitory. The code may also be carried by
any
computer-readable carrier medium, such as a transient medium or signal, e.g.,
a signal
transmitted over a communications network.
[150] Although the description has been described with respect to
particular
embodiments thereof, these particular embodiments are merely illustrative, and
not
restrictive. Controls can be provided to allow modifying various parameters of
the
compositing at the time of performing the recordings. For example, the
resolution,
number of frames, accuracy of depth position may all be subject to human
operator
changes or selection.
[151] Any suitable programming language can be used to implement the
routines of
particular embodiments including C, C++, Java, assembly language, etc.
Different
programming techniques can be employed such as procedural or object oriented.
The
routines can execute on a single processing device or multiple processors.
Although the
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
steps, operations, or computations may be presented in a specific order, this
order may be
changed in different particular embodiments. In some particular embodiments,
multiple
steps shown as sequential in this specification can be performed at the same
time.
[152] Some embodiments may be implemented as a system that includes one or
more
processors and logic encoded in one or more non-transitory computer-readable
storage
media for execution by the one or more processors. The logic when executed is
operable
to cause the one or more processors to perform embodiments described herein.
[153] Some embodiments may be implemented as a system that includes one or
more
processors and a non-transitory storage medium storing processor-readable
instructions.
The processor-readable instructions when executed by the one or more
processors of the
system cause the system to carry out embodiments described herein.
[154] Some embodiments may be implemented as a non-transitory computer-
readable
storage medium storing computer-readable code. The computer-readable code when
executed by one or more processors of a computer cause the computer to carry
out
embodiments described herein.
[155] Some embodiments may be implemented as a non-transitory computer-
readable
storage medium with program instructions stored thereon. The program
instructions
when executed by one or more processors are operable to cause the one or more
processors to perform embodiments described herein.
[156] Some embodiments may be implemented as a non-transitory computer-
readable
storage medium for use by or in connection with a instruction execution
system,
apparatus, system, or device. Particular embodiments can be implemented in the
form of
control logic in software or hardware or a combination of both. The control
logic, when
executed by one or more processors, may be operable to perform that which is
described
in particular embodiments.
46
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[157] Some embodiments may be implemented as a non-transitory processor-
readable
storage medium including instructions executable by one or more digital
processors. The
instructions when executed by the one or more digital processors perform
embodiments
described herein.
[158] Some embodiments may be implemented as a carrier medium carrying
computer-readable code. The computer-readable code when executed by one or
more
processors of a computer causes the computer to carry out embodiments
described herein.
[159] Some embodiments may be implemented as processor-implementable code
provided on a computer-readable medium. The computer-readable medium may
include
a non-transient storage medium, such as solid-state memory, a magnetic disk,
optical
disk, etc., or a transient medium such as a signal transmitted over a computer
network.
The processor- implementable code when executed by one or more processors of a
computer causes the computer to carry out embodiments described herein.
[160] Particular embodiments may be implemented by using a programmed
general
purpose digital computer, by using application specific integrated circuits,
programmable
logic devices, field programmable gate arrays, optical, chemical, biological,
quantum or
nanoengineered systems, components and mechanisms may be used. In general, the
functions of particular embodiments can be achieved by any means as is known
in the art.
Distributed, networked systems, components, and/or circuits can be used.
Communication, or transfer, of data may be wired, wireless, or by any other
means.
[161] It will also be appreciated that one or more of the elements depicted
in the
drawings/figures can also be implemented in a more separated or integrated
manner, or
even removed or rendered as inoperable in certain cases, as is useful in
accordance with a
particular application. It is also within the spirit and scope to implement a
program or
code that can be stored in a machine-readable medium to permit a computer to
perform
any of the methods described above.
47
CA 03191221 2023- 2- 28
WO 2022/045898
PCT/NZ2021/050044
[162] As used in the description herein and throughout the claims that
follow, "a",
"an", and "the" includes plural references unless the context clearly dictates
otherwise.
Also, as used in the description herein and throughout the claims that follow,
the meaning
of -in" includes "in" and -on" unless the context clearly dictates otherwise.
[163] Thus, while particular embodiments have been described herein,
latitudes of
modification, various changes, and substitutions are intended in the foregoing
disclosures, and it will be appreciated that in some instances some features
of particular
embodiments will be employed without a corresponding use of other features
without
departing from the scope and spirit as set forth. Therefore, many
modifications may be
made to adapt a particular situation or material to the essential scope and
spirit.
48
CA 03191221 2023- 2- 28
