Language selection

Search

Patent 2308028 Summary

Third-party information liability

Some of the information on this Web page has been provided by external sources. The Government of Canada is not responsible for the accuracy, reliability or currency of the information supplied by external sources. Users wishing to rely upon this information should consult directly with the source of the information. Content provided by external sources is not subject to official languages, privacy and accessibility requirements.

Claims and Abstract availability

Any discrepancies in the text and image of the Claims and Abstract are due to differing posting times. Text of the Claims and Abstract are posted:

  • At the time the application is open to public inspection;
  • At the time of issue of the patent (grant).
(12) Patent: (11) CA 2308028
(54) English Title: SPHERICAL VIEWING/PROJECTION APPARATUS
(54) French Title: APPAREIL DE VISUALISATION/PROJECTION SPHERIQUE
Status: Expired
Bibliographic Data
(51) International Patent Classification (IPC):
  • G03B 37/04 (2006.01)
  • G02B 27/18 (2006.01)
  • H04N 7/15 (2006.01)
(72) Inventors :
  • NALWA, VISHVJIT S. (United States of America)
(73) Owners :
  • FULLVIEW, INC. (United States of America)
(71) Applicants :
  • AT&T CORP. (United States of America)
(74) Agent: KIRBY EADES GALE BAKER
(74) Associate agent:
(45) Issued: 2003-08-26
(22) Filed Date: 1996-10-15
(41) Open to Public Inspection: 1997-05-31
Examination requested: 2000-06-02
Availability of licence: N/A
(25) Language of filing: English

Patent Cooperation Treaty (PCT): No

(30) Application Priority Data:
Application No. Country/Territory Date
08/565,501 United States of America 1995-11-30

Abstracts

English Abstract

The present invention relates to a plurality of sensors' fields of view being redirected by reflective surfaces. Portions of the fields of view of the sensors are removed to produce a first sensor modified field of view with a modified field of view boundary, and a second sensor modified field of view with a modified field of view boundary. Images obtained from the modified fields of view are combined along a modified field of view boundary.


French Abstract

La présente invention concerne une pluralité de champs de vues de capteurs redirigés par des surfaces réfléchissantes. Des parties des champs de vues des capteurs sont retirées pour produire un premier champ de vues de capteur modifié avec une limite de champ de vues modifiée, et un second champ de vues de capteur modifié avec une limite de champ de vues modifiée. Les images obtenues depuis les champs de vues modifiés sont regroupées le long d'une limite de champ de vues modifiée.

Claims

Note: Claims are shown in the official language in which they were submitted.



12

Claims

A method of producing a composite image with a plurality of image processing
devices each having an individual field of view, comprising the steps o~
for at least one of the plurality of image processing devices, redirecting at
least a
portion of the field of view with a reflective area;
merging images corresponding to the individual fields of view to produce the
composite image having a corresponding field of view, wherein at least
portions of each of
two merged fields of view have substantially similar viewing directions where
they are
merged and the viewing directions within these portions where they are merged
appear to
originate substantially from points that are offset from each other.

2. The method of claim 1, wherein the field of view corresponding to the
composite
image has at least one blind region that encompasses at least a portion of an
edge of a
reflective area and that lies between portions of two individual fields of
view.

3. The method of claim 2, wherein images corresponding to the individual
fields of view
are merged electronically by merging data representative of the fields of view
to produce data
representative of the composite image and data representative of the composite
image is
stored by electronic means.

4. The method of claim 3, further comprising displaying the composite image
selectively
using a control means to select a portion of the composite image to be
displayed.

5. The method of claim 4, further comprising displaying at least a portion of
the
composite image remotely after transmitting this portion over a communication
network.

6. The method of claim 1, wherein the number of image processing devices
having a
field of view redirected at least in part by a reflective surface is at least
three.

7. The method of claim 2, wherein the number of image processing devices
having a
field of view redirected at least in part by a reflective surface is at least
three.

8. The method of claim 3, wherein the number of image processing devices
having a
field of view redirected at least in part by a reflective surface is at least
three.


13

9. The method of claim 4, wherein the number of image processing devices
having a
field of view redirected at least in part by a reflective surface is at least
three.

10. The method of claim 5, wherein the number of image processing devices
having a
field of view redirected at least in part by a reflective surface is at least
three.

11. A viewing apparatus having a composite field of view, comprising:
at least two image processing devices, each having an individual field of
view;
at least one reflective area that redirects at least part of the field of view
associated
with one of the image processing devices, the individual fields of view being
merged to
produce the composite field of view, at least portions of each of two merged
fields of view
having substantially similar viewing directions where they are merged, and the
merged fields
of view creating in the composite field of view at least one blind region that
encompasses at
least a portion of an edge of a reflective area and that lies between portions
of two individual
fields of view; and
a memory for storing data representative of said composite field of view
electronically.

12. The viewing apparatus of claim 11, wherein the number of image processing
devices
having a field of view redirected at least in part by a reflective surface is
at least three.

13. A viewing apparatus having a composite field of view, comprising:
at least two image processing devices, each having an individual field of
view;
at least one reflective area that redirects at least part of the field of view
associated
with one of the image processing devices, the individual fields of view being
merged to
produce the composite field of view, at least portions of each of two merged
fields of view
having substantially similar viewing directions where they are merged, and the
merged fields
of view creating in the composite field of view at least one blind region that
encompasses at
least a portion of an edge of a reflective area and that lies between portions
of two individual
fields of view;
a memory for storing data representative of said composite field of view
electronically; and


14

control moans for selectively retrieving said data from said memory in
response to
user signals produced in response to a user's input.

14. The viewing apparatus of claim 13, wherein the number of image processing
devices
having a field of view redirected at least in part by a reflective surface is
at least three.

15. A viewing apparatus having a composite field of view, comprising:
at least two image processing devices, each having an individual field of
view;
at least one reflective area that redirects at least part of the field of view
associated
with one of the image processing devices, the individual fields of view being
merged to
produce the composite field of view, at least portions of each of two merged
fields of view
having substantially similar viewing directions where they are merged, and the
merged fields
of view creating in the composite field of view at least one blind region that
encompasses at
least a portion of an edge of a reflective area and that lies between portions
of two individual
fields of view;
a memory for storing data representative of said composite field of view
electronically;
control means for selectively retrieving said data from said memory in
response to
user signals produced in response to a user's input; and
communication network interface means for transmitting said data retrieved
from said
memory over a communications network and for receiving said user signals over
said
communications network.

16. The viewing apparatus of claim 15, wherein the number of image processing
devices
having a field of view redirected at least in part by a reflective surface is
at least three.

17. A panoramic viewing apparatus, comprising:
a plurality of image processing devices, each having an optical center and a
field of
view; and
a pyramid shaped element having a plurality of reflective side facets facing
in
different directions, each of the plurality of reflective side facets
redirecting a field of view
associated with one of the plurality of image processing devices to create a
plurality of virtual
optical centers within the pyramid shaped element, each of the virtual optical
centers being


15

offset from each other to create a blind region that encompasses an edge of
the reflective side
facet associated with the image processing device.

18. The panoramic viewing apparatus of claim 17, wherein the image processing
devices
are image capture devices.

19. The panoramic viewing apparatus of claim 17, wherein the image processing
devices
are image producing devices.

20. The panoramic viewing apparatus of claim 17, wherein the pyramid shaped
element
has four reflective side facets.

21. The panoramic viewing apparatus of claim 17, wherein each of the optical
centers are
offset in a direction toward a base of the pyramid shaped element from a
position located in a
plane, the plane being parallel to the base and containing a vertex of the
pyramid shaped
element.

22. The panoramic viewing apparatus of claim 21, wherein the image processing
devices
are image capture devices.

23. The panoramic viewing apparatus of claim 21, wherein the image processing
devices
are image producing devices.

24. A panoramic viewing apparatus, comprising:
a plurality of image processing devices, each having a first optical center
and a first
field of view;
a pyramid shaped element having a plurality of reflective side facets facing
in
different directions, each of the plurality of reflective side facets
redirecting a field of view
associated with one of the plurality of image processing devices to create a
plurality of virtual
optical centers within the pyramid shaped element, each of the virtual optical
centers being
offset from each other to create a blind region that encompasses an edge of
the reflective side
facet associated with the image processing device; and
a second image processing device having a second optical center located on a
line that
passes through a vertex of the pyramid shaped element and is perpendicular to
a base of the


16

pyramid shaped element, the second image processing device having a second
field of view
toward the base.

25. The panoramic viewing apparatus of claim 24, wherein the image processing
devices
are image capture devices.

26. The panoramic viewing apparatus of claim 24, wherein the image processing
devices
are image producing devices.

27. The panoramic viewing apparatus of claim 24, wherein the second optical
center is
positioned along the line such that the second field of view does not
encompass a base edge
of the plurality of reflective side facets.

28. The panoramic viewing apparatus of claim 24, wherein each of the first
optical
centers are offset in a direction toward the base from a position located in a
plane, the plane
being parallel to the base and containing the vertex.

29. The panoramic viewing apparatus of claim 28, wherein the image processing
devices
are image capture devices.

30. The panoramic viewing apparatus of claim 28, wherein the image processing
devices
are image producing devices.

Description

Note: Descriptions are shown in the official language in which they were submitted.


CA 02308028 2000-06-02
1
SPHERICAL VIEWING/PROJECTION APPARATUS
This is a division of co-pending Canadian Patent Application Serial No.
2,187,834, filed
October 15, 1996.
Background of the Invention
Field of the Invention:
The present invention relates to a viewing system, more particularly, a
spherical viewing
system.
Description of the Related Art:
In an effort to operate more efficiently, it is desirable to perform some
tasks using
telepresence. For example, many businesses now holdmeetingsusingtelepresence.
Telepresence
is also useful for distance learning and remote viewing of events such as
concerts and sporting
events. A more realistic telepresence is provided to a user by providing the
user with the
capability to switch between views, and thereby mimic, for example, looking
around a meeting
room.
In the past, when several views were made available to a user, several cameras
with
different optical centers were used. Such a situation is illustrated in Fig.
1. Fig. 1 illustrates
cameras 2, 4, 6 and 8 with optical centers 10, 12,14 and 16, respectively.
When the user decided
to change views, he or she simply switched between cameras. In more
sophisticated systems,
when a user decided to change views, he or she was able to obtain a view from
optical centers
10, 12, 14 or 16 as well as from additional optical centers 18, 20, 22, 24 or
26. Views associated
with optical centers such as 18, 20, 22, 24 and 26 were obtained by using
views from the two
cameras nearest to the selected optical center. For example, a view from
optical center 18 was
obtained by using the views from cameras 2 and 4 and interpolating between the
two views so
as to simulate a view from optical center 18. Such procedures introduced
irregularities into
views. In addition, forming these interpolated views required a large amount
of computational
power and time, and thereby made this technique expensive and slow to respond
to a user's
commands. This computational overhead also limited the number of users that
can
simultaneously use the system.

CA 02308028 2002-10-22
2
Summary of the Invention:
One embodiment of the present invention provides an omnidirectional or
panoramic
viewer where several cameras have a common optical center. The field of view
of each of the
cameras is arranged to form a continuous 360 degree view of an area when taken
as a whole.
The user can sweep through 360 degrees of viewing, where each view has the
same or nearly
the same optical center, by simply using the output of one camera or the
combination of two
cameras without requiring the computational overhead of interpolation used in
the prior art.
Such an arrangement may be used to enhance use of virtual meeting rooms by
allowing a
viewer to see the meeting room in a more natural format. This format
corresponds closely to
a person sitting in the actual meeting who simply turns his or her head to
change the view at a
particular time.
In another embodiment of the present invention, several users may use the
viewing
system simultaneously. The data from several cameras is used to provide a
continuous 360
degree view of an area and that data is made available to each user. Each user
simply selects
the data associated with the portion of the view in which he or she is
interested.
In yet another embodiment, the present invention provides a panoramic or
omnidirectional viewing system that can be used to provide a selectable view
to users via a
communications network such as a telephone or cable TV network. The
information
associated with the views produced by this embodiment of the invention may be
transmitted
over a communications network so that each user can select the portion of data
desired for a
particular view.
In still another embodiment of the present invention, a nearly spherical view
is
provided to a user by placing a camera at the common optical center of the
viewer.
In accordance with one aspect of the present invention there is provided a
method of
producing a composite image with a plurality of image processing devices each
having an
individual field of view, comprising the steps of: for at least one of the
plurality of image
processing devices, redirecting at least a portion of the field of view with a
reflective area;
merging images corresponding to the individual fields of view to produce the
composite
image having a corresponding field of view, wherein at least portions of each
of two merged
fields of view have substantially similar viewing directions where they are
merged and the

CA 02308028 2002-10-22
2a
viewing directions within these portions where they are merged appear to
originate
substantially from points that are offset from each other.
In accordance with another aspect of the present invention there is provided a
viewing
apparatus having a composite field of view, comprising: at least two image
processing
devices, each having an individual field of view; at least one reflective area
that redirects at
least part of the field of view associated with one of the image processing
devices, the
individual fields of view being merged to produce the composite field of view,
at least
portions of each of two merged fields of view having substantially similar
viewing directions
where they are merged, and the merged fields of view creating in the composite
field of view
at least one blind region that encompasses at least a portion of an edge of a
reflective area and
that lies between portions of two individual fields of view; and a memory for
storing data
representative of said composite field of view electronically.
In accordance with yet another aspect of the present invention there is
provided a
panoramic viewing apparatus, comprising: a plurality of image processing
devices, each
having an optical center and a field of view; and a pyramid shaped element
having a plurality
of reflective side facets facing in different directions, each of the
plurality of reflective side
facets redirecting a field of view associated with one of the plurality of
image processing
devices to create a plurality of virtual optical centers within the pyramid
shaped element,
each of the virtual optical centers being offset from each other to create a
blind region that
encompasses an edge of the reflective side facet associated with the image
processing device.
Brief Description of the Drawings:
The present invention, taken in conjunction with the invention disclosed in co-

pending Canadian Patent Application Serial No. 2,187,834, filed October 15,
1996, will be
discussed in detail hereinbelow with the aid of the accompanying drawings, in
which:
Fig. I illustrates a prior art multiple camera viewing system;
Fig. 2 illustrates a four camera omnidirectional or panoramic viewing system
using a
four-sided pyramid with reflective surfaces;
Fig. 3 illustrates how a reflective surface of the pyramid is used to provide
each
camera with a common optical center;
Fig. 4 illustrates the top view of the pyramid illustrating the camera
positions;

CA 02308028 2002-10-22
2b
Fig. 5 illustrates an eight-sided pyramid with reflective side surfaces;
Fig 6 is a top view of the pyramid of Fig. 5;
Fig. 7 is a block diagram of a system to control data produced by the cameras.

CA 02308028 2000-06-02
3
Fig. 8 illustrates the association between the data received from the cameras
and
the view presented to a user;
Fig. 9 illustrates an addressing scheme for the memory of Fig. 7;
Fig. 10 is a block diagram of the controller of Fig. 7;
Fig. 11 illustrates using a telecommunications network to provide a selection
of
views to a plurality of users;
Fig. 12 illustrates a second embodiment for providing a selection of views to
multiple users over a telecommunications network;
Fig. 13 illustrates the viewing system of Fig. 2 with a fifth camera; and
Fig. 14 illustrates a top view of the pyramid of Fig.-2 with displaced virtual
optical centers.
Description of the Preferred Embodiment:
Fig. 2 illustrates a four camera system for providing a 360 degree view to a
user,
where the cameras each have a common virtual optical center within the
pyramid.
Pyramid 40 has reflective sides 42, 44, 46 and 48. In a preferred embodiment,
the
reflective sides form a 45 degree angle with a plane parallel to base 50 and
passing
through the vertex of pyramid 40. Cameras 52, 54, 56 and 58 are associated
with
pyramid reflective surfaces 48, 42, 44, and 46, respectively. The cameras may
be image
2o gathering devices such as an optical scanner. As a result, camera 52 views
a reflection
from surface 48 to enable it to view objects in the direction of arrow 60.
Camera 54
views a reflection from surface 42 to view objects in the direction of arrow
62. Camera
56 views a reflection from surface 44 to view objects in the direction of
arrow 64, and
camera 58 views a reflection from surface 46 to view objects in the direction
of arrow 66.
Each camera ha~ a 90 degree field of view. The combination of the four cameras
viewing
reflections from their associated reflective surfaces on pyramid 40, produce a
360 degree
view of the area surrounding pyramid 40. It is desirable to locate the optical
center of
each camera on a plane that is parallel to base 50 and intersects vertex 70 of
pyramid 40.
Each camera's optical center should also be located on a line that passes
through vertex
3o 70 and is perpendicular to the base line of the camera's associated
reflective surface. For
example, the optical center of camera 54 is located on line 72. Line 72 is
perpendicular
to base line 74 of reflective surface 42. Line 72 is in a plane that passes
through vertex
70 and is parallel to base 50. Likewise, the optical center of camera 56 is
positioned on
line 76 which is perpendicular to baseline 78, the optical center of camera 58
is
positioned on line 80 which is perpendicular to base line 82, and the optical
center of
camera 52 is positioned on base line 84 which is perpendicular to base line
86.

CA 02308028 2000-06-02
4
Each camera optical center is positioned on one of the above described lines
at a
distance X from vertex 70 and each camera has its optical axes or direction of
view
pointing perpendicular to base 50. (The distance X should be such that the
reflective
surface reflects as much of the camera's field of view as desired; however,
the defects in
the reflective surface become more visible when the camera is moved closer to
the
reflective surface.) This positioning of optical centers results in the
cameras sharing a
virtual optical center located at position 90. Virtual optical center 90 is
located a distance
X from the vertex 70 on a line that passes through vertex 70 and is
perpendicular to base
50.
1o Fig. 3 illustrates another view of pyramid 40 where only camera 54 is shown
for
the sake of simplicity. Camera 54 is positioned on line 72 so as to have a
virtual optical
center at position 90 within pyramid 40. If camera 54 has a 90 degree field of
view in the
direction perpendicular to base 50, and if the optical center of camera 54 is
at a distance
of X from vertex 70 along line 72, camera 54 has a 90 degree view in the
direction of
arrow 62. In similar fashion, cameras 56, 58, and 52 have 90 degree views in
the
direction of arrows 64, 66, and 60, respectively. This arrangement
inexpensively
produces a 360 degree field of view of an area because cameras with a 90
degree field of
view have relatively inexpensive optics.
Fig. 4 is a top view of pyramid 40. Fig. 4 illustrates the placement of the
optical
2o center of camera 54 along line 72. Line 72 should be in a plane that
passing through
vertex 70 and is parallel to base 50. The line should also be perpendicular to
base line 74
of pyramid 40. The camera's optical center should be positioned a distance X
from
vertex 70 along line 72. The distance X should be such that the reflective
surface reflects
as much of the camera's field of view as desired. Point 100 is located on base
50 at a
position where a line from vertex 70 perpendicularly intersects base 50. In a
similar
fashion, the optical centers of cameras 56, 58 and 52 are positioned the
distance X along
lines 76, 80 and 84, respectively.
Fig. 5 illustrates an eight-sided pyramid 120. Pyramid 120 has reflective
surfaces
122 where each of surfaces 122 form a 45 degree angle with a plane that passes
through
3o vertex 130 and is parallel to base 124. As with the four-sided pyramid of
Fig. 2, each
reflective surface of Fig. 5 may have a camera associated with it. Each
camera's optical
center is positioned on a line that is in a plane that passes through vertex
130 and is
parallel to base 124. The line is perpendicular to base line 132 of the
reflective surface
associated with the camera to be positioned. Using an eight-sided pyramid
offers the
advantage of using cameras with only a 45 degree field of view to obtain a 360
degree

CA 02308028 2000-06-02
view. Cameras with only a 45 degree field of view have inexpensive optics and
enable a
360 degree view to be constructed using relatively inexpensive components.
Fig. 6 is a top view of pyramid 120. As discussed with regard to Fig. 5, each
camera's optical center is positioned along a line 134 which is in a plane
that passes
5 through vertex 130 and is parallel to base 124. The optical centers are
positioned a
distance X along line 134 which is perpendicular to the appropriate base line
132. Point
140 is on base 124 at the point of intersection between base 124 and a line
that passes
through vertex 130 and is perpendicular to base 124.
Pyramids having more or less reflective sides may be used. The advantage of
1o using pyramids having a large number of sides is that cameras with moderate
to small
fields of view may be used. Cameras with moderate fields of view have
relatively
inexpensive optics. The number of sides used in a pyramid is somewhat limited
by the
cost of providing a large number of cameras. A 360 degree view of a scene may
be
provided using a pyramid having three reflective sides. It may be expensive to
use only a
three-sided pyramid in order to provide a 360 degree field of view. This
embodiment of
the invention uses three cameras each with a 120 degree field of view, and
cameras with
such a wide field of view use relatively expensive optical components.
In applications where a full 360 degree view is not desired, it is possible to
build a
viewer that does not have a camera associated with each reflective surface of
the pyramid.
2o Fig. 7 illustrates a block diagram of a system for controlling data
produced by the
cameras of a viewing device such as the viewing device described in figs. 2
through 4.
Cameras 52, 54, 56 and 58 obtain a 360 degree view of an area via their
associated
reflective surfaces of pyramid 40. The image signal or output signal of
cameras 52, 54,
56 and .58 are passed through analog to digital converters (AID) 160, 162,
164, and 166,
respectively. The output of the camera. can be thought of a~s a stream of
pixels and the
output of the A/Ds can be thought of as data representative of the pixels from
the
cameras. The output of the A/Ds are passed through mux 170. Mux 170 allows the
pixel
data from each of the A/Ds to reach memory 172. Controller 174 cycles the
select lines
of mux 170 so that the outputs of all of the A/Ds are stored in memory 172.
Mux 170 is
3o switched at a rate that is four times the pixel rate of the cameras. If
more or less cameras
are used, the rate at which mux 170 is switched will be increased or slowed
accordingly.
It is also possible to eliminate mux 170 and to store the output of each A/D
in a separate
memory. Controller 174 is implemented using a microprocessor which provides
control
signals to counters that control the switching of mux 170 and counters used to
provide
addressing to memory 172. The control signals to the counters include reset,
enable and
a starting offset.

CA 02308028 2000-06-02
6
As a result of the pixel information being passed to memory 172, memory 172
contains a 360 degree view of a scene. Pixel information stored in memory 172
is passed
through digital to analog converter (D/A) 176 and to video display 178. The
actual
portion of memory 172 that is passed to video display 178 via D/A 176 is
controlled via
user input device 180. User input device 180 may be a common device such as a
mouse,
joystick, or keyboard. The user may simply lean a joystick to the right to
shift his view to
the right, lean the joystick to the left to shift the view to the left, or
leave the joystick in
the center to keep the view unchanged. Based on the input from user device
180,
controller 174 varies offsets and starting addresses that are used to provide
addressing to
1 o memory l 72.
Fig. 8 illustrates the relationship between the data provided by the cameras
and
the view available to the user. Since the cameras share a virtual optical
center, the view
can be thought of as a cylindrical view. Sector 200 can be thought of as
representing the
information provided by camera 52, sector 202 can be thought of as
representing the
information provided by camera 54, sector 204 can be thought of as
representing the
information provided by camera 56, and sector 206 can be thought of as
representing the
information provided by camera 58. The surface of the cylinder in each sector
can be
thought of as a collection of columns, where each column is composed of
pixels. For
example, sector 200 can be thought of as a collection of columns including
columns 210,
212, 214 and 216. Likewise, the output produced by camera 54 can be thought of
as a
collection of columns which include column 218 in sector 202 and the output of
camera
58 can include columns such as column 220 in sector 206. The column of pixels
near the
sector boundaries are closer together then the columns near the center of a
sector. This
occurs because the cameras capture the image on a plane while Fig. 8 shows the
columns
projected onto a cylindrical surface.
Fig. 9 illustrates how memory 172 is divided to provide easy access to
different
views based on signals from user input device 180. Sections 230, 232, 234 and
236
correspond to sectors 206, 200, 202 and 204, respectively. Each of sections
230, 232,
234 and 236 can be thought of as a block within memory 172. The blocks in
memory
172 are broken into columns of sequential addresses. The first column of
memory
segment 230 corresponds to the first column. of pixels of sector 206. The
number of
memory positions associated with a column should be at least sufficient to
have one
location for each pixel in a particular column. For example, if a column of
pixels from
Fig. 8 includes 1000 pixels, each column associated with the memory segments
of Fig. 9

CA 02308028 2000-06-02
7
should have at least 1000 locations. The number of columns associated with a
particular
memory segment should be at least equal to the number of columns associated
with a
particular section of the cylinder of Fig. 8.
If a camera scans in a horizontal direction, sequential pixels are written in
adjacent columns of a particular memory segment by simply changing an offset
to a
counter generated address. The overall write address is generated by adding
the offset to
the counter's output. This offset is changed at the rate in which the
horizontally scanned
pixels are received. After a horizontal scan is completed, the counter is
incremented and
once again the offsets are changed at the horizontal scan rate. As a result,
when
to addressing a particular segment of memory during a write cycle, the columns
are
addressed by changing the offset at the horizontal pixel scan rate, and
incrementing the
counter at the vertical scan rate. This type of addressing scheme is used for
accessing
columns within each memory segment. When addressing different memory segments
during a write cycle, a write segment offset is added to the sum of the
counter output and
the column offset. The write segment offset permits addressing memory segments
230,
232, 234, and 236. The segment offset is changed at the same rate as mux 170
is
switched.
Pixel data is read from memory 172 in a similar fashion. The sum of a counter
output and two sets of offsets are used to generate a read address. Once an
initial starting
2o column has been picked, the read address is generated by switching a read
column offset
at a rate that is equal to the horizontal scan rate of a video display. After
reading one
horizontal scans worth of data, the read counter is incremented and the read
column
offsets are changed at a rate equal to the horizontal scan rate of the
display. As a result,
the offset addresses are changing at the display's horizontal display rate and
the counter is
incremented at a rate equal to the vertical scan rate of a display. It is
possible to read data
out at a rate faster or slower than required by the video display; however, if
read out
faster, a buffer memory should be used, if read out slower the video display
may appear
choppy to the viewer.
It should be noted that the cylindrical arrangement of pixels of Fig. 8 is
typically
3o displayed on a flat or nearly flat display. As a result, the image is
displayed by

CA 02308028 2000-06-02
8
compensating for converting between a cylindrical surface and a flat surface.
This may
be carried out using a simple conversion algorithm within a common digital
signal
processing integrated circuit. Methods for these types of conversions are well
known in
the art and can be found in "A Guided Tour of Computer Vision, Vishvjit S.
Nalwa,
Addison-Wesley Publishing Co., Reading, Massachusetts, 1993". It is also
possible to
carry out the conversion using a very high resolution display.
It should be noted that if the view selected by a user corresponds exactly to
the
view of a particular camera, such as camera 52, columns 240-248 are read from
memory
170. Column 240 is the first column in segment 232 and column 248 is the last
column
to in segment 232. If the user decides to move the view in a counter-clockwise
direction,
the start column will shift to the right so that the read operation begins at
column 246 and
ends at column 250. It should be noted that column 246 is the second column
associated
with memory segment 232 which has the pixel data from camera 52, and that
column 250
is the first column of pixel data associated with camera 56. As the user
shifts the view,
t s the starting column shifts in relationship to the user's commands. For
example, if the
user indicates that the view should shift in a counter-clockwise direction,
the start column
of Fig. 9 moves to the right, similarly, if the viewer indicates that the view
should shift in
a clockwise direction, the start column shifts to the left. As before, columns
are
addressed by using offsets, if the offsets involve moving between memory
segments, a
2o read segment offset is added to the sum of the column offset and counter
output.
It should be recalled that the columns near the sector boundaries of Fig. 8
are
closer together. As a result, when the user commands a change in a view and
when the
border of that view is near a sector boundary, the start column changes by a
larger
number of columns for a given angular rotation of the view. Conversely, when
the border
25 of the view is near the center of the sector, the start column changes by a
smaller number
of columns for a given angular rotation.
Fig. 10 illustrates a block diagram of controller 174. Controller 174 includes
microprocessor 270 and memory 272. Memory 272 includes RAM and ROM. Processor
270 receives commands on line 274 from user input device 180. Microprocessor
270
3o controls start, stop and reset of counter 276. Counter 276 controls the
select lines of mux

CA 02308028 2000-06-02
9
170. Counter 276 counts at a rate that is four times the horizontal scan rate
of the
cameras. Write address generator 278 provides write addressing for memory 172.
Write
address generator 278 includes a counter, register foi storing offsets and
adder for adding
the offsets and counter output. Microprocessor 270 controls the offset
selection and the
counters used by write address generator 278. The write addresses are formed
as
described with regard to Fig. 9. Read address generator 280 provides read
addresses to
memory 172. Read address generator 280 includes a counter, register for
storing offsets
and adder for adding the offsets and counter output. As with write address
generator 278,
microprocessor 270 controls the offset selection and the counters of read
address
generator 280. Microprocessor 270 also controls the starting column used by
the
counters based on inputs provided on line 274 from user input 180.
The write and read addresses are provided to memory 172 separately if memory
172 is implemented using a two port memory. If memory 172 is implemented with
a
single port memory, the write and read addresses are multiplexed to memory
172.
Fig. 11 illustrates an embodiment where a panoramic viewer is used to provide
views to several users over a communications network. In this embodiment, all
of the
columns of pixel data are read from memory 172 and placed on bus 300. Buffer
memories 302, 304, 306 and 308 receive the data from bus 300. The buffer
memories are
enabled only when desired columns are available on bus 300. The buffer
memories then
2o pass their information to modems 310, 312, 314 and 316, which then provide
the pixel
data to telecommunications network 318. Telecommunications network 318 then
delivers the information from the modems to the users. In one example, a user
receives
information from modem 316 using modem 320. Modem 320 then provides the pixel
information that was in memory 308 to a local video memory 322. Video memory
provides the pixel information to display 324 for viewing. The user at this
location
controls the view using user input device 326 which may be a device such as a
mouse,
keyboard or joystick. Modem 320 transmits the user input device signals over
telecommunications network 318 to modem 316 which then provides the signals to
enable controller 330. Enable controller 330 receives a signal from controller
174 that
3o indicates which column is being read from memory 172. When the appropriate
column

CA 02308028 2000-06-02
;0
is available, the enable controller 330 enables each of the buffer memories to
receive the
columns of data specif ed by the user input device signals received over the
communication network. As discussed with regard to Fig. 9, enable controller
330 simply
moves the start column based on signals from the user input device. In this
embodiment,
enable controller 330 enables the input to the buffer memory when the pixel
data from the
start column is on bus 300. Enable controller 330 disables the input to the
buffer memory
when the total number of columns of pixels to be viewed are provided to the
buffer
memory. Figure 11 illustrates a system where four users can individually
control their
viewing; however, more users may be accommodated by simply increasing the
number of
1o buffer memories, modems, and ports on enable controller 330.
Fig. 12 illustrates another embodiment in which multiple viewers can use the
panoramic viewer. As the pixel data is read from memory 172, all of the data
is passed
over telecommunications network 318 to telecommunications bridge 350. The
information from memory 172 is provided to bridge 350 via modem 344; however,
the
data may be passed to bridge 350 without use of modem 344 if a digital
connection is
made between memory 172 and bridge 350. Bridge 350 then distributes all of the
data
received from memory 172 to each user in communication with bridge 350. If
bridge 350
provides analog link to users, a modem should be used at each user port. If
the bridge has
a digital link to the user ports, a modem is not required. In the case of an
analog link, the
2o data from memory 172 passes from modem 344 via bridge 350 to modem 360 at a
user
port. Modem 360 passes the pixel data to video memory 362. Video memory 362
then
passes the pixel information to video display 364 under control of a user via
user input
device 366. User input device may be a mouse, joystick or computer keyboard.
In this
embodiment, the entire contents of memory 172 is fed to video memory 362. The
data
read from memory 362 and passed to video display 364 is controlled using user
input
device 366 in a fashion similar to that which was described with regard to
Fig. 9.
Fig. 13 illustrates the viewing system of Fig. 2 with a fifth camera. Camera
or
image gathering device 400 is located in pyramid 40 with the optical center of
camera
400 located at virtual optical center 90. Camera 400 views objects in the
direction of
3o arrow 410. It is desirable to provide camera 400 with a wide angle lens.
The resulting

CA 02308028 2002-10-22
wide angle view coupled with the views of the remaining four cameras, provides
a nearly
spherical view. If the cameras of Fig. 13 are replaced with image producing
devices, the
nearly spherical viewing system becomes a nearly spherical projection system.
It should be
noted, that a camera or projection device, may be placed at the virtual
optical center of
viewing/projection devices having pyramids with three, four or more sides. It
should also be
noted that edges 420 of the reflective surfaces should be beveled to avoid
undesirable
obstruction of camera 400's Held of view.
Fig. 14 illustrates a top view of the pyramid of Fig. 2. In reference to Fig.
2, the
individual image processing devices in the form of cameras 52, 54, 56 and 58
have been
moved upward in the direction of base 50. As a result, virtual optical centers
500, 502, 504
and 506, which are the points where the viewing directions of the individual
fields of view of
cameras 52, 54, 56 and 58, respectively, appear to originate, are moved away,
i.e., offset,
from virtual optical center 90. It is desirable to move the virtual optical
centers so that
camera 52 captures an image between lines 508 and 510, camera 54 captures an
image
between lines 512 and 514, camera 56 captures an image between lines S 16 and
518, and
camera 58 captures an image between lines 520 and 522. This results in the
cameras not
capturing useful images from narrow planar shaped regions when the images from
the
cameras are merged as described above. In particular, planar regions 524, 526,
528 and 530
are not used by the cameras and form blind regions in the composite merged
image. This
offers the advantage of removing portions of the cameras' fields of view that
are received
from the edges of the reflective areas of the pyramid. Eliminating these
portions of the fields
of view alleviates the need to provide image processing that compensates for
image artifacts
at the edges. It is desirable to keep virtual optical centers 500, 502, 504
and 506 closely
clustered so that planes 524, 526, 528 and 530 are only as thin as necessary
to avoid edge
artifacts. By maintaining such thin planes, the need to process the images at
their common
boundaries is removed while minimizing the noticeable effect seen by a user.

Representative Drawing
A single figure which represents the drawing illustrating the invention.
Administrative Status

For a clearer understanding of the status of the application/patent presented on this page, the site Disclaimer , as well as the definitions for Patent , Administrative Status , Maintenance Fee  and Payment History  should be consulted.

Administrative Status

Title Date
Forecasted Issue Date 2003-08-26
(22) Filed 1996-10-15
(41) Open to Public Inspection 1997-05-31
Examination Requested 2000-06-02
(45) Issued 2003-08-26
Expired 2016-10-17

Abandonment History

There is no abandonment history.

Payment History

Fee Type Anniversary Year Due Date Amount Paid Paid Date
Request for Examination $400.00 2000-06-02
Registration of a document - section 124 $50.00 2000-06-02
Application Fee $300.00 2000-06-02
Maintenance Fee - Application - New Act 2 1998-10-15 $100.00 2000-06-02
Maintenance Fee - Application - New Act 3 1999-10-15 $100.00 2000-06-02
Maintenance Fee - Application - New Act 4 2000-10-16 $100.00 2000-09-21
Maintenance Fee - Application - New Act 5 2001-10-15 $150.00 2001-10-11
Extension of Time $200.00 2002-08-22
Maintenance Fee - Application - New Act 6 2002-10-15 $150.00 2002-09-30
Final Fee $300.00 2003-06-10
Maintenance Fee - Patent - New Act 7 2003-10-15 $150.00 2003-10-06
Maintenance Fee - Patent - New Act 8 2004-10-15 $200.00 2004-09-28
Maintenance Fee - Patent - New Act 9 2005-10-17 $200.00 2005-04-04
Maintenance Fee - Patent - New Act 10 2006-10-16 $250.00 2006-03-31
Maintenance Fee - Patent - New Act 11 2007-10-15 $250.00 2007-07-10
Maintenance Fee - Patent - New Act 12 2008-10-15 $250.00 2008-05-22
Maintenance Fee - Patent - New Act 13 2009-10-15 $250.00 2009-01-09
Maintenance Fee - Patent - New Act 14 2010-10-15 $250.00 2010-04-01
Maintenance Fee - Patent - New Act 15 2011-10-17 $450.00 2011-03-21
Maintenance Fee - Patent - New Act 16 2012-10-15 $450.00 2012-01-03
Maintenance Fee - Patent - New Act 17 2013-10-15 $450.00 2013-10-15
Maintenance Fee - Patent - New Act 18 2014-10-15 $450.00 2014-10-03
Maintenance Fee - Patent - New Act 19 2015-10-15 $450.00 2015-04-20
Registration of a document - section 124 $100.00 2018-08-23
Registration of a document - section 124 $100.00 2018-08-23
Registration of a document - section 124 $100.00 2018-09-14
Owners on Record

Note: Records showing the ownership history in alphabetical order.

Current Owners on Record
FULLVIEW, INC.
Past Owners on Record
AT&T CORP.
LUCENT TECHNOLOGIES GUARDIAN CORP.
LUCENT TECHNOLOGIES INC.
NALWA, VISHVJIT S.
Past Owners that do not appear in the "Owners on Record" listing will appear in other documentation within the application.
Documents

To view selected files, please enter reCAPTCHA code :



To view images, click a link in the Document Description column. To download the documents, select one or more checkboxes in the first column and then click the "Download Selected in PDF format (Zip Archive)" or the "Download Selected as Single PDF" button.

List of published and non-published patent-specific documents on the CPD .

If you have any difficulty accessing content, you can call the Client Service Centre at 1-866-997-1936 or send them an e-mail at CIPO Client Service Centre.


Document
Description 
Date
(yyyy-mm-dd) 
Number of pages   Size of Image (KB) 
Cover Page 2000-08-07 1 29
Cover Page 2003-07-23 1 33
Claims 2000-06-02 1 30
Abstract 2000-06-02 1 13
Description 2000-06-02 12 687
Drawings 2000-06-02 7 131
Claims 2002-10-22 5 215
Description 2002-10-22 13 726
Representative Drawing 2000-08-07 1 7
Correspondence 2003-06-10 1 36
Fees 2005-04-04 1 23
Prosecution-Amendment 2002-04-22 2 55
Correspondence 2002-08-22 1 28
Correspondence 2002-10-09 1 13
Prosecution-Amendment 2002-10-22 12 467
Assignment 2000-06-02 3 94
Correspondence 2000-07-05 1 1
Fees 2006-03-31 1 24
Fees 2007-07-10 1 23
Fees 2008-05-22 1 26
Fees 2009-01-09 1 27
Fees 2010-04-01 1 26
Fees 2011-03-21 1 23
Fees 2012-01-03 1 24
Fees 2013-10-15 1 24
Fees 2015-04-20 1 26
Fees 2014-10-03 1 24