Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
1
SUMIVIARY
One form of the present invention is a unique charting system.
Other forms include unique systems and methods to generate
panoramic images and to create charts based on the panoramic inzages. Yet
another form includes unique systems and methods to manage charts based
on panoramic images.
Another form includes providing a computer system and rnethod
that generates panoramic images and related charts. The system allows a
user to specify which images the system should use for generating the
panoramic image. The system then generates a panoramic image. An
image-based chart and a graphic-based chart can be generated from the
panoramic image. A user can input chart data on one or both of the charts
using one or more input devices. A user can also select an option to have
the system import chart data from an external source and display the data
on the chart. Any inputted and imported data are displayed on the
applicable chart for the user to view and analyze. A user can toggle
between the individual images, the panoramic image, and the charts. A
user can overlay one image or chart over the other as desired. A user can
also replace an individual image in the panoramic image as desired.
Yet other forms, embodiments, objects, advantages, benefits,
features, and aspects of the present invention will become apparent from
the detailed description and drawings contained herein.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
2
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a diagrammatic view of a computer system of one
embodiment of the present invention.
FIG. 2A illustrates a first portion of a high-level process flow
diagram for the system of FIG. 1.
FIG. 2B illustrates a second portion of a high-level process flow
diagram for the system of FIG. 1.
FIG. 3 is a process flow diagram for the system of FIG. 1
demonstrating the stages involved in generating a panoramic image.
FIG. 4 is a process flow diagram for the system of FIG. 1
demonstrating the stages involved in generating an image-based chart from
a panoramic image.
FIG. 5 is a process flow diagram for the system of FIG. 1
demonstrating the stages involved in generating a graphic-based chart from
a panoramic image.
FIG. 6 is a process flow diagram for the system of FIG. 1
demonstrating the stages involved in managing a panoramic image.
FIG. 7 is a first simulated set of images that illustrate an example of
individual images that can be used by the system of FIG. 1 for generating a
panoramic image, as described in the procedures of FIGS. 2-3.
FIG. 8 is a first visually simulated illustration of the programmatic
process used by the system of FIG. 1 to align or wrap images to a
panoramic plane, as described in the procedure of FIG. 3.
FIG. 9 is a first illustration of a panoramic image generated by the
system of FIG. 1, as described in the procedures of FIGS. 2-3.
FIG. 10 is a first illustration of an image-based chart generated by
the system of FIG. 1, as described in the procedures of FIGS. 2A-2B and
FIG. 4.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
3
FIG. 11 is a first illustration of a graphic-based chart generated by
the system of FIG. 1, as described in the procedures of FIGS. 2A-2B and
FIG. 5.
FIG. 12 is a second simulated set of images that illustrate an
example of individual images that can be used by the system of FIG. 1 for
generating a panoramic image, as described in the procedures of FIGS. 2-3.
FIG. 13 is a second visually simulated illustration of the
programmatic process used by the system of FIG. 1 to align or wrap images
to a panoramic plane, as described in the procedure of FIG. 3.
FIG. 14 is a second illustration of a panoramic image generated by
the system of FIG. 1, as described in the procedures of FIGS. 2-3.
FIG. 15 is a second illustration of an image-based chart generated
by the system of FIG. 1, as described in the procedures of FIGS. 2A-2B and
FIG. 4.
FIG. 16 is a second illustration of a graphic-based chart generated
by the system of FIG. 1, as described in the procedures of FIGS. 2A-2B and
FIG. 5.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
4
DETAILED DESCRIPTION
For the purposes of promoting an understanding of the principles of
the invention, reference will now be made to the embodiment illustrated in
the drawings and specific language will be used to describe the same. It
will nevertheless be understood that no limitation of the scope of the
invention is thereby intended. Any alterations and further modifications in
the described embodiments, and any further applications of the principles
of the invention as described herein are contemplated as would normally
occur to one skilled in the art to which the invention relates. FIG. I is a
diagrammatic view of computer system 10 of one embodiment of the
present invention. Computer system 10 includes client workstation 12.
While client workstation 12 is illustrated as being a client computer, it
should be understood client workstation 12 may be arranged to include both
a client and server. Furthermore, it should be understood that while one
computer is illustrated, more than one computer may be utilized in
alternative embodiments.
Client workstation includes one or more processors or CPUs 14 and
one or more types of memory 16. Each memory 16 includes a removable
memory device, although not shown to preserve clarity. The processor may
be comprised of one or more components configured as a single unit.
Alternatively, when of a multi-component form, a processor may have one
or more components located remotely relative to the others. One or more
components of each processor may be of the electronic variety defining
digital circuitry, analog circuitry, or both. In one embodiment, processor is
of a conventional, integrated circuit microprocessor arrangement, such as
one or more PENTIUM III or PENTIUM 4 processors supplied by INTEL
Corporation of 2200 Mission College Boulevard, Santa Clara, Calif. 95052,
USA.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
Memory (removable or generic) is one form of coniputer-readable
device. Memory may include one or more types of solid-state electronic
memory, magnetic memory, or optical memory, just to natne a few. By
way of non-limiting example, memory may include solid-state electronic
5 Random Access Memory (RAM), Sequentially Accessible Memory (SAM)
(such as the First-In, First-Out (FIFO) variety or the Last-In-First-Out
(LIFO) variety), Programmable Read Only Memory (PROM),
Electronically Programmable Read Only Memory (EPROM), or
Electrically Erasable Programmable Read Only Memory (EEPROM); an
optical disc memory (such as a DVD or CD ROM); a magnetically encoded
hard disc, floppy disc, tape, or cartridge media; or a combination of any of
these memory types. Also, memory may be volatile, nonvolatile, or a
hybrid combination of volatile and nonvolatile varieties.
Client workstation 12 includes a display 22 and one or more input
devices 24. Input devices 24 may include one or more operator input
devices such as a keyboard, electronic pen input device, mouse, track ball,
light pen, microtelecommunicator, gum pocket size measurement device,
and/or a cavity detection device, to name just a few representative
examples. Alternatively or additionally, client workstation 12 includes a
printer 25. Alternatively or additionally, client workstation 12 has access to
external data 36 in other systems and/or devices, such as external charting
programs and digital cameras, to name a few non-limiting examples.
Client workstation 12 also includes a user interface 26, business logic 28,
and database 30. Database 30 stores image data 32 and other information
34 such as chart data. Business logic 28 includes the computer executed
procedures described in FIGS. 2-6. Business logic 28 interfaces with user
interface 26 to communicate with a user, and business logic 28 interfaces
with database 30 to retrieve and store data. In one embodiment, client
workstation 12 is disconnected from computer network 20. In another
embodiment, client workstation 12 is connected to network 20.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
6
Although only one client workstation 12 is shown to preserve
clarity, more client workstations and/or servers could also be present. In
such instances, multiple client workstations 12, displays 22, input devices
24, and printers 25 may be of the same respective type, or a heterogeneous
combination of different computing devices. When more computers are
present, client workstation 12 can be coupled to other computers over
computer network 20. Computer network 20 could be in the form of a
Local Area Network (LAN), Municipal Area Network (MAN), Wide Area
Network (WAN), such as the Internet, a combination of these, or such other
network arrangement as would occur to those skilled in the art. The one or
more features provided by client workstation 12 can be provided on the
same computer or varying other arrangements of computers at one or more
physical locations and still be within the spirit of the invention. As one
non-limiting example, a client-server arrangement can be used where the
database 30 is located on one or more central servers that are accessible
over computer network 20 by user interface 26 and business logic 28 that is
present on one or more client computers 12. As another non-limiting
example, a 3-tier/n-tier arrangement can be used over computer network 20
where the database 30 is located on one or more server computers, where at
least some of the business logic 28 runs on one or more server computers,
and where the user interface 26 resides on one or more client computers 12.
Various other computer architecture arrangements are possible as is
appreciated by one of ordinary skill in the computer art, and such
arrangements are still within the spirit of the invention. The operating logic
of system 10 can be embodied in signals in programming instructions,
dedicated hardware, transmitted over computer network 20, or a
combination of these.
In one embodiment, system 10 serves as an image charting system,
with client workstation 12 containing a user interface 26, business logic 28,
and database 30 that allow for creation and management of charts and
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
7
related data. As one non-limiting example, system 10 can be used by a
dentist to manage patient dental charts and associated information. The
system is not limited to dentistry, or the field of medicine, as will be
understood by one in the art. The system can be used in various industries
where generating a panoramic image and using the panoramic image as part
of a chart for further analysis would be useful. As a few non-limiting
examples, the system can be used for engineering, real estate,
pharmaceuticals, and manufacturing.
Referring additionally to FIGS. 2A-2B, one embodiment for
implementation with system 10 is illustrated in flow chart form as
procedure 100, which demonstrates a high level process flow diagram of
some of the features provided by system 10. In one form, procedure 100 is
at least partially implemented in the operating logic of system 10.
Procedure 100 begins on FIG. 2A at start point 101 with generating a
panoramic image from multiple individual images or from importing an
existing panoramic image (stage 102). An image-based chart is generated
from the panoramic image (stage 104). Alternatively or additionally to the
image-based chart, a graphic-based chart is generated from an outline of the
panoramic image (stage 106). A user can input chart data onto one or both
of the charts using one or more input devices (stage 108). A few non-
limiting examples of input devices include a mouse, keyboard, electronic
pen, personal digital assistant (PDA), cavity detection device, and a gum
pocket size measurement device. This is not an exhaustive list, and various
other types of input devices can be used to input chart data onto one or both
of the charts.
The user can select an option to retrieve and display external chart
data that is imported from an external source (stage 110). Chart data can be
imported from a variety of data sources and/or devices, such as an external
program, a digital camera, or a personal digital assistant (PDA), to name a
few non-limiting examples. Turning now to FIG 2B, the inputted and/or
imported chart data is displayed on the applicable chart to the user on
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
8
display 22 (stage 112). The user can toggle between the individual images,
the panoramic image and the charts and/or overlay one over the other as
desired (stage 114). In one embodiment, the toggle feature provides the
user with the ability to view the information in a variety of ways and to
navigate among the charts and underlying images easily. There may be
some instances where overlaying one image or chart on top of another
image or chart is useful, such as for a dentist diagnosing a certain dental
condition. If desired, the user can replace an individual image included in
the panoramic image (116) with a new image. Upon replacement, a revised
panoramic image can be generated in accordance with the procedure of
FIG. 3. The process then ends at end point (118).
Turning now to FIG. 3, procedure 120 demonstrates the stages
involved in generating a panoramic image (stage 102 in FIG. 2A). In one
forin, procedure 120 is at least partially implemented in the operating logic
of system 10. Procedure 120 begins at start point 121 with receiving
images in an electronic format (stage 122). Images that are received can
include images that were captured with a digital camera, or that were
scanned into an electronic format from a film-based image, to name a few
non-limiting examples. The user can select an option in user interface 26 to
specify from where the system can retrieve the images. At least one object
in each of the images is programmatically or manually identified (stage
124). In one embodiment, system 10 programmatically identifies the
objects. Alternatively or additionally, the user has an option to identify at
least some of the objects to assist system 10 in making a proper
identification. An example of an object in the case of a dental image
includes a tooth. A series of boundary locations around each of the objects
is programmatically or manually identified (stage 126). Each of the images
is aligned or warped to a panoramic plane (stage 128). A panoramic image
is generated from a mosaic of the aligned images (stage 130). The
procedure then ends at end point 132.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
9
Turning now to FIG. 4, procedure 140 demonstrates the stages
involved in generating an image-based chart from a panoramic image
(stage 104 in FIG. 2A). In one form, procedure 140 is at least partially
implemented in the operating logic of system 10. Procedure 140 begins at
start point 141 with generating an image-based chart that includes a copy of
the panoramic image or the original panoramic image (stage 142). The user
can input and/or import data onto the chart using one or more input devices
and/or external sources (stage 144). The inputted and/or imported data is
displayed in an overlay fashion upon (e.g. on top of) the panoramic image
in the chart (stage 146). By storing and/or displaying the chart data in an
overlay fashion, the underlying image in the chart is preserved.
Alternatively or additionally, the user can more easily work with the chart,
such as by turning the display of chart data on and off as desired.
Alternatively or additionally, the chart data can be permanently saved
directly on the chart in non-overlay fashion. The procedure ends at end
point 148.
Turning now to FIG. 5, procedure 150 demonstrates the stages
involved in generating a graphic-based chart from a panoramic image
(stage 106 in FIG. 2A). In one form, procedure 150 is at least partially
implemented in the operating logic of system 10. Procedure 150 begins at
start point 151 with generating a graphic-based chart that includes at least
an outline of the contents of the panoramic image (stage 152). In the case
of a panoramic dental image, the outline can include teeth, soft tissue areas,
and/or pocket depth, to name a few non-limiting examples. The user can
input and/or import data onto the chart using one or more input devices
and/or external sources (stage 154). The inputted and/or imported data is
displayed in an overlay fashion upon (e.g. on top of) the panoramic image
in the chart (stage 156). Again, by storing and/or displaying the chart data
in an overlay fashion, the underlying image in the chart is preserved.
Alternatively or additionally, the user can more easily work with the chart,
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
such as by turning the display of chart data on and off as desired.
Alternatively or additionally, the chart data can be permanently saved
directly on the chart in non-overlay fashion. The procedure ends at end
point 158.
5 Turning now to FIG. 6, procedure 160 demonstrates the stages
involved in managing a panoramic image (stage 116 in FIG. 2B). In one
form, procedure 160 is at least partially implemented in the operating logic
of system 10. Procedure 160 begins at start point 161 with storing in
database 30 location information (other data 34) for each of the images to
10 associate each image with a particular location in the panoramic image
(stage 162). A user can select an area of the panoramic image from the
panoramic image itself or from one of the charts (stage 164). Location
information is then retrieved from database 30 to determine which of the
images is associated with the selected area of the panoramic image (stage
166). The particular one or more of the images associated with the selected
area is displayed on display 22 (stage 168). A user can specify an option to
replace a selected particular one of the images with a replacement image
(stage 170). Upon replacement, a revised panoramic image can be
generated in accordance with the procedure of FIG. 3. The process then
ends at end point 172.
A first hypothetical example will now be described in FIGS. 7-11 to
illustrate the procedures of FIGS. 2-6. A dental example will be used, but
one of ordinary skill in the art will appreciate that other fields of art
related
to or outside of the field of medicine are also within the spirit of the
invention. FIG. 7 illustrates a simulated set of dental images 200 used for
generating a panoramic image, as described in FIGS. 2-3. Turning now to
FIG. 8 (with continued reference to FIG. 7), a visually simulated
illustration of the programmatic process used by system 10 in carrying out
the procedures of FIGS. 2-3 is shown. Individual dental images 200 are
received by system 10 in an electronic format (stage 122). Dental images
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
11
200 were captured with a digital camera, but film-based images that have
been scanned to an electronic format, or other similar images types could
also be used. In the current example, the dental images 200 are a full-
mouth series of images, but more images or less images than a full-mouth
series could be used. After the user selects an option to specify from where
systern 10 can retrieve the dental images, at least one object (e.g. tooth in
this example) in each of the images is programmatically identified (stage
124) by business logic 28. Alternatively or additionally, the user has an
option to identify at least some of the objects in images 200 to assist system
10 in making a proper identification.
A series of boundary locations around each of the objects is
programmatically identified (stage 126). Using the information obtained in
the identification steps, system 10 has aligned or warped 210 each of dental
images 200 to a panoramic plane (stage 128). Turning now to FIG. 9 with
continued reference to FIG 8, a panoramic image 212 is generated from a
mosaic 214 of the aligned images 210 (stage 130).
Turning now to FIG. 10, an illustration of an image-based dental
chart 216 generated by system 10 in accordance with the procedures of
FIGS - 2A-2B and FIG. 4 is shown. Image-based dental chart 216 includes
a copy of panoramic dental image 212 (stage 142), and also includes
symbols 218 and 220 (chart data) that were input by a user from an input
device 24 (stage 144). The chart data (218 and 220) is displayed in an
overlay fashion on top of the panoramic image included in chart 216 (stage
146). By overlaying the chart data (218 and 220), the underlying
panoramic dental image is preserved. Alternatively or additionally, the
user can more easily work with the chart, such as by turning the display of
chart data on and off as desired.
Turning now to FIG. 11, an illustration of graphic-based dental
chart 222 generated by system 10 in accordance with the procedures of
FIGS. 2A-2B and FIG. 5 is shown. In the current example of a panoramic
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
12
dental image, graphic-based chart 222 includes an outline of the teeth, soft
tissue areas, and pocket depth (stage 152). Graphic-based dental chart 222
includes symbols 224, 226, and 228 (chart data) that were input by a user
from an input device 24 (stage 154). The chart data (224, 226, and 226) is
displayed in an overlay fashion upon (e.g. on top of) the panoramic image
in chart 222 (stage 156). Again, by storing and/or displaying the chart data
in an overlay fashion, the underlying image in the chart is preserved.
Alternatively or additionally, the user can more easily work with the chart,
such as by turning the display of chart data on and off as desired.
As another non-limiting medical example, a user, such as a doctor
or other medical professional, uses the system and method of the current
invention to capture images of a basal cell carcinoma over time. The user
takes measurements of the basal cell carcinoma and records the
measurements and/or other notes on a corresponding medical digital chart
(such as 216 and/or 222) that was created using system 10. Alternatively or
additionally, if the photos were taken with the same distance between the
camera and the carcinoma, then comparison measurements can be made
from the digital images and/or digital charts themselves, since the scale is
the same from picture to picture. This allows the user to chart growth size,
shape, and color of the basal cell carcinoma over time.
Another hypothetical example will now be described in FIGS. 12-
16 to further illustrate the procedures of FIGS. 2-6. A real estate example
will be used, but one of ordinary skill in the art will again appreciate that
other fields of art related to or outside of the field of real estate are
within
the spirit of the invention. FIG. 12 illustrates a simulated set of real
estate
images 240 used for generating a panoramic real estate image, as described
in FIGS. 2-3. Turning now to FIG. 13 (with continued reference to FIG.
12), a second visually simulated illustration of the process used by system
10 in carrying out the procedures of FIGS. 2-3 is shown.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
13
In the current real estate example, the individual real estate images
240 are received by system 10 in an electronic format (stage 122). Images
240 were captured with a digital camera, but film-based images that have
been scanned to an electronic format, or other similar image types could
also be used. Three of the images (242, 244, and 246) are of neighboring
houses on the same street. The fourth iniage 248 represents the finished
appearance of a house that is under construction on that same street. This
fourth image 248 was taken of another identical home by the same builder
to simulate what the home under construction will look like when it is
finished. The system programmatically identifies the three actual houses
(242, 244, and 246) from the street (stage 124) and the user manually
identifies the fourth 248 image, and a series of boundary locations around
each of the houses is identified (stage 126). Images that the system would
not be able to determine a similarity with other images can be manually
added by a user, such as in the case of the fourth image 248 in the current
example. Alternatively or additionally, all of the images can be manually
identified by a user. Using the information obtained in the identification
steps, system 10 has aligned or warped 250 each of real estate images 240
to a panoramic plane (stage 128). Turning now to FIG. 14 with continued
reference to FIG 13, a panoramic real estate image 252 is generated from a
mosaic 254 of the aligned images 250 (stage 130).
Turning now to FIG. 15, an illustration of an image-based real
estate chart 256 generated by system 10 in accordance with the procedures
of FIGS. 2A-2B and FIG. 4 is shown. Image-based real estate chart 256
includes a copy of panoramic real estate image 252 (stage 142), and also
includes hand-written notes 257, 258, 259, and 260 (chart data) that were
input by a user from an input device 24- (stage 144). While the current
example illustrates handwriting data ori the chart, it will be understood that
data can be handwritten, typed, imported from another system, or be a
combination of these, to name a few non-limiting input method examples.
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
14
The chart data (257, 258, 259, and 260) is displayed in an overlay fashion
on top of the panoramic image included in chart 256 (stage 146). By
overlaying the chart data (257, 258, 259, and 260), the underlying
panoramic image is preserved.
Turning now to FIG. 16, an illustration of graphic-based real estate
chart 262 generated by system 10 in accordance with the procedures of
FIGS. 2A-2B and FIG. 5 is shown. In the current example of a panoramic
real estate image, graphic-based chart 262 includes an outline of the homes,
landscaping, and street (stage 152). Graphic-based real estate chart 262
includes hand-written notes 264, 265, 266, and 268 (chart data) that were
input by a user from an input device 24 (stage 154). It will again be
understood that data can be handwritten, typed, imported from another
system, or be a combination of these, to name a few non-limiting input
method examples. The chart data (264, 265, 266, and 268) is displayed in
an overlay fashion upon (e.g. on top of) the panoramic real estate image in
chart 262 (stage 156) so the underlying image in the chart is preserved. As
mentioned previously, images from other industries could also be used.
In yet another embodiment of the present invention, a method is
disclosed that comprises generating a panoramic image in an electronic
format; generating an image-based chart from the panoramic image;
generating a graphic-based chart from the panoramic image, said graphic-
based chart including a graphical outline of the panoramic image; and
wherein the image-based chart and graphic-based chart are operable to
receive and display data that is input using at least one of a plurality of
input devices.
In another embodiment of the present invention, method is
disclosed that comprises receiving a plurality of images in an electronic
format; identifying at least one object in each of the images; identifying a
series of boundary locations around each of the objects; aligning each of
the images to a panoramic plane; generating a panoramic image from the
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
aligned images; and generating at least one electronic chart that includes a
chart image created from the panoramic image, said chart being operable to
receive data that was input using at least one of a plurality of input
devices,
and said chart being further operable to display the data overlaid on top of
5 the chart image.
In yet another embodiment of the present invention, a system is
disclosed that comprises a business logic routine; a first computer coupled
to the business logic routine, said first computer including at le:ast one of
a
plurality of input devices, said computer further including a display device;
10 and wherein the business logic routine is operable to generate a panoramic
image from a plurality of images, to generate at least one electronic chart
that includes a chart image created from the panoramic image, to display
the at least one chart on the display to a user upon request, anct to receive
data input from the user from the at least one of the input devices, said data
15 input including data to be displayed on the at least one chart.
In yet a further embodiment of the present invention, an apparatus is
disclosed that comprises a device encoded with logic executable by one or
more processors to: generate a panoramic image in an electronic format
from a plurality of images; generate an image-based chart frorrn the
panoramic image; generate a graphic-based chart from the panoramic
image, said graphic-based chart including a graphical outline of the
panoramic image; receive chart data from at least one of a plurality of input
devices; and display the chart data overlaid on top of at least one of the
charts.
All publications, prior applications, and other documents cited
herein are hereby incorporated by reference in their entirety as if each had
been individually incorporated by reference and fully set forth_
A person of ordinary skill in the computer software art will
recognize that the client and/or server arrangements, user interface screen
content, and data layouts could be organized differently to include fewer or
CA 02581754 2007-03-22
WO 2006/039402 PCT/US2005/035024
16
additional options or features than as portrayed in the illustrations and
still
be within the spirit of the invention.
While the invention has been illustrated and described in detail in
the drawings and foregoing description, the same is to be considered as
illustrative and not restrictive in character, it being understood that only
the
preferred embodiment has been shown and described and that all
equivalents, changes, and tnodifications that come within the spirit of the
inventions as described herein and/or by the following claims are desired to
be protected.
