Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
CA 02850883 2014-05-01
Image Processing
CROSS-REFERENCE TO RELATED APPLICATIONS
10011 This application claims priority to and benefit of U.S. Provisional
Patent Application entitled
"Mobile Device Identification Application," filed May 1, 2013 and given Ser.
No. 13/874,815 on filing
(pending correction to a provisional application serial number); U.S.
Provisional Patent Application
entitled "Visual Search," filed April 4, 2014 and having Ser. No. 61/975,691;
U.S. Provisional Patent
Application entitled "Visual Search Advertising," filed April 7, 2014 and
having Ser. No. 61/976,494; and
U.S. Provisional Patent Application entitled Image "Processing," filed May 1,
2014 and having Ser. No.
61/987,156. The above provisional patent applications are hereby incorporated
herein by reference.
BACKGROUND
(002) Field of the Invention
[0031 The invention is in the field of Image processing, and more particularly
In the field of
characterizing content of images.
10041 Related Art
(005) It is typically more difficult to extract information from images as
compared to text data.
However, a significant fraction of information is found in Images. The
reliability of automated image
recognition systems is highly dependent on the contents of an image. For
example, optical character
recognition is more reliable than facial recognition. It is a goal of image
recognition to tag an image.
Tagging refers to the identification of tags (words) that characterize the
content of an image. For
example an image of a car may be tagged with the words "car," "Ford Granada,"
or "White 1976 Ford
Granada with broken headlight." These tags include varying amounts of
information and, as such, may
vary in usefulness.
SUMMARY
1
CA 02850883 2014-05-01
[006] Embodiments of the invention include a two pronged approach to tagging
of images. The first
prong is to perform automated image recognition on an image. The automated
image recognition
results in a review of the image. The image review includes one or more tags
Identifying contents of the
image and optionally also a measure of confidence representative of the
reliability of the automated
Image recognition, The second prong in the approach to tagging of images
includes a manual tagging of
the image. Manual tagging includes a person viewing each image, considering
the content of the image,
and manually providing tags representative of the content of the image.
Automated image recognition
has an advantage in that the cost, in time or money, of analyzing each image
can be relatively low.
Manual tagging of images has an advantage of higher accuracy and reliability.
[007] Embodiments of the invention combine both automated image recognition
and manual Image
recognition. In some embodiments automated image recognition is performed
first. The resulting
image review typically includes both one or more tags characterizing the image
and a measure of
confidence in the accuracy of these tags. If the confidence is above a
predetermined threshold, then
these tags are associated with the image and provided as an output of the
tagging process, If the
confidence is below the predetermined threshold, then a manual review of the
image is performed. The
manual review results in additional and/or different tags that characterize
the contents of the image. In
some embodiments, the automated image recognition and the manual review of the
image are
performed in parallel. The manual review is optionally cancelled or aborted if
the automated image
recognition results in one or more tags having a confidence above the
predetermined threshold.
[0081 in some embodiments recognition of an image can be upgraded. Upgrading
of the image
recognition process Includes a request for further or improved tags
representative of the content of the
image. For example, if automated image recognition results in the tags "white
car," an upgrade of this
recognition may result in the tags "white Ford Granada." In some embodiments,
an upgraded review
makes use of an expert human reviewer. For example, the above example may
include the use of a
2
CA 02850883 2014-05-01
human reviewer with an expert knowledge of automobiles. Other examples of
reviewer expertise are
discussed elsewhere herein.
[13091 Various embodiments of the invention include features directed toward
improving the accuracy
of image recognition while also minimizing cost. By way of example, these
features include efficient use
of human reviewers, real-time delivery of image tags, and/or seamless upgrades
of image recognition.
The approaches to image recognition disclosed herein are optionally used to
generate image tags
suitable for performing Internet searches and/or selecting advertisements. For
example, in some
embodiments, image tags are automatically used to perform a Google search
and/or sell advertising
based on Google's Ad Words.
[00101 Various embodiments of the invention include an image processing system
comprising an I/O
configured to communicate an image and image tags over a communication
network; an automatic
Identification interface configured to communicate the image to an automatic
identification system and
to receive a computer generated review of the image from the automatic
Identification system, the
computer generated review including one or more image tags identifying
contents of the image;
destination logic configured to determine a first destination to send the
image to, for a first manual
review of the image by a first human reviewer; image posting logic configured
to post the image to the
destination; review logic configured to receive the a manual review of the
image from the destination
and to receive the computer generated review, the manual review including one
or more image tags
identifying contents of the image; response logic configured to provide the
image tags of the computer
generated review and the image tags of the manual review to the communication
network; memory
configured to store the image; and a microprocessor configured to execute at
least the destination logic.
MOW Various embodiments of the invention include a method of processing an
image, the method
comprising receiving an image from an image source; distributing the image to
an automated image
identification system; receiving a computer generated review from the
automated image identification
3
CA 02850883 2014-05-01
system, the computer generated review including one or more image tags
assigned to the image by the
automated image identification system and a measure of confidence, the measure
of confidence being a
measure of confidence that the image tags assigned to the Image correctly
characterize contents of the
Image; placing the image In an image queue; determining a destination; posting
the image for manual
review to a first destination, the first destination including a display
device of a human image reviewer;
and receiving a manual image review of the image from the destination, the
image review Including one
or more Image tags assigned to the image by the human image reviewer, the one
or more Image tags
characterizing contents of the image.
BRIEF DESCRIPTION OF THE DRAWINGS
100121 FIG. 1 illustrates an image processing system, according to various
embodiments of the
invention.
[0013] FIG. 2 illustrates an image capture screen, according to various
embodiments of the invention.
(00141 FIG. 3 illustrates search results based on an image analysis, according
to various embodiments of
the invention.
(00151 FIG. 4 illustrates methods of processing an image, according to various
embodiments of the
invention.
[00161 FIG. 5 illustrates alternative methods of processing an image,
according to various embodiments
of the Invention.
(00171 FIG. 6 illustrates methods of managing a reviewer pool, according to
various embodiments of
the invention.
100181 FIG. 7 illustrates methods of receiving image tags In real-time,
according to various
embodiments of the invention.
(0019] FIG. 8 Illustrates methods of upgrading art image review, according to
various embodiments of
the invention.
4
CA 02850883 2014-05-01
DETAILED DESCRIPTION
(00201 FIG. 1 illustrates an Image Processing System 110, according to various
embodiments of the
invention. Image Processing System 110 is configured for tagging of images and
may include one or
more distributed computing devices. For example, Image Processing System 110
may include one or
more servers located at geographically different places. Image Processing
System 110 is configured to
communicate via a Network 115. Network 115 can include a wide variety of
communication networks,
such as the Internet and/or a cellular telephone system. Network 115 is
typically configured to
communicate data using standard protocols such as IP/TCP, FTP, etc. The images
processed by Image
Processing System 110 are received from Image Sources 120 (individually
labeled 120A, 120B, etc.).
Image Sources 120 can include computing resources connected to the Internet
and/or personal mobile
computing devices. For example Image Source 120A may be a web server
configured to provide a social
networking website or a photo sharing service. Image Source 120B may be a
smart phone, camera, or
other portable image capture device. Image sources may be identified by a
universal resource locator,
an Internet protocol address, a MAC address, a cellular telephone identifier,
and/or the like. In some
embodiments Image Processing System 110 is configured to receive Images from a
large number of
Image Sources 120.
(00211 Part of the image tagging performed by Image Processing System 110
Includes sending images
to Destinations 125 (individually labeled 125A, 125B, etc.). Destinations 125
are computing devices of
human Image reviewers and are typically geographically remote from Image
Processing System 110.
Destinations 125 include at least a display and data entry devices such as a
touch screen, keyboard
and/or microphone. Destinations 125 may include personal computers, computing
tablets,
smartphones, etc. In some embodiments, Destinations 125 include a (computing)
application
specifically configured to facilitate review of images. This application is
optionally provided to
Destinations 125 from Image Processing System 110. In some embodiments, Image
Processing System
CA 02850883 2014-05-01
110 is configured for human image reviewers to log into a user account from
Destinations 125.
Destinations 125 are typically associated with an individual reviewer and may
be identified by an
internet protocol address, a MAC address, a login session Identifier, cellular
telephone identifier, and/or
the like. In some embodiments, Destinations 125 include an audio to text
converter. Image tagging
data provided by a human Image reviewer at a member of Destinations 125 is
sent to Image Processing
System 110. The image tagging data can include textual image tags, audio data
including verbalized
tags, and/or non-tag information such as upgrade requests or inappropriate
(explicit) material
designations.
100221 Image Processing System 110 includes an I/0 (input/output) 130
configured for communicating
with external systems. I/O 130 can include routers, switches, modems,
firewalls, and/or the like. I/O
130 is configured to receive images from Image Sources 120, to send the images
to Destinations 125, to
receive tagging data from Destinations 125, and optionally to send image tags
to Image Sources 120.
100231 Image Processing System 110 further includes Memory 135. Memory 135
includes hardware
configured for the non-transient storage of data such as images, image tags,
computing instructions, and
other data discussed herein. Memory 135 may include, for example, random
access memory (RAM),
hard drives, optical storage media, and/or the like. Memory 135 is configured
to store specific data, as
described herein, through the use of specific data structures, indexing, file
structures, data access
routines, security protocols, and/or the like.
[00241 Image Processing System 110 further Includes at least one Processor
140. Processor 140 Is a
hardware device such as an electronic microprocessor. Processor 140 is
configured to perform specific
functions through hardware, firmware or loading of software instructions into
registers of Processor
140. Image Processing System 110 optionally includes a plurality of Processor
140. Processor 140 is
configured tb execute the various types of logic discussed herein.
6
CA 02850883 2014-05-01
(0025) Images received by Image Processing System 110 are first stored in an
Image Queue 145. Image
Queue 145 is an ordered list of images pending review, stored in a sorted
list. Images stored in Image
Queue 145 are typically stored in association with image identifiers used to
reference the images and
may have different priorities. For example, images received from a photo
sharing website may have
lower priority than images received from a smartphone. Generally, those images
for which a requester
is waiting to receive image tags representing an image in real-time are given
higher priority relative to
those for which the image tags are used for some other purpose. Image Queue
145 is optionally stored
in Memory 135.
[00261 Within Image Queue 145 images are optionally stored in association with
an image identifier or
index, and other data associated with each image. For example, an image may be
associated with
source data relating to one of Image Sources 120. The source data can include
geographic information
such as global positioning system coordinates, a street and/or city name, a
zip code, and/or the like.
The source data may include an internet protocol address, a universal resource
locator, an account
name, an identifier of a smartphone, and/or the like. Source data can further
Include Information about
a language used on a member of Image Sources 120, a requested priority, a
search request (e.g., an
request to do an Internet search based on image tags resulting from the
image), and/or the like,
(0027) In some embodiments, an image within Image Queue 145 is stored in
association with an
indication of a particular subset of the image, the subset typically including
an item of particular
interest. For example, a requestor of image tags may be interested in
obtaining image tags relating to
the contents of a particular subset of an image. This can occur when an image
includes several objects.
To illustrate, considering an image of a hand with a ring on one of the
fingers, the user may wish to
designate the ring as being a particular area of interest. Some embodiments of
the invention include an
application configured for a user to specify the particular item of interest
by clicking on the object or
7
CA 02850883 2014-05-01
touching the object on a display of Image Source 120B. This specification
typically occurs prior to
sending the image to Image Processing System 110.
[0028] If an image is stored in association with an indication that a
particular subset of the image is of
particular importance, then an Image Marking Logic 147 is optionally used to
place a mark on the image.
The mark being disposed to highlight the particular subset. This mark may be
made by modifying pixels
of the image corresponding to the subset and this mark allows a human image
reviewer to focus on the
marked subset. For example, the image may be marked with a rectangle or circle
prior to the image
being posted to one or more of Destinations 125. In alternative embodiments,
Image Marking Logic 147
is included within an application configured to execute on one or more of
Image Sources 120 or
Destinations 125. Image Marking Logic 147 includes hardware, firmware, and/or
software stored on a
non-transient computer readable medium.
[0029] Under the control of Processor 140, images within Image Queue 145 are
provided to an
Automatic Identification interface 150. The images are provided thus as a
function of their priority and
position In Image Queue 145. Automatic Identification interface 150 is
configured to communicate the
image, and optionally any data associated with the image, to an Automatic
Identification System 152.
Automatic identification Interface 150 is further configured to receive a
computer generated review of
the image from Automatic Identification System 152, the computer generated
review including one or
more image tags identifying contents of the image. In some embodiments,
Automatic Identification
Interface 150 is configured to communicate the image and data via Network 115
in a format appropriate
for an application programming interface (API) of Automatic Identification
System 152. In some
embodiments, Automatic Identification System 152 is included within Image
Processing System 110 and
Automatic Identification Interface 150 includes, for example, a system call
within an operating system or
over a local area network.
8
CA 02850883 2014-05-01
(00301 Automatic Identification System 152 is a computer automated system
configured to review
Images without a need for human input on a per picture basis. The output of
Automatic Identification
System 152 is a computer generated image review (e.g., a review produced
without human input on a
per picture basis.) Rudimentary examples of such systems are known in the art.
See, for example,
Kooaba, Ciarifai, AlchemyAPI and Catchoom. Automatic Identification System 152
is typically configured
to automatically identify objects within a two dimensional image based on
shapes, characters and/or
patterns detected within the image. Automatic Identification System 152 is
optionally configured to
perform optical character recognition and/or barcode interpretation. In some
embodiments, Automatic
Identification System 152 is distinguished from systems of the prior art in
that Automatic Identification
System 152 is configured to provide a computer generated review that is based
on the image subset
Indication(s) and/or image source data, discussed elsewhere herein.
(0031] In addition to one or more image tag(s), a computer generated review
generated by Automatic
Identification System 152 optionally includes a measure of confidence
representative of a confidence
that the one or more image tags correctly identify the contents of the image.
For example, a computer
generated review of an image that is primarily characters or easily
recognizable shapes may have a
greater confidence measure than a computer generated review of an image that
consists of abstract or
ill-defined shapes. Different automated image recognition systems may produce
different confidence
levels for different types of images. Automatic Identification interface 150
and Automatic Identification
System 152 are optional in embodiments in which automatic identification is
performed by a third party.
(0032) Image Processing System 110 further includes a Reviewer Pool 155 and
Reviewer Logic 157
configured to manage the Reviewer Pool 155. Reviewer Pool 155 includes a pool
(e.g., group or set) of
human image reviewers. Each of the human image reviewers is typically
associated with a different
member of Destinations 125. Memory 135 is optionally configured to store
Reviewer Pool 155. In some
embodiments, the human image reviewers included in Reviewer Pool 155 are
classified as "active and
9
CA 02850883 2014-05-01
"inactive," For the purposes of this disclosure, an active human image
reviewer is considered to be one
that is currently providing image tags or has indicated that they are prepared
to provide image tags with
minimal delay. In embodiments that include both active and inactive human
image reviewers, the active
reviewers are those that are provided image for review. The number of active
reviewers may be
moderated in real-time in response to a demand for image reviews. For example,
the classification of a
human image reviewer may be changed from inactive to active based on a number
of unviewed Images
in Image Queue 145. An inactive reviewer is one that is not yet active, that
has let the review of an
image expire, and/or has indicated that they are not available to review
images. Inactive reviewers may
request to become active reviewers. Inactive reviewers who have made such a
request can be
reclassified as active human image reviewers when additional active human
image reviewers are
needed. The determination of which inactive reviewers are reclassified as
active reviewers is optionally
dependent on a reviewer score (discussed elsewhere herein).
100331 Reviewer Logic 157 is configured to manage Reviewer Pool 155. This
management optionally
includes the classification of human image reviewers as active or inactive.
For example, Reviewer Logic
157 may be configured to monitor a time that a human image reviewer takes to
review an image and, If
a predetermined maximum review time (referred to herein as an image expiration
time), changing the
classification of the human image reviewer from active to inactive. In another
example, Reviewer Logic
157 may be configured to calculate a review score for a human image reviewer.
In some embodiments,
the review score is indicative of the completeness, speed and/or accuracy of
image reviews performed
by the particular human image reviewer. The review score can be calculated or
changed based on
review times and occasional test Images. These test Images may be, for example
images placed in Image
Queue 145 that have been previously reviewed by a different human image
reviewer. The review score
may also be a function of monetary costs associated with the human image
reviewer. Reviewer Logic
157 includes hardware, firmware, and/or software stored on a non-transient
computer readable
CA 02850883 2014-05-01
medium. In some embodiments, reviewer scores are manually determined by human
moderators.
These human moderators review Images and the tags assigned to these images by
human image
reviewers. Moderators are optionally sent a statistical sampling of reviewed
images and they assign a
score to the tagging of the images. This score is optionally used in
determining reviewer scores.
[0034] In some embodiments, Reviewer Logic 157 is configured to monitor status
of human Image
reviewers in real-time. For example, Reviewer Logic 157 may be configured to
monitor the entry of
individual words or keystrokes as entered by a reviewer at Destination 125A.
This monitoring can be
used to determine which reviewers are actively reviewing images, which
reviewers have just completed
review of an image, and/or which reviewers have not been providing tag input
for a number of seconds
or minutes. The entry of tag words using an audio device may also be monitored
by Reviewer Logic 157.
[0035] In some embodiments, members of Reviewer Pool 155 are associated with a
specialty in Which
the human image reviewer has expertise or special knowledge in. For example, a
reviewer may be an
expert in automobiles and be associated with that specialty. Other specialties
may include art, plants,
animals, electronics, music, food medical specialties, clothing, clothing
accessories, collectables, etc. As
is discussed elsewhere herein, a specialty of a reviewer may be used to select
that reviewer during an
Initial manual review and/or during a review upgrade.
[0036] The review score and/or specialty associated with a human image
reviewer are optionally used
by Reviewer Logic 157 to determine which inactive reviewer to make active,
when additional active
reviewers are required. Reviewer Logic 157 includes hardware, firmware, and/or
software stored on a
non-transient computer readable medium.
[0037] Image Processing System 110 further includes Destination Logic 160.
Destination Logic 160 is
configured to determine one or more destinations (e.g., Destinations 125) to
send an image to for
manual review. Each of Destinations 125 is associated with a respective human
image reviewer of
Reviewer Pool 155. The determinations made by Destination Logic 160 are
optionally based on
11
CA 02850883 2014-05-01
characteristics of the human image reviewer at the determined destination. The
destination may be a
computing device, smartphone, tablet computer, personal computer, etc. of the
human image reviewer,
In some embodiments, the destination is a browser from which the reviewer has
logged into Image
Processing System 110. In some embodiments, determining the destination
includes determining an
MAC address, session Identifier, Internet protocol and/or universal resource
locator of one of
Destinations 125. Destination Logic 160 includes hardware, firmware and/or
software stored on a non-
transient computer readable medium.
[0038] Typically, Destination Logic 160 Is configured to determine
Destinations 125 associated with
active rather than inactive human image reviewers as determined by Reviewer
Logic 157. Destination
Logic 160 is also typically configured to determine Destinations 125 based on
review scores of reviewers.
For example, those reviewers having higher reviewer scores may be selected for
higher priority reviews
. relative to reviewers having lower reviewer scores. Thus, the determination
of a member of
Destinations 125 can be based on both reviewer scores and image review
priority.
100391 in some embodiments, Destination Logic 160 is configured to determine
one or more members
of Destinations 125 based on the real-time monitoring of the associated
reviewers' input activity. As
discussed elsewhere herein, this monitoring may be performed by Reviewer Logic
157 and can include
detection of individual words or keystrokes entered by a human image reviewer.
In some
embodiments, Destination Logic 160 is configured to favor selecting
Destination 125A at which a human
image reviewer has just completed a review of an image relative to Destination
1258 at which a human
image reviewer is currently typing image tags on a keyboard.
[0040] In some embodiments, Destination Logic 160 is configured to use image
tags received via
Automatic Identification System 152 to determine one or more members of
Destinations 125. For
example, if an image tag of "car" is received via Automatic identification
interface 150 then Destination
12
CA 02850883 2014-05-01
Logic 160 can use this information to select a member of Destinations 125
associated with a human
image reviewer that has a specialty in automobiles.
[0041] The value of an image review may also be considered in the selection of
a destination for
manual review. For example, an image review of high value may lead to the
determination of a
destination associated with a human image reviewer having a relatively high
review score, while an
image review of lower value may lead to the determination of a destination
associated with a human
image reviewer having a relatively lower review score. In some embodiments,
for some image reviews,
Destination Logic 160 is configured to select among Destinations 125 so as to
minimize a time required
to review an image, e.g., to minimize a time until the image tags of the
manual review are provided to
Network 115.
[00421 Destination Logic 160 is optionally configured to determine multiple
destinations for a single
image. For example, a first destination may be selected and then, following an
upgrade request, a
second destination may be determined. The upgrade request may come from the
Image Source 120A or
from a human Image reviewer associated with the first destination. In some
embodiments, Destination
Logic 160 is configured to determine multiple destinations, to which the image
will be posted to in
parallel. For example, two, three or more destinations, each associated with a
different human image
reviewer, may be determined and the same image posted to all determined
destinations in parallel. As
used In this context, "in parallel" means that the image is posted to at least
a second destination before
any part of a review is received from the first destination.
[0043] In various embodiments, there are a variety of reasons that two or more
destinations may be
determined by Destination Logic 160. For example, a request for an upgraded
review may require a
human image reviewer having a particular specialty. Referring to the
automotive example, an image
that is first tagged with the tag "white car" may result in an upgrade quest
for more information.
Destination Logic 160 may be configured to then select a destination
associated with a human image
13
CA 02850883 2014-05-01
reviewer have a specialty in automobiles, e.g., a reviewer who can provide the
tags "1976 Ford
Granada."
10044] Another instance that may require a second destination occurs when the
manual review of an
image takes too long. Typically, the tagging of an image should occur within
an allotted time period or
the review is considered to expire. The allotted time period is optionally a
function of the priority of the
image review. Those reviews that are intended to occur in real-time may have a
shorter time period
relative to lower priority reviews. If the review of an image expires, Image
Processing System 110 is
optionally configured to provide the Image to an additional human image
reviewer associated with a
destination determined by Destination Logic 160.
100451 Another instance that may require a second destination occurs when a
first human reviewer
makes an upgrade request. For example, the request to upgrade the review
resulting in a tag of "cars'
may come from the human image reviewer that provided the tag "car." While this
example is simplistic,
other examples may include images of more esoteric subject matter such as
packaged integrated
circuits.
(0046) Image Processing System 110 further Includes Image Posting Logic 165
configured to post
images for manual review to Destinations 125 determined by Destination Logic
160. Posting typically
includes communicating the images to one or more Destinations 125 via Network
115. In various
embodiments, Image Posting Logic 165 is further configured to provide
information associated with the
Image to Destinations 125. For example, Image Posting Logic 165 may post,
along with the image, an
indication of a subset of the image (e.g., subset identification), an image
marked by Image Marking Logic
147, information identifying a source of the image (e.g., source data
discussed elsewhere herein), a
priority of the review of the image, an image expiration period, location
information associated with the
image, and/or the like. As discussed elsewhere herein, source data can
includes a universal resource
14
CA 02850883 2014-05-01
locator, global positioning coordinates, longitude and latitude, an account
identifier, an Internet
protocol address, a social account, an photo sharing account, and/or the like.
[0047) In some embodiments Image Posting Logic 165 is configured to provide an
Image for manual
review to more than one of Destinations 125 at the approximately the same
time. For example, an
image may be provided to Destination 125A and Destination 125B in parallel.
"Parallel delivery" means,
for example, that the Image Is provided to both Destinations 125A and 1258
before tagging information
is received back from either of these Destinations 125.
[00481 In some embodiments, Image Posting Logic 165 Is configured to provide
an Image for manual
review to one or more of Destinations 125 prior to receiving image tags from
Automatic Identification
System 152. Alternatively, in some embodiments, Image Posting Logic 165 is
configured to wait until a
computer generated review for the image is received from Automatic
Identification System 152, prior to
posting the image to one or more of Destinations 125. In these embodiments,
the computer generated
review (including image tags) is optionally also posted to the one or more of
Destinations 125 in
association with the image.
(00491 Image Posting Logic 165 Is optionally configured to post identifiers of
images along with the
images. Image Posting Logic 165 includes hardware, firmware and/or software
stored on a non-
transient computer readable medium.
[00501 Image Processing System 110 further includes Review Logic 170
configured to manage the
manual and automated reviews of images. This management includes monitoring
progress of reviews,
receiving reviews from Automatic Identification System 152 and/or Destinations
125. The received
reviews include image tags as discussed elsewhere herein. In some embodiments,
Review Logic 170 Is
configured to control posting of the image to one of Destinations 125 based on
a measure of
confidence. The measure of confidence being representative of a confidence
that one or more Image
tags already received are correct. These one or more image tags may be
received from Automatic
CA 02850883 2014-05-01
Identification System 152 and/or one of Destinations 125. For example, In some
embodiments if the
confidence of an image review by Automatic Identification System 152 is
greater than a predetermined
threshold, then Review Logic 170 may determine that manual review of the image
is not necessary. The
predetermined threshold can be a function of the value of the image review, of
the priority of the image
review, of the number and quality of the available Destinations 125, and/or
the like. Review Logic 170
includes hardware, firmware, and/or software stored on a non-transient
computer readable medium.
[00511 In some embodiments, if an image was sent to Automatic Identification
System 152 in parallel
with being sent to one or more of Destinations 125, then the receipt of a
review from Automatic
Identification System 152 having a confidence above a predetermined threshold
may result in
cancellation of the manual review at the one or more of Destinations 125 by
Review Logic 170.
Likewise, If an image Is sent to multiple Destinations 125 in parallel, and an
Image review is received
from a first of these Destinations 125, then Review Logic 170 is optionally
configured to cancel the
review requests for the image at the other Destinations 125. In some
embodiments, Review Logic 170 is
configured to cancel the review request at the other Destinations 125 once a
keystroke or word is
received from the first of the Destinations 125.
10052J In some embodiments Review Logic 170 is configured to monitor activity
of a human image
reviewer in real-time. This monitoring can include receiving review inputs
from Destinations 125 on a
word by word or individual keystroke basis. As discussed elsewhere herein, the
words and/or
keystrokes are optionally passed on to one of Image Sources 120 as they are
received by Review Logic
170. The monitoring of a manual reviewer's activity can be used to determine
when the review of an
Image expires and/or the progress in completing a manual image review. The
status of a human image
reviewer may be provided by Review Logic 170 to Reviewer Logic 157 in real-
time. Using this status,
Reviewer Logic 157 may change the status of the reviewer from active to
inactive, adjust a stored review
score of the reviewer, establish or change a specialty for the reviewer,
and/or the like.
16
CA 02850883 2014-05-01
(00531 In some embodiments Review Logic 170 is configured to control posting
of images to
Destinations 125 by receiving measures of confidence (e.g., of the accuracy of
image reviews) and
sending responsive signals to Destination Logic 160 and/or Image Posting Logic
165. As such, Review
Logic 170 can be configured to control posting of an image to one or more of
Destinations 125 based on
a measure of confidence. The measure of confidence being representative of a
confidence that one or
more image tags correctly identify the contents of the image. In some
embodiments, Review Logic 170
is configured to receive reviews from manual image reviewers that include
information other than
Image tags. For example, Review Logic 170 may receive an upgrade request from
a human image
reviewer and cause an upgraded image review to be requested. Review Logic 170
is optionally
configured to process other non-tag information received in a manual or
computer generated review.
This information can include identification of the image as being improper
(e.g., obscene), identification
of the image as containing no identifiable objects, identification of the
image as having been sent to a
reviewer of the wrong specialty, and/or the like.
[0054] In some embodiments, Review Logic 170 is configured to adjust the
confidence of an image
review by comparing image reviews of the same image from multiple sources.
These image reviews may
all be computer generated, all be manual reviews, or include at least one
computer generated review
and at least one manual review.
(00551 In some embodiments, Review Logic 170 is configured to provide Image
tags received as part of
a first (computer generated or manual) review and to provide the received
image tags to a human image
reviewer at Destinations 1258. An agent (e.g., a browser or special purpose
application) executing on
Destination 1258 is optionally configured to provide the Image tags of the
first review to a display of
Destination 1258. In this manner, the human image reviewer at Destination 125B
can edit (add to,
delete and/or replace) the image tags of the first review. For example, Image
tags received from
Destination 125A may be provided to Destination 1258 for modification.
17
CA 02850883 2014-05-01
[0056] In some embodiments, Review Logic 170 is configured to calculate review
scores based on the
results of image reviews received from Destinations 125, the time taken for
these image reviews, and
the accuracy of these image reviews.
[00571 In some embodiments Review Logic 170 is configured to provide image
reviews to a source of
the image, e.g., one of Image Sources 120, using a Response Logic 175. The
image reviews may be
provided when the image review is complete, on a character by character basis,
or on a word by word
basis, When provided on a character by character basis or a word by word
basis, the image tags are
optionally provided to the source of the image as the characters or words are
received from a human
image reviewer. Optionally Response Logic 175 is configured to provide the
image review via Network
115.
[00581 Image reviews are not necessarily returned to one of Image Sources 120.
For example, if Image
Source 120A is a photo sharing service or a social networking website, image
reviews of images from
Image Source 120A may be stored in association with an account on the photo
sharing service or the
social networking website. This storage can be in Memory 135 or at a location
external to Image
Processing System 110, such as at a webserver hosting the website.
[0059] In some embodiments, Response Logic 175 is configured to execute a
search based on image
tags received in a computer generated and/or manual Image review. The results
of this search can be
provided to a source of the image, e.g., Image Source 120A or 120B. For
example, in some
embodiments a user uses a smartphone to create an Image with a camera of Image
Source 120A. The
image is provided to Image Processing system 110 which generates an image
review of the image using
Automatic Identification System 152 and Destination 125A. The image review
includes image tags that
are then automatically used to perform an internet search (e.g., a google or
yahoo search) on the image
tags. The results of this internet search are then provided to the user's
smartphone.
18
CA 02850883 2014-05-01
[00601 In some embodiments, Response Logic 175 Is configured to provide image
tags of a computer
generated and/or manual review to an Advertising System 180. Advertising
System 180 is configured to
select advertisements based on the image tags. The selected advertisements are
optionally provided to
the source of the image used to generate the image tags. For example, Response
Logic 175 may provide
the tags "1976 Ford Granada with broken headlight" to Advertising System 180
and, In response,
Advertising System 180 may select advertisements for replacement headlights.
If the source of the
image used to generate these tags is a website, the advertisements may be
displayed on the website.
Specifically, if the source of the image is an account on a photo sharing or
social networking website,
then the advertisements may be displayed on that account. Advertising System
180 is optionally
Included in Image Processing System 110. Advertising System 180 is optionally
configured to take bids
for providing advertising in response to specific tags. Advertising System 180
optionally includes
Google's Adwords.
100611 Image Processing System 110 optionally further includes Content
Processing Logic 185
configured to extract images for tagging from members of Image Sources 120.
Content Processing Logic
185 is configured to parse webpages including images and optionally text, and
extract images from
these webpages for tagging. The resulting image tags may then be provided to
Advertising System 180
for selection of advertisements that can be placed on the webpage from which
the image was extracted.
In some embodiments, Content Processing Logic 185 is configured to emulate
browser functions in
order to load images that would normally be displayed on a webpage. These
images may be displayed
on a webpage associated with a specific account, a social networking site, a
photo sharing site, a
blogging site, a news site, a dating site, a sports site, and/or the like.
Content Processing Logic 185 is
optionally configured to parse metadata tags in order to identify Images.
100621 FIG. 2 illustrates an Image Capture Screen 210, according to various
embodiments of the
Invention. Image Capture Screen 210 as illustrated is generated by, for
example, an application
19
CA 02850883 2014-05-01
executing on a smartphone or other Image Source 120. Image Capture Screen 210
is includes features
configured to capture an image, mark a specific area of interest, and receive
image tags. Specifically,
Image Capture Screen 210 includes a Shutter Button 220 configured to take a
picture. Once the picture
is taken it is optionally automatically sent via Network 115 to Image
Processing System 110 for tagging.
Image Capture Screen 210 optionally further includes a Rectangle 230
configured to highlight a point of
interest within the image. Rectangle 230 is controllable (e.g., movable) by
selecting and/or dragging on
the screen using a user input device. On a typical smartphone this user input
device may include a touch
screen responsive to a finger touch. As described elsewhere herein, the
point/region of interest may be
provided to Image Processing System 110 in association with an image to be
tagged.
100631 Image Capture Screen 210 further includes a Field 240 showing a
previously captured Image and
resulting image tags. In the example, show the previously captured image
includes the same white cup
without the Rectangle 230 and the image tags Include "White Starbucks Coffee
Cup." Also shown is text
stating "Slide for options."
(0064) FIG. 3 Illustrates search results based on an image analysis, according
to various embodiments of
the invention. These results are optionally displayed automatically or in
response to selecting the "Slide
for options" input shown in FIG. 2. They may be generated by automatically
executing an internet
search on the image tags. Illustrated in FIG. 3 are a Sponsored Advertisement
310, Related Images 320
and other search results 330. The search results are optionally generated
using Advertising System 180
and image tags generated using Image Processing System 110. A user may of the
option of reviewing
previously tagged images. This history can be stored on Image Source 120A or
in Memory 135.
(0065] FIG. 4 illustrates methods of processing an image, according to various
embodiments of the
invention. In these methods an image is received. The image is provided to
both Automatic
Identification System 152 and at least one of Destinations 125. As a result,
both computer generated
and manual image reviews are produced. The methods illustrated in FIG. 4 are
optionally performed
CA 02850883 2014-05-01
using embodiments of the system illustrated in FIG. L The method steps
illustrated in FIGs. 4-5 may be
performed in a variety of alternative orders.
[0066] In a Receive Image Step 410 and image is received by Image Processing
System 110. The image
is optionally received from one of Image Sources 120 via Network 115. The
image may be in a standard
format such as TIF, JPG, PNG, GIF, etc. The image may be one of a sequence of
images that form an
Image sequence of a video. The image may have been captured by a user using a
camera. The image
may have been captured by a user from a movie or television show. In some
embodiments Receive
Image Step 410 includes a user using an Image capture application to capture
the image and
communicate the image to Image Processing System 110. This application may be
disposed within a
camera, television, video display device, multimedia device, and/or the like.
Receive Image Step 410 is
optionally facilitate using Content Processing Logic 185,
100671 In an optional Receive Subset Identification Step 415, data identifying
one or more subsets of
the image is received by Image Processing System 110. Typically, the one or
more subsets Include a set
of Image pixels in which an item of particular Interest is located. The one or
more subsets may be
identified by pixel locations, screen coordinates, areas, and/or points on the
received image. In some
embodiments, the subsets are selected by a user using a touch screen or cursor
of one of Image Sources
120.
[0068] In an optional Receive Source Data Step 420, source data regarding the
source of the image,
received in Receive Image Step 410, Is received by Image Processing System
110. As discussed
elsewhere herein, the source data can include geographic information, an
Internet protocol address, a
universal resource locator, an account name, an identifier of a smartphone,
information about a
language used on a member of Image Sources 120, a search request, user account
information, and/or
the like. In some embodiments, source data is automatically sent by an
application/agent running on
21
CA 02850883 2014-05-01
Image Source 120. For example, global positioning system coordinates may
automatically be generated
on a smartphone and provided to Image Processing System 100.
100691 In an optional Receive Analysis Priority Step 425 a priority for the
tagging of the Image, received
in Receive Image Step 410, is received within Image Processing System 110. In
some embodiments, the
priority is manually entered by a user of Image Source 120A. In some
embodiments, the priority is
dependent on an amount paid for the review of the image. In some embodiments,
the priority is
dependent on a type of Image Sources 120A. For example, images received from a
static website may
automatically be given a lower priority relative to images received from a
handheld mobile device. An
image whose source is identified by a universal resource locator may be given
a lower priority relative to
images whose source Is identified by a mobile telephone number. As such, the
priority is optionally
derived from the source data received in Receive Source Data Step 420.
[00701 The image and data received in Steps 410-425 are optionally received
together and optionally
stored in Memory 135.
[0071] In a Distribute Image Step 430, the image, and optionally any
associated data received in Steps
415-425, Is distributed to Automatic Identification System 152 via Automatic
Identification Interface
150. This distribution may be internal to Image Processing System 110 or via
Network 115.
[00721 In a Receive Automated Response Step 435, a computer generated Image
review Is received
from Automatic Identification System 152. The computer generated image review
includes one or more
image tags assigned to the image by Automatic Identification System 152. The
computer generated
image review also includes a measure of confidence. The measure of confidence
is a measure of
confidence that the image tags assigned to the image correctly characterize
contents of the Image. For
example, an image including primarily easily recognizable characters may
receive a higher measure of
confidence relative to an image of abstract shapes.
22
CA 02850883 2014-05-01
[0073] In an Optional Determine Confidence Step 440, the measure of confidence
Included In the image
review is compared with one or more predetermined levels. The predetermined
levels are optionally a
function of the priority of the image review, a price of the image review, a
source of the image, and/or
the like. In an Optional Confident? Step 445 the process proceeds to an
optional Perform Search Step
450 if the confidence of the computer generated image review is above the
predetermined threshold(s)
and proceeds to a Queue Image Step 460 if the confidence of the computer
generated image Is below
the predetermined threshold(s). Determine Confidence Step 440 is optionally
performed using Review
Logic 170.
[00741 in Perform Search Step 450, the image tags assigned to an image are
used to perform a search.
For example, the image tag "Ford car" may be used to automatically perform a
googie search using the
words "Ford" and "car."
[0075] In a Provide Results Step 455, the image tags assigned to the image and
optionally the results of
a search performed In Perform Search Step 450 are provided to a requester of
the image review. For
example, if the image was received from Image Source 120A and Image Source
120A is a smartphone,
then the image tags and search results are typically provided to the
smartphone. If the Image was
received from a member of Image Sources 120, such as a website, that the image
tags and optional
search results may be provided to a host of the website, to a third party, to
Advertising System 180,
and/or the like. In some embodiments, the image tags are automatically added
to the website such that
the image tags are searchable, e.g., can be searched on to find the reviewed
image.
[00761 In Queue Image Step 460, the image is placed in Image Queue 145. This
placement optionally
includes marking a subset of the image using Image Marking Logic 147. As
described elsewhere herein,
the marking is typically configured to identify objects of particular interest
in the image. Advancement
of the image in Image Queue 145 may be dependent on the image's review
priority, the source of the
23
CA 02850883 2014-05-01
image, available human image reviewers, the measure of confidence of the
computer generated review
of the image, and/or the like.
[00771 In a Determine Destination Step 465 one or more members of Destinations
125 are determined
for the manual review of the image. The determination of a destination is
optionally based on image
tags included in a computer generated image review received from Automatic
Identification System 152;
optionally based on specialties of human image reviewers at different
Destinations 120; optionally
based on review scores of these human image reviewers, and/or based on other
criteria discussed
herein.
[00781 In a Post Image Step 470, the image is posted to at least one of the
Destinations 125 determined
in Determine Destination Step 465. In some embodiments, Post Image Step 470
includes posting the
image to more than one of Destinations 125 in parallel. The image is
optionally posted via Network 115
and is optionally posted along with a mark highlighting a subset of the image,
source data for the image,
a time before review expiration for the image, image tags for the image
received from Automatic
Identification System 152, and/or the like.
[00791 In a Receive Review Step 475, a manual review of the image is received
from one or more of the
determined Destination(s) 125. The manual image review may include one or more
image tags assigned
to the image by a human image reviewer. The one or more image tags are
representative of the content
of the image. The manual review may also include an upgrade request, an
indication that the image is
unreviewable, an indication that the image is Improper, an indication that the
review expired, and/or
the like.
[0080] In an image Tagged? Step 480 the progress of the method is dependent on
whether image tags
were received in Receive Review Step 475. If image tags characterizing the
content of the image were
received then the method optionally proceeds with Perform Search Step 450 and
Provide Results Step
455. In these steps the image tags included in the manual image review and
optionally the computer
24
CA 02850883 2014-05-01
generated image review are used. Use of the image tags in the computer
generated image review may
be dependent on the confidence measure of this review.
[0081) Steps 460-475 are optional If In Step 445 the confidence measure is
found to be above the
predetermined threshold(s).
(0082] In an optional Upgrade? Step 485 the progress of the method is
dependent on whether an
upgrade request has been received. If such a request has been received then
the method proceeds to
Determine Destination Step 465 wherein a second/different member of
Destinations 125 is determined.
The determination may depend on image tags received in the manual image review
received In Receive
Review Step 475. The upgrade request may be received from a human image
reviewer or from a
requester of the image review (from Image Source 120A or 1208, etc.). The
upgrade request may be
received after the requestor has had a chance to review the image tags
provided in Provide Results Step
455. For example, the requestor may first receive image tags consisting of
"white car" and then request
a review upgrade because they desire further information. The review upgrade
may result in the image
being provided to a human image reviewer with a specialty in automobiles. This
human image review
can add to the existing image tags to produce "white car, 1976 Ford Granada."
In some embodiments,
the requester can add source data indicating a subset of the image when
requesting a review upgrade.
For example, the reviewer may wish to indicate particular interest in a broken
headlight. This serves to
direct the human image reviewers attention to this feature of the image,
produce tags that include
"broken headlight," and result in a search (Perform Search Step 450), directed
toward broken headlights
for a 1976 Ford Granada.
(00831 in some embodiments, upgrade request are generate automatically by
Review Logic 170. For
example if an image review appears too brief, e.g., just "car,' then Review
Lotic 170 may automatically
initiate a review upgrade. In some embodiments, the automatic generation of
upgrade requests is
based on the presence of keywords within a manual image review. For example,
certain review
CA 02850883 2014-05-01
specialties are associated with lists of keywords. In some embodiments, when
one of these keywords
are received in a manual image review and an automated review upgrade is
initiated. The review
upgrade preferably includes a human Image reviewer having a specialty
associated with the received
keyword. In a specific example, one specialty includes "automobile?' and is
associated with the
keywords "car," "truck," "van," "convertible," and "Ford." When one of these
keywords is received In a
manual image review, Review Logic 170 checks with Review Logic 157 to
determine if a human image
reviewer having a specialty in "automobiles" is currently active. If so, then
an automatic upgrade is
initiated and the image is sent to the Destination 1258 of the reviewer having
the "automobiles"
specialty.
[0084) If no upgrade requests are made, then In an End Step 490, the process
Is completed.
[0085) FIG. 5 illustrates alternative methods of processing an Image,
according to various embodiments
of the invention. In these methods, at least some of Steps 430-445 are
performed In parallel with at
least some of Steps 460-475. The manual image review is in Steps 460-475 may
be begun before the
computer generated review of Steps 430-445 is complete, thus, the manual image
review Is started
before the confidence measure of the computer generated review is known. If,
in Confident? Step 445,
the confidence measure is found to be above the predetermined threshold(s),
then Steps 460-475 are
optionally aborted.
100861 Referring to FIG. 6, various embodiments of the invention include
methods of managing of a
reviewer pool. The methods including receiving an Image for review; selecting
(determining) a first
member of Destinations 125 using Destination Logic 160; posting the received
image to the first
member of Destinations 125; using Review Logic 170 to monitor progress of a
manual image review of
the image at the first member of Destinations 125, the monitoring indicating
that the review has taken
more than a predetermined review time; changing the status of a human Image
reviewer associated
with the first member of Destinations 125 from active to inactive in response
to the review taking more
26
CA 02850883 2014-05-01
than the predetermined review time; selecting a second member of Destinations
125; posting the image
to the second member of Destinations; and receiving a manual image review from
the second member
of Destinations 125.
[0087] Referring to FIG. 7, various embodiments of the invention include
methods of providing real-
time feedback of a manual image review. The methods comprising posting an
image to a member of
Destinations 125; detecting a first key stroke (or audio) entered by a human
image reviewer at the
member of Destinations 125; detecting completion of a first word of an image
tag at the member of
Destinations 125; delivering the first word to a source of the image;
following delivery of the first word
detecting completion of a second word of the image tag at the member of
Destinations 125; delivering
the second word to the source of the image; detection completion of the image
tag (e.g., by detecting a
carriage return); and associating the image tag with the image.
10088) Referring to FIG. 8, various embodiments of the invention include
methods of managing an
upgrade of an image review. The methods comprising receiving an image;
selecting a first member of
Destinations 125; posting the image to the first member of Destinations 125;
receiving a first manual
image review from the first member of Destinations 125; detecting an upgrade
request (the upgrade
request being received from a source of the image or from the first member of
Destinations 125);
selecting a second member of Destinations 125; posting the image to the second
member of
Destinations 125; optionally posting the first manual image review to the
second member of
Destinations 125 In association with the Image; receiving a second manual
image review from the
second member of Destinations 125; and delivering the second manual image
review and optionally the
first manual image review to a source of the Image.
100891 Several embodiments are specifically illustrated and/or described
herein. However, it will be
appreciated that modifications and variations are covered by the above
teachings and within the scope
of the appended claims without departing from the spirit and intended scope
thereof. For example, the
27
CA 02850883 2014-05-01
Images discussed herein are optionally part of a video sequence of a video.
Human image reviews may
provide image tags at Destinations 125 using audio input. The audio input can
be converted to text in
real-time using audio to text conversion logic disposed on Destinations 125
and/or Image Processing
System 110. Image tags are optionally processed by spellcheck logic. As used
herein, the term "Real-
time" means without unnecessary delay such that a user can easily wait for
completion.
10090] The embodiments discussed herein are illustrative of the present
invention. As these
embodiments of the present invention are described with reference to
illustrations, various
modifications or adaptations of the methods and or specific structures
described may become apparent
to those skilled in the art. All such modifications, adaptations, or
variations that rely upon the teachings
of the present invention, and through which these teachings have advanced the
art, are considered to
be within the spirit and scope of the present invention. Hence, these
descriptions and drawings should
not be considered in a limiting sense, as it is understood that the present
invention is in no way limited
to only the embodiments illustrated.
10091] Computing systems referred to herein, (e.g., Image Processing System
110, Images Sources 120
and Destinations 125), can comprise an integrated circuit, a microprocessor, a
personal computer, a
server, a distributed computing system, a communication device, a network
device, or the like, and
various combinations of the same. A computing system may also comprise
volatile and/or non-volatile
memory such as random access memory (RAM), dynamic random access memory
(DRAM), static
random access memory (SRAM), magnetic media, optical media, nano-media, a hard
drive, a compact
disk, a digital versatile disc (DVD), and/or other devices configured for
storing analog or digital
information, such as in a database. The various examples of logic noted above
can comprise hardware,
firmware, or software stored on a computer-readable medium, or combinations
thereof. A computer-
readable medium, as used herein, expressly excludes paper. Computer-
implemented steps of the
methods noted herein can comprise a set of instructions stored on a computer -
readable medium that
28
CA 02850883 2014-05-01
when executed cause the computing system to perform the steps. A computing
system programmed to
perform particular functions pursuant to instructions from program software is
a special purpose
computing system for performing those particular functions. Data that is
manipulated by a special
purpose computing system while performing those particular functions is at
least electronically saved in
buffers of the computing system, physically changing the special purpose
computing system from one
state to the next with each change to the stored data. The logic discussed
herein may include hardware,
firmware and/or software stored on a computer readable medium. This logic may
be implemented in
an electronic device to produce a special purpose computing system.
29
