Note: Descriptions are shown in the official language in which they were submitted.
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
SYNCHRONIZED LIGHT SOURCE FOR ROLLING SHUTTER IMAGERS
RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional Patent
Application Serial
No. 61728397, filed November 20, 2012, hereby incorporated by reference in its
entirety as if
fully set forth herein.
BACKGROUND OF THE INVENTION
[0002] 1. Technical Field
[0003] This invention relates generally to the field of medical video
equipment. More
specifically, the invention comprises a synchronized light source for rolling
shutter imagers, i.e.
a control system for an LED light source for use with an endoscopic or similar
camera system
with a CMOS-type imager.
[0004] 2. Description of the Related Art
[0005] There are many types of light sources and related control systems
for endoscopic
video. Many of these use a high-output lamp or incandescent bulb such as metal
halide, quartz-
halogen, or xenon types. These lamps are typically used in a mode of a fixed
luminous intensity,
and the intensity of light transmitted out of the device is controlled by
means of a moving and
variable mechanical aperture, an example being an iris, which blocks some or
all of the light
being generated by the lamp to the receiving fiber optic light guide.
[0006] Control of the intensity of this light is important for various
functional and safety
reasons. Improper light levels can cause under-exposure or over-exposure,
forcing the camera
system to overcompensate in ways that reduce or limit the image quality and
camera
1
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
performance. Safety concerns involving high light transmission can include
skin and tissue
burns, and possible ignition of flammable materials.
[0007] Adoption of high power Light-Emitting Diode (LED) technologies is
becoming
common in many fields and industries, including medical endoscopy, reducing
overall power and
cost while increasing product reliability and service life. LED light output
intensity can be
controlled by varying the amount of electrical current driving the LED device.
This method has
several drawbacks including inefficiency, a practical minimum for the lower
end of light output,
and a tendency for the color, or output wavelength(s), of the LED lamp to
drift with intensity.
Being a solid state device, it is also common practice to control overall LED
light output in a
switched Pulse-Width Modulation (PWM) fashion. For the human eye, film
cameras, and some
video cameras, this pulsed light is effectively integrated into an "average"
that when applied at
an appropriate frequency can be virtually indistinguishable from a constant
light source. For
human vision persistence, this frequency is typically about 30 pulses per
second.
[0008] The PWM method of light intensity control works well with video
cameras with
frame-transfer imagers such as CCDs (Charge-Coupled Devices), so long as the
switching
frequency of the light is equal or greater than the camera's rate of frame
capture, typically 60
exposures per second for a video camera. It is convenient to use an integer
multiple of the frame
rate, such as double, for the PWM switching frequency. It is also beneficial
to synchronize the
light source with the camera's frame rate to avoid a frequency mismatch with
can result in a
beating, flickering or "strobing" image.
[0009] However, with the recent adoption of CMOS (Complementary Metal
Oxide
Semiconductor) imagers into medical endoscopy, PWM-controlled LED light
sources present a
challenge. Specifically, the challenge relates to CMOS imagers with a "rolling-
shutter"-type
2
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
exposure architecture, the most common type, as these are not frame-transfer
devices. Instead,
each line of the raster image is exposed in a cascading overlapped sequence,
with lines being
read out while other lines are exposing. The exposure of one line will thusly
never start and stop
at the same time as another line, even though the resultant time duration is
same, and their
exposures will overlap one another in time.
[0010] Thus, traditional frame-rate based PWM control is unsuitable, as
individual lines
or groups of lines may have a significantly different light exposure than
other lines, creating
undesirable regions of differing exposure within the image. The number of
different regions of
exposure is equal to twice the relative PWM frequency, and the complementary
size of the light
and dark regions being directly proportional to the PWM duty cycle. A PWM
system not
synchronous to the imager frame rate would additionally cause a "roll" of this
effect, where the
output video would have these regions in different places of the current image
frame than the
subsequent image frame.
[0011] What is desired, therefore, is a method for PWM control of an LED
light source
for use with CMOS imagers that does not produce an undesirable exposure effect
to the video
image.
3
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
SUMMARY OF THE INVENTION
[0012] In a one aspect, the invention resides in a lighting system
adapted for use with a
rolling shutter imager having a horizontal line rate comprising: a lamp; a
drive circuit coupled to
the lamp capable of energizing the lamp for switching the lamp on and off; and
a lamp control
circuit coupled to the drive circuit with a means for receiving a line timing
signal from an
imaging system, the lamp control circuit synchronously energizing the lamp
with the line timing
signal via the drive circuit, the line timing signal being based upon the
horizontal line rate of the
imager.
[0013] In another aspect, the invention resides in an illumination
control system for
passing timing information to a lighting system from an imaging system
utilizing a rolling
shutter imager operating at a horizontal line rate, comprising: an electronic
connector; and a
digital electronic signal carried by the electronic connector wherein digital
pulses are timed to be
synchronous with the horizontal line rate of the imager.
[0014] In yet another aspect, the invention resides in an illumination
control system for
passing timing information to a lighting system from an imaging system
utilizing a rolling shutter
imager operating at a horizontal line rate and a pixel clock rate, comprising:
an electronic connector;
and a digital electronic signal carried by connector wherein the signal is a
clock derived from the
pixel clock rate, wherein the lighting system utilizes foreknowledge of the
horizontal line rate timing
of the imager to derive the horizontal line rate from the clock signal.
4
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG. 1 is a block diagram of a presently preferred embodiment of
the present
invention; and
[0017] FIG. 2 is a timing diagram illustrating various timing
relationships of the signals
in the embodiment of FIG. 1.
[0018] FIG. 3 is a block diagram of an illumination control system for
passing timing
information from an imaging system to a lighting system (that may or may not
be integrated
within a common housing or on a common circuit board), in the exemplary
context of an
endoscopic video camera system where the imaging system includes a rolling
shutter imager
operating at a horizontal line rate and where the lighting system receives a
digital electronic
signal having digital pulses that are synchronous with the horizontal line
rate of the rolling
shutter imager; and
[0019] FIG. 4 is a block diagram of an illumination control system for
passing timing
information from an imaging system to a lighting system that is similar to
FIG. 3, where the
imaging system includes a rolling shutter imager operating at a horizontal
line rate and, in more
detail, at a pixel clock rate, and where the lighting system receives a
digital electronic signal
corresponding to a clock signal oscillating at the pixel clock rate and where
the lighting system
contains suitable processing capability to derive the horizontal line rate of
the imager from the
clock signal and other known parameters.
[0020] The invention and its various embodiments can now be better
understood by
turning to the following detailed description of the preferred embodiments
which are presented
as illustrated examples of the invention defined in the claims. It is
expressly understood that the
CA 02949362 2016-11-16
WO 2015/178879
PCT/US2014/038563
invention as defined by the claims may be broader than the illustrated
embodiments described
below.
6
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] FIG.1 shows a block diagram of a presently preferred embodiment of
the
invention. The illustrated embodiment, and others, is based on the
foundational observation that
a "rolling shutter" imager can be said to have a "horizontal line rate" while
operating, the rate at
which entire horizontal lines of raster data are read out. Further, the imager
has a "pixel clock
rate", the rate at which each pixel within a line is individually read or
"clocked" out from the
sensor, this pixel clock rate typically being hundreds to thousands of times
faster than the line
rate. The line rate is directly tied to the sensor's shutter mechanism, and
therefore the exposure,
as the amount of time exposure of a line is changed by increasing or
decreasing the number of
lines between when the line is read out, and when it is to start exposing
again, or ceases being
cleared.
[0022] As shown, the preferred embodiment relates to an endoscopic or
similar video
system that comprises a camera having a Rolling Shutter Imager 103 and a Light
Source 114 that
is pulsed on and off with a PWM signal 111 and associated drive circuitry 112
in order to
strategically illuminate the subject matter that is focused onto the Rolling
Shutter Imager 103. In
the preferred embodiment, the Rolling Shutter Imager 103 is implemented with a
CMOS imager.
However, other imager technologies may use a rolling shutter approach to
exposure. In addition,
the preferred Light Source 114 is comprised of one or more LEDs, but the
preferred and/or
alternative embodiments may be implemented with any suitable light source that
now exists or is
later developed. It should be understood, therefore, that any reference to an
LED is a reference
to any suitable light source.
[0023] In more detail, Camera Timing Logic 101 provides timing signals
102 to the
rolling shutter imager 103 (e.g. a CMOS Imager) which generates an Imager
Video Signal 104.
7
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
Image Processor 105 processes the Imager Video Signal 104, and generates a
Video Output
Signal 106 that drives a Video Output Device 107, and an Intensity Signal 109
that is fed back to
the Lamp Control Logic 110, which indicates whether more or less light is
desired from LED
114 in order to achieve a desired exposure. The Camera Timing Logic 101 also
provides a Line
Timing Signal 108, which is used by the Lamp Control Logic 101 to synchronize
a PWM signal
111 to the Drive Circuitry 112, which in turn outputs the Drive Current Pulses
113 to the light
source 114 (e.g. LED) in linear correlation to the PWM signal 111.
[0024] FIG. 2 shows a typical timing relationship of some signals of the
preferred
embodiment. A line timing signal 201 (e.g. HSYNC), generated by Camera Timing
Logic 101 is
applied to the Lamp Control logic 110 to generate the time-base and
synchronicity of the drive
current pulses 113. An Intensity Signal 202 is applied, which may be an analog
signal (as
shown) or a digital signal. This Intensity signal 202 is interpreted by the
Lamp Control Logic
110 to determine the relative duty cycle or percent intensity. Finally, the
Lamp Control Logic
110 outputs the PWM signal 203 which is used to drive the Light Source 113,
e.g. to produce an
LED emitter current with LED Drive Current Pulses, which has timing based upon
the timing of
HSYNC and a duty cycle based upon the Intensity Signal 202, wherein the more
intensity
indicated by the Intensity Signal 202, the longer the PWM signal 111 remains
in the active, or
current-driving state.
[0025] For more context, the preferred embodiment of FIGS. 1 and 2 may be
implemented in the context of a medical imaging device: e.g. an endoscopic
video camera with a
CMOS imager having a rolling shutter; a light source using an LED for
illumination of the scene
to be imaged by said camera; LED drive circuitry, typically comprising
transistors or similar
current-switching devices; control circuitry with a means of controlling the
average LED light
8
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
output intensity using a PWM technique to control said LED drive circuitry,
and a signal from
the camera to the control circuitry that is synchronous to the rate of line
readout of the CMOS
imager. The control circuitry flashes the LED on and off at a rate based upon
the line frequency
of the CMOS imager exposure system, where one pulse (or plurality of pulses)
happens per unit
time elapsed while a line of the CMOS imager is read and restarts its exposure
before moving on
to the next line. For example, if a standard HD imager with 1920 horizontal
pixels by 1080
vertical lines were to be exposed and read at 60 frames per second, the line
frequency would be
1080 (number of lines per frame) times 60 (number of frames per second) which
is equal to
64,800 lines per second, or approximately one line every 15.432usec. A timing
circuit would
generate a pulse synchronous with this line rate of the CMOS imager, what
would typically be
called a digital horizontal sync pulse, commonly known as HSYNC. It should be
noted that this
HSYNC pulse is not typically the same as the HSYNC signal or timing element of
the output
video of the camera, as rolling shutter imagers typically are not, and in many
cases cannot be,
operated line-synchronously with conventional video transport standards, an
example of which is
the 1080p video format as outlined in SMPTE-274M. The control circuitry would
receive this
imager HSYNC digital pulse from the imaging system, typically by means of an
electrical cable
and electrical connectors, and use its frequency and position in time as the
PWM time base. By
means of the drive circuitry, the control circuitry a pulses the LED current
at this frequency and
position in time with the duty cycle of each drive pulse equivalent to the
desired percent intensity
desired from the LED, as determined by direct user input or calculated by
camera processing.
An illustration of one potential timing implementation can be found in Figure
2.
[0026] As this line-based PWM system operates considerably faster than
the frame-based
PWM of conventional systems, an LED with appropriate on and off response times
is required,
9
CA 02949362 2016-11-16
WO 2015/178879
PCT/US2014/038563
along with drive circuitry capable of driving high LED currents at this rate
with pulse widths of
possibly very short duration. The ratio between the pulse widths of this line-
based method
versus the conventional frame-based method is approximated by the number of
lines in the
imager. For example, a 1% PWM pulse duration for a frame-based system
operating at 60
frames per second would be 0.01 x 1/60 = 160usec, assuming one pulse per
frame, whereas the
equivalent 1% PWM pulse duration for the line-based system described above
would be 0.01 x
1/64,800 = 150nsec, which is a 1:1080 ratio. As frame rates and sensor
resolutions increase,
specialty LEDs and drive circuitry may be required for desired results.
Additionally, the drive
circuitry of a this line-based system would typically consume more power and
dissipate more
heat as opposed to a frame-based system, as it has as to drive many more off-
to-on and on-to-off
transitions per frame, each of which will dissipate heat in the device as it
switches in an analog
fashion between states. For the above example, this would be 2 x 1080 = 2160
transitions per
frame in this line-based system versus 2 or 4 transitions per frame in a frame-
based system. For
this reason, it would be advantageous to pulse the LED only once per line,
instead a plurality of
pulses per line, to keep the dissipated power, mechanical and electrical
requirements, as well as
cost, to a minimum.
[0027]
Figures 3 represents a typical medical endoscopy situation where the Imaging
System 20 and Lighting System 30 are separate units, though it should be noted
that the two
systems may be combined into the same enclosure. Between these two systems is
an
illumination control system that carries the horizontal line rate signal
between the two systems.
This would typically be physically implemented by an electrical connector
present on both
systems, and an electrical cable between. In this system, the signal passed
between the systems
is a digital pulse representation of the horizontal line rate of the imager in
the imaging system.
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
Based on this signal, the lighting system 30 synchronously energizes the
related light source (e.g.
LEDs or other lamps) with the horizontal line rate of the imaging system 20.
[0028] Figure 4 represents a system very similar to one in Figure 3, with
the exception
that the signal passed from the imaging system to the lighting system is a
clock signal, based
upon the pixel clock rate of the imager. As shown, the lighting system 30
includes a suitable
means for deriving the horizontal line rate from the clock signal. In such
case, the lighting
system would require some foreknowledge of the timing of the imager and the
imaging system,
which could be programmed into the lighting system, or passed to the lighting
system from the
imaging system be means of another electrical interface, such as a serial
communications port.
[0029] Another embodiment of the invention would be a generic form of the
aforementioned system, wherein the signal from the camera to the control
circuitry is the video
output of the camera itself, and the control circuitry extracts the PWM time
base from the line
interval of the video signal, which may be of a video standard such as SMPTE
274M. This
embodiment allows for a more generic interface between the camera and the
control circuitry,
such that the two devices may use standard interfaces to achieve the proper
synchronization.
This has the potential advantage of less specialized and dedicated interface,
as it can be done
without direct interface to the imager timing logic. This embodiment is
advantageous when the
light source and camera elements of the system are not contained in the same
unit or enclosure.
For best results this method requires the same time relationship between the
imager line rate and
the output video line rate. These may not always be the exactly the same
duration or
synchronicity in all implementations of video cameras, and therefore is not a
universal solution.
However, in the case where it is, this interface has the additional advantage
of also carrying the
picture level information, as it is the video itself, and therefore the
lighting system would have
11
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
the information required to adjust the brightness of the light automatically,
should this be
desired.
[0030] Another embodiment of the invention would be in the case where
there is a
duration of time where lines of the CMOS imager are not being read out, and
all lines of the
image are being exposed. This is a common practice in a multiple frame rate
imaging system,
such as one that operates at both 50 and 60 frames per second, dependent on
the video standard
of the country that the imaging system is being used in. For example, this
duration where all
lines are being exposed can be 16% of the total frame duration, but may also
be longer or shorter
by design. During these "idle" exposure times, the LED may continue to be
pulsed at the same
frequency that would otherwise correspond to the horizontal line frequency as
if the lines were
being read out at this rate. The LED may also be turned off or on entirely
during this idle time.
The LED may also be turned on for a portion of this time, either in a pulsed
or constant fashion,
for either a fixed or variable percentage of this idle time, which may or may
not correspond to
the duty cycle or frequency used during the non-idle period of the frame.
[0031] Another embodiment of the invention would be the case where the
lamp is
modulated on and off on a line-by-line basis. In other words, the lamp would
be turned on for
single or multiple lines, alternating with being turned off for the subsequent
line or multiple
lines. The advantage of this type of system is that a slower, and therefore
less expensive, drive
circuit may be employed, as the pulses are longer by nature, or where the lamp
itself cannot be
turned on and off at the faster rates of the aforementioned embodiments. This
further has the
advantage of lower frequency radiated and conducted emissions created by the
high power
switching. However, the disadvantage of this type of line modulation is that
to achieve
completely uniform exposure across all lines or regions of the image, the time
of the imager's
12
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
total light exposure can only be increased or decreased by the factor of the
number of lines on,
plus the number of lines off For example, if a power output of 75% is desired
from the lamp, it
may be flashed such that it is on for three lines, and off for one line. What
follows is that the
rolling shutter should be set such that the exposure time is in increments of
four lines, to ensure
that all lines receive a 3:1 ratio of lamp on-time to lamp off-time. Thus, the
fewer lines to create
the power ratio needed, the more increments of shutter are usable for a given
number of
horizontal imager lines. The larger the exposure increment is, the lower the
number of possible
exposure values, and therefore a coarser control of the exposure. This can be
a disadvantage
when smooth shutter operation is desired in an automatic exposure system. The
minimum
exposure increment is two lines, alternating one line on and on line off, to
achieve a 50% light
output from the lamp. Most sensors have an even number of horizontal image
lines, but odd
numbered increments of shutter exposure can be used if they divide evenly into
the total number
of imager lines. A 1080-line system is common for HD, and as the number 1080
has 3 and 5 as
factors, these exposure increments would be feasible.
[0032] The following table illustrates the most practical ratios for this
method, up to a 5-
line exposure increment:
Lines Lines Average Lamp Exposure
On Off Power Increment
1 1 50% 2
1 2 33% 3
2 1 67% 3
1 3 25% 4
3 1 75% 4
2 2 50% 4
4 1 80% 5
13
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
3 2 60% 5
2 3 40% 5
1 4 20% 5
[0033] In the above table, the 2:2 line ratio is of note, as it yields
the same lamp power
result as the 1:1 line ratio, but it flashes the lamp at half the frequency.
Thus, with the on and off
times being slower, an even slower drive circuit or lamp could be utilized, at
the cost of higher
exposure increment.
[0034] Many other embodiments are possible without departing from the
spirit and scope
of the present invention. Therefore, it must be understood that the
illustrated embodiment has
been set forth only for the purposes of example and that it should not be
taken as limiting the
invention as defined by the following claims. For example, notwithstanding the
fact that the
elements of a claim are set forth below in a certain combination, it must be
expressly understood
that the invention includes other combinations of fewer, more or different
elements, which are
disclosed in above even when not initially claimed in such combinations.
[0035] The words used in this specification to describe the invention and
its various
embodiments are to be understood not only in the sense of their commonly
defined meanings,
but to include by special definition in this specification structure, material
or acts beyond the
scope of the commonly defined meanings. Thus if an element can be understood
in the context
of this specification as including more than one meaning, then its use in a
claim must be
understood as being generic to all possible meanings supported by the
specification and by the
word itself
[0036] The definitions of the words or elements of the following claims
are, therefore,
defined in this specification to include not only the combination of elements
which are literally
14
CA 02949362 2016-11-16
WO 2015/178879 PCT/US2014/038563
set forth, but all equivalent structure, material or acts for performing
substantially the same
function in substantially the same way to obtain substantially the same
result. In this sense it is
therefore contemplated that an equivalent substitution of two or more elements
may be made for
any one of the elements in the claims below or that a single element may be
substituted for two
or more elements in a claim. Although elements may be described above as
acting in certain
combinations and even initially claimed as such, it is to be expressly
understood that one or more
elements from a claimed combination can in some cases be excised from the
combination and
that the claimed combination may be directed to a sub-combination or variation
of a sub-
combination.
[0037] Insubstantial changes from the claimed subject matter as viewed by
a person with
ordinary skill in the art, now known or later devised, are expressly
contemplated as being
equivalently within the scope of the claims. Therefore, obvious substitutions
now or later known
to one with ordinary skill in the art are defined to be within the scope of
the defined elements.
[0038] The claims are thus to be understood to include what is
specifically illustrated and
described above, what is conceptually equivalent, what can be obviously
substituted and also
what essentially incorporates the essential idea of the invention.
