Note: Descriptions are shown in the official language in which they were submitted.
CA 02847028 2016-04-26
RESILIENT SIGNAL ENCODING
BACKGROUND
1. Technical Field
[0001] The present disclosure relates to the field of encoding digital signals
to reduce
bandwidth utilization. In particular, to a system and method for resilient
signal encoding.
2. Related Art
[0002] A digital signal that is composed of successive frames of information
(a.k.a.
data) may be encoded using various mechanisms in order to reduce the bandwidth
required to transmit the signal. One such mechanism is inter-frame encoding
where in
some frames may be encoded as independently decodable frames (a.k.a. i-frames)
while
the remaining frames each may be encoded relative to an independently
decodable frame
as difference frames (p-frames) or relative to another p-frame. The mechanism
is
susceptible to some p-frames becoming undecodable when the frame relative to
which
they were encoded is lost or corrupted in transmission. Some mechanisms (e.g.
Internet
Engineering Task Force (IETF), Ott, J., Wenger, S., Sato, N., Burmeister, C.,
and J. Rey,
"Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based
Feedback (RTP/AVPF)", RFC 4585, DOT 10.17487/RFC4585, July 2006) are used
whereby a receiver of the encoded frames may inform the encoder of the frames
when
some received frames are undecodable.
[0003] Typically a trade-off is made between resilience to frame loss or
corruption and
a degree of bandwidth utilization (i.e. compression) in either or both of a
downlink
channel and an uplink channel.
BRIEF DESCRIPTION OF DRAWINGS
[0004] The system and method may be better understood with reference to the
following drawings and description. The components in the figures are not
necessarily to
1
CA 02847028 2016-04-26
scale, emphasis instead being placed upon illustrating the principles of the
disclosure.
Moreover, in the figures, like referenced numerals designate corresponding
parts
throughout the different views.
[0005] Other systems, methods, features and advantages will be, or will
become,
apparent to one with skill in the art upon examination of the following
figures and
detailed description.
[0006] Fig. 1 is a schematic representation of a time series of encoded frames
of a
digital signal.
[0007] Fig. 2 is a schematic representation of a method for resilient signal
encoding.
[0008] Fig. 3 is a schematic representation of system for resilient signal
encoding.
DETAILED DESCRIPTION
[0009] A system and method for resilient signal encoding is described herein.
The
system and method provide for encoding a digital signal to reduce bandwidth
required to
transmit the encoded signal while mitigating the impact of frames lost or
corrupted
during transmission. The system and method may be used in applications such
as, for
example, video conferencing, telepresence, streaming media, video editing and
other
similar applications.
[0010] Figure 1 is a schematic representation of a time series of encoded
frames of an
encoded signal. The encoded signal is derived from an input signal in
accordance with
the system and method for resilient signal encoding. The input signal may be a
digital
signal that comprises a sequence of data samples (e.g. frames) that represent,
for
example, a video stream or other similar content. For illustrative purposes
the input
signal describes herein represents a video stream comprising a sequence of
video frames
Si -S7 but the illustration is not intended to be limiting in any way. In
Figure 1 time is
represented in the horizontal axis progressing from left (earlier time) to
right (later time).
2
CA 02847028 2016-04-26
[0011] A first frame of the input signal Si may be encoded as an independently
decodable frame (I1) and placed in the encoded signal as encoded frame El. The
independently decodable frame El may be transmitted to a receiver for
decoding. The
receiver may decode the independently decodable frame without reference to any
other
frames in the encoded signal as decoded frame Dl. A subsequent frame of the
input
signal S2 may be encoded as a difference frame (P2-1) that represents a
content
difference between the first frame Si and the subsequent frame S2. The first
frame acts
as a reference frame for the encoding of the subsequent frame. The difference
frame E2
may be transmitted to the receiver for decoding. The receiver may decode the
difference
frame E2 by referencing the independently decodable frame El. The
independently
decodable frame El, which corresponds to the first frame Si, acts as a
reference frame
for the decoding of the difference frame E2. Further frames of the input
signal S3 and S4
may be encoded as difference frames E3 and E4 in a manner similar to that
described
above with reference to the subsequent frame S2.
[0012] When the receiver successfully decodes a difference frame, the receiver
may
send an acknowledgement (ACK) indicating that the difference frame was
successfully
decoded. In response to receiving the acknowledgement (ACK), the frame
associated
with acknowledgement S3 may be assigned as the reference frame and further
subsequent
frames of the input signal S5, S6 and S7 may be encoded as difference frames
E5, E6 and
E7 that represent a content difference between the frame S5, S6 and S7 and the
reference
frame S3. The receiver may send an acknowledgement for each difference frame
that is
successfully decoded. Alternatively the receiver may only send an
acknowledgement for
difference frames that are marked as acknowledgement-requested and that have
been
successfully decoded. During encoding some difference frames such as, for
example,
every Nth frame may be marked as acknowledgement-requested before they are
transmitted. Difference frames that are marked as acknowledgement-requested
may be
candidate reference frames. The interval N for marking candidate reference
frames may
be a number between 2 and a predetermined upper limit (e.g. 3000) that may be
preconfigured or may be user specified. In a further alternative, a difference
frame may
be marked as acknowledgement-requested (e.g. a candidate reference frame)
based on the
3
CA 02847028 2016-04-26
content of the frame and/or one or more frames occurring before or after the
frame, for
example, to minimize the difference between the reference frame and
subsequently
encoded frames. In another alternative, frames designated as golden frames
(e.g. as
specified in Internet Engineering Task Force (IETF), Bankoski, J., Koleszar,
J., Quillio,
L., Salonen, J., Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding Guide",
RFC
6386, DOT 10.17487/RFC6386, November 2011) may be marked as acknowledgement-
requested. In the example illustrated in Figure 1, encoded frame E3 was
successfully
decoded and an acknowledgement was sent. Subsequently, frames S5, S6 and S7
are
encoded as difference frames E5, E6 and E7 relative to S3 that corresponds to
frame E3
associated with the sent acknowledgement.
[0013] Alternatively, or in addition, the receiver may send an acknowledgement
whether or not a difference frame was successfully decoded. The
acknowledgement may
include a binary variable, flag or other similar indicator that has one value
or state to
signify successful decoding and another value or state to signify a failure to
successfully
decode the difference frame. The acknowledgement may further include one or
more
indicators of the outcomes (e.g. success or failure) of decoding earlier
frames. For
example, the acknowledgement may also include indicators of the outcomes of
decoding
of K previous frames. The number of previous frames K may be a number (e.g. 7,
15, 23
or 31) that may be preconfigured or may be user specified. The acknowledgement
may
take the form of a binary bit mask in which a first bit represents the outcome
of decoding
a current frame and K further bits represent the outcome of decoding K
previous frames.
In a further alternative, the indicator of the outcome of decoding each frames
may
assigned to a class where the class specifies the number of times (e.g. 32,
16, 8 or 0) that
the indicator will be repeated (e.g. included in an acknowledgement). The
inclusion of an
indicator of the outcome of decoding previous frames may be limited to
previous frames
that are candidate reference frames (e.g. frames that are mark acknowledgement-
requested). The redundant inclusion of indicators of the outcome of decoding
one or
more previous frames in the acknowledgement allows the system and method to be
resilient when one or more acknowledgements are lost during transmission. An
indicator
4
CA 02847028 2016-04-26
that fails to be returned a first lost acknowledgement may be successfully
returned in a
subsequent acknowledgement.
[0014] A failure to successfully decode frame E6 is shown resulting in no
acknowledgement being sent for E6. Encoded frame E7 is successfully decoded
despite
the failure to decode E6 because E3 was successfully decoded to D3 and E7 was
encoded
relative to S3. Due to the failure to decode E6, no acknowledgement was sent
but the
system and method may receive an acknowledgement of the successful decoding of
E7
instead. Difference frame E7 may, for example, be marked as acknowledgement-
requested when no acknowledgement is received for E6. Subsequently encoded
frames
may be encoded relative to S7. Alternatively, when no acknowledgement is
received for
E6 subsequent frames may continue to be encoded relative to the current
reference frame
S3 until an acknowledgement is received for a subsequent candidate reference
frame, for
example the next Nth frame.
[0015] In the illustrated example of Figure 1, S4 is encoded relative to Si
even though
acknowledgement was sent for E3. After an acknowledge is sent, encoding of
subsequent frames may continue to use a previous reference frame for some time
due to
various factors including propagation delay, caching, pipelining and other
similar latency
related factors.
[0016] Alternatively, or in addition, encoding may operate in one of two
possible
modes: normal mode and safe mode. Selection of one of the two modes may be
responsive to feedback from the receiver (e.g. acknowledgements received). In
safe
mode encoding only uses as a reference frame a frame for which an
acknowledgement,
including a positive indication of successfully decoding, has been received.
In normal
mode encoding may use as a reference frame any frame including a frame for
which no
acknowledgement including a positive indication of successfully decoding has
been
received. Encoding may switch from normal mode to safe mode when, for example,
an
acknowledgement is received that includes an indicator of unsuccessful
decoding of a
frame. Encoding may switch from safe mode to normal mode when an abatement
criterion is met. The abatement criterion may include, for example, when a
positive
CA 02847028 2016-04-26
indication of successfully decoding has been received for L success frames,
where L is a
number that may be preconfigured or may be user specified. Alternatively, the
abatement
criterion may specify a minimum time duration during which encoding remains in
safe
mode to mitigate frequent switching between safe and normal mode. Encoding may
be
configured to start either in normal mode or safe mode.
[0017] The independently decodable frame and each of the difference frames is
uniquely identifiable using, for example, sequence numbers that may be
included in the
encoded frames. Each difference frame may also include identification of the
reference
frame from which it was encoded and that may be used to decode the difference
frame.
Each acknowledgement sent by the receiver may identify the frame that was
successfully
decoded using, for example, the sequence number of the frame.
[0018] The system and method may encode and transmit only a single
independently
decodable frame thereby mitigating the bandwidth required to transmit the
encoded
signal. Further independently decodable frames may be encoded and transmitted
at any
time responsive to input such as, for example, a force reset request or
detection of the loss
or failure to decode a previous independently decodable frame. When the
receiver
successfully decodes an independently decodable frame, the receiver may send
an
acknowledgement (ACK) indicating that the independently decodable frame was
successfully decoded. In the absence of receiving an acknowledgement, a
further
independently decodable frame may be encoded and transmitted to the receiver.
[0019] The system and method for resilient signal encoding may use various
forms of
inter-frame compression either lossy or lossless. The inter-frame compression
may be
used in combination with other compression techniques (e.g. predictive
encoding). The
system and method for resilient signal encoding may use encoding formats such
as, for
example, 3. International Telecommunication Union; ITU-T
H.264;
Telecommunication Standardization Sector of ITU, (05/2003), Series H, Advanced
video
coding for generic audiovisual services or International Telecommunication
Union; ITU-
T H.265; Telecommunication Standardization Sector of ITU, (04/2013), Series H,
High
Efficiency Video Coding including scalable and multiview extensions of these
formats
6
CA 02847028 2016-04-26
such as scalable video coding (SVC), multiview video coding (MVC), 3-
dimensional
coding (3D), and The WebM Project VP8 or VP9 coding.
[0020] The acknowledgements may be received using in-band signaling in
accordance
with a video codec syntax being used for the encoded frames (e.g. H.264), as
part of an
ITU-T Supplementary Enhancement Information (SEI) message, or using a user
data
extension mechanism. Alternatively or in addition, the acknowledgements may be
sent
out-of-band using mechanisms such as, for example, as part of an IETF Session
Initiation
Protocol (SIP) message, as part of a Moving Picture Expert Group (MPEG) green
MPEG
set of metadata dedicated to resource saving, as part of an IETF Real Time
Communication Web (RTCWeb) mechanism, as part of an IETF codec control message
or as part of an IETF Audio-Visual Profile with Feedback (AVPF) message.
[0021] The acknowledgement-requested marking of a frame may be sent as a flag
or as
a syntax element using in-band signaling in accordance with a video codec
syntax being
used for the encoded frames (e.g. H.264), as part of an ITU-T SEI message or
using a
user data extension mechanism. Alternatively, or in addition, the
acknowledgement-
requested marking of a frame my be sent out-of-band using mechanisms such as,
for
example, as part of a MPEG green MPEG set of metadata dedicated to resource
saving,
as part of an IETF RTCWeb mechanism or as part of an IETF SIP message.
[0022] Fig. 2 is a representation of a method for resilient signal encoding.
The method
200 may, for example, be implemented using the system 300 described herein
with
reference to Figure 3. The method 200 includes the following acts. Receiving a
signal as
a sequence of frames 202. Encoding a first frame, of the received frames, as
an
independently decodable frame (a.k.a. an i-frame) and assigning or designating
the first
frame as a reference frame 204. Encoding subsequent frames, of the received
signal, as
difference frames (a.k.a. p-frames) representing a difference between a
current frame, of
the subsequent frames, and the reference frame 206. Transmitting the
independently
decodable frame and the difference frames, as they are encoded, to a receiver
208.
Receiving one or more acknowledgements (a.k.a. ACK) each indicating the
outcome of
frame decoding 210. An acknowledgement may be sent by the receiver when an
7
CA 02847028 2016-04-26
independently decodable frame or a difference frame has been successfully
decoded. An
acknowledgement may be sent by the receiver when a golden frame has been
successfully decoded. The receiver may only send an acknowledgement for
difference
frames that are marked as acknowledgement-requested. The acknowledgement may
further include one or more indicators of the outcomes (e.g. success or
failure) of
decoding earlier frames. For example, the acknowledgement may also include
indicators
of the outcomes of decoding of K previous frames. Assigning or designating the
received
frame corresponding to a difference frame associated with a received
acknowledgement
as the reference frame 212. Subsequently encoded difference frames as
described in act
206 may represent a difference between the current frame and the newly
assigned
reference frame. In an alternative embodiment, the method may further include
switching between a safe mode according to act 212 described above and a
normal mode
where encoding may use as a reference frame any frame including a frame for
which no
acknowledgement including a positive indication of successfully decoding has
been
received.
[0023] Figure 3 is a schematic representation of a system for resilient signal
encoding
300. The system 300 comprises a processor 302, memory 304 (the contents of
which are
accessible by the processor 302) and an I/0 interface 306. The memory 304 may
store
instructions which when executed using the process 302 may cause the system
300 to
render the functionality associated with a signal receiver 308, an encoder
310, a signal
transmitter 312 and a decoder 314. Alternatively, or in addition, the
instructions when
executed using the process 302 may configure the system 300 to implement the
acts of
method 200. In addition the memory 304 may store information in data
structures
including, for example, signal frames 316, encoded frames 318, reference frame
320,
decoded frames 322 and golden frames.
[0024] The processor 302 may comprise a single processor or multiple
processors that
may be disposed on a single chip, on multiple devices or distributed over more
that one
system. The processor 302 may be hardware that executes computer executable
instructions or computer code embodied in the memory 304 or in other memory to
8
CA 02847028 2016-04-26
perform one or more features of the system. The processor 302 may include a
general
purpose processor, a central processing unit (CPU), a graphics processing unit
(GPU), an
application specific integrated circuit (ASIC), a digital signal processor
(DSP), a field
programmable gate array (FPGA), a digital circuit, an analog circuit, a
microcontroller,
any other type of processor, or any combination thereof.
[0025] The memory 304 may comprise a device for storing and retrieving data,
processor executable instructions, or any combination thereof The memory 304
may
include non-volatile and/or volatile memory, such as a random access memory
(RAM), a
read-only memory (ROM), an erasable programmable read-only memory (EPROM), or
a
flash memory. The memory 304 may comprise a single device or multiple devices
that
may be disposed on one or more dedicated memory devices or on a processor or
other
similar device. Alternatively or in addition, the memory 304 may include an
optical,
magnetic (hard-drive) or any other form of data storage device.
[0026] The memory 304 may store computer code, such as signal receiver 308,
encoder
310, signal transmitter 312 and decoder 314 as described herein. The computer
code may
include instructions executable with the processor 302. The computer code may
be
written in any computer language, such as C, C++, assembly language, channel
program
code, and/or any combination of computer languages. The signal receiver 308
may
receive a signal as a sequence of frames. The encoder 310 may encode a first
frame, of
the received frames, as an independently decodable frame and assign the first
frame as a
reference frame. The encoder 310 may encode subsequent frames, of the received
signal,
as difference frames representing a difference between a current frame, of the
subsequent
frames, and the reference frame. The signal transmitter 312 may transmit the
independently decodable frame and the difference frames, as they are encoded,
to a
receiver. The signal receiver 308 may also receive acknowledgements associated
with
successfully decoded frames. When an acknowledgement is received, the received
frame
corresponding to the difference frame associated with a received
acknowledgement may
be assigned as the reference frame. Subsequently encoded frames of the
received frames
may be encoded relative to the newly assigned reference frame by the encoder
310. The
9
CA 02847028 2016-04-26
system 300 may not include the decoder 314 when an encoded signal is only
transmitted
to the receiver (e.g. in one-way transmission). When encoded signals are
transmitted to
the receiver and also received by the system (e.g. in two-way transmission)
the system
300 may include the decoder 314. The decoder 314 decodes the received encoded
frames
and when a frame is successfully decoded, the decoder 314 may send an
acknowledgement to the transmitter of the encoded frames. The receiver may
only send
an acknowledgement for difference frames that are marked as acknowledgement-
requested.
[0027] The I/0 interface 306 may be used to connect devices such as, for
example, data
transmission media and other components of the system 300.
[0028] All of the disclosure, regardless of the particular implementation
described, is
exemplary in nature, rather than limiting. The system 300 may include more,
fewer, or
different components than illustrated in Figure 3. Furthermore, each one of
the
components of system 300 may include more, fewer, or different elements than
is
illustrated in Figure 3. Flags, data, databases, tables, entities, and other
data structures
may be separately stored and managed, may be incorporated into a single memory
or
database, may be distributed, or may be logically and physically organized in
many
different ways. The components may operate independently or be part of a same
program or hardware. The components may be resident on separate hardware, such
as
separate removable circuit boards, or share common hardware, such as a same
memory
and processor for implementing instructions from the memory. Programs may be
parts of
a single program, separate programs, or distributed across several memories
and
processors.
[0029] The functions, acts or tasks illustrated in the figures or described
may be
executed in response to one or more sets of logic or instructions stored in or
on computer
readable media. The functions, acts or tasks are independent of the particular
type of
instructions set, storage media, processor or processing strategy and may be
performed by
software, hardware, integrated circuits, firmware, micro code and the like,
operating
alone or in combination. Likewise, processing strategies may include
multiprocessing,
CA 02847028 2016-04-26
multitasking, parallel processing, distributed processing, and/or any other
type of
processing. In one embodiment, the instructions are stored on a removable
media device
for reading by local or remote systems. In other embodiments, the logic or
instructions
are stored in a remote location for transfer through a computer network or
over telephone
lines. In yet other embodiments, the logic or instructions may be stored
within a given
computer such as, for example, a CPU.
[0030] While various embodiments of the system and method for on-demand user
control have been described, it will be apparent to those of ordinary skill in
the art that
many more embodiments and implementations are possible within the scope of the
present invention. Accordingly, the invention is not to be restricted except
in light of the
attached claims and their equivalents.
11
