Note: Descriptions are shown in the official language in which they were submitted.
CA 02738329 2011-05-02
TITLE
Picture Level Adaptive Frame/Field Coding For Digital Video Content
TECHNICAL FIELD
[0001] The present invention relates to encoding and decoding of digital video
content. More specifically, the present invention relates to frame mode and
field mode
encoding and decoding of digital video content at the picture level as used in
the MPEG-
4 Part 10 AVC/H.264 video coding standard.
BACKGROUND
[0002] Video compression is used in many current and emerging products. It is
at
the heart of digital television set-top boxes (STBs), digital satellite
systems (DSSs), high
definition television (HDTV) decoders, digital versatile disk (DVD) players,
video
conferencing, Internet video and multimedia content, and other digital video
applications.
Without video compression, digital video content can be extremely large,
making it
difficult or even impossible for the digital video content to be efficiently
stored,
transmitted, or viewed.
[0003] The digital video content comprises a stream of pictures that can be
displayed as an image on a television receiver, computer monitor, or some
other
electronic device capable of displaying digital video content. A picture that
is displayed
in time before a particular picture is in the "forward direction" in relation
to the particular
picture. Likewise, a picture that is displayed in time after a particular
picture is in the
"backward direction" in relation to the particular picture.
[0004] Video compression is accomplished in a video encoding, or coding,
process in which each picture is encoded as either a frame or as two fields.
Each frame
comprises a number of lines of spatial information. For example, a typical
frame
contains 480 horizontal lines. Each field contains half the number of lines in
the frame.
For example, if the frame comprises 480 horizontal lines, each field comprises
240
horizontal lines. In a typical configuration, one of the fields comprises the
odd numbered
lines in the frame and the other field comprises the even numbered lines in
the frame.
1
CA 02738329 2011-05-02
The field that comprises the odd numbered lines will be referred to as the
"top" field
hereafter and in the appended claims, unless otherwise specifically denoted.
Likewise,
the field that comprises the even numbered lines will be referred to as the
"bottom" field
hereafter and in the appended claims, unless otherwise specifically denoted.
The two
fields can be interlaced together to form an interlaced frame.
[0005] The general idea behind video coding is to remove data from the digital
video content that is "non-essential." The decreased amount of data then
requires less
bandwidth for broadcast or transmission. After the compressed video data has
been
transmitted, it must be decoded, or decompressed. In this process, the
transmitted video
data is processed to generate approximation data that is substituted into the
video data to
replace the "non-essential" data that was removed in the coding process.
[00061 Video coding transforms the digital video content into a compressed
form
that can be stored using less space and transmitted using less bandwidth than
uncompressed digital video content. It does so by taking advantage of temporal
and
spatial redundancies in the pictures of the video content. The digital video
content can be
stored in a storage medium such as a hard drive, DVD, or some other non-
volatile storage
unit.
[0007] There are numerous video coding methods that compress the digital video
content. Consequently, video coding standards have been developed to
standardize the
various video coding methods so that the compressed digital video content is
rendered in
formats that a majority of video encoders and decoders can recognize. For
example, the
Motion Picture Experts Group (MPEG) and International Telecommunication Union
(ITU-T) have developed video coding standards that are in wide use. Examples
of these
standards include the MPEG-l, MPEG-2, MPEG-4, ITU-T H261, and ITU-T 'H263
standards.
[0008] Most modern video coding standards, such as those developed by MPEG
and ITU-T, are based in part on a temporal prediction with motion compensation
(MC)
algorithm. Temporal prediction with motion compensation is used to remove
temporal
redundancy between successive pictures in a digital video broadcast.
[0009] The temporal prediction with motion compensation algorithm typically
utilizes one or two reference pictures to encode a particular picture. A
reference picture
is a picture that has already been encoded. By comparing the particular
picture that is to
2
CA 02738329 2011-05-02
be encoded with one of the reference pictures, the temporal prediction with
motion
compensation algorithm can take advantage of the temporal redundancy that
exists
between the reference picture and the particular picture that is to be encoded
and encode
the picture with a higher amount of compression than if the picture were
encoded without
using the temporal prediction with motion compensation algorithm. One of the
reference
pictures may be in the backward direction in relation to the particular
picture that is to be
encoded. The other reference picture is in the forward direction in relation
to the
particular picture that is to be encoded.
[00101 However, as the demand for higher resolutions, more complex graphical
content, and faster transmission time increases, so does the need for better
video
compression methods. To this end, a new video coding standard is currently
being
developed. This new video coding standard is called the MPEG-4 Part 10
AVC/H.264
standard.
[00111 The new MPEG-4 Part 10 AVC/H.264 standard calls for a number of new
methods in video compression. For example, one of the features of the new MPEG-
4
Part 10 AVC/H.264 standard is that it allows multiple reference pictures,
instead of just
two reference pictures. The use of multiple reference pictures improves the
performance
of the temporal prediction with motion compensation algorithm by allowing the
encoder
to find the reference picture that most closely matches the picture that is to
be encoded.
By using the reference picture in the coding process that most closely matches
the picture
that is to be encoded, the greatest amount of compression is possible in the
encoding of
the picture. The reference pictures are stored in frame and/or field buffers.
[00121 As previously stated, the encoder can encode a picture as a frame or as
two fields. A greater degree of compression could be accomplished if, in a
sequence of
pictures that is to be encoded, some of the pictures are encoded as frames and
some of the
pictures are encoded as fields.
SUMMARY OF THE INVENTION
[0013] In one of many possible embodiments, the present invention provides a
method of encoding, decoding, and bitstream generation of digital video
content. The
digital video content comprises a stream of pictures which can each be intra,
predicted, or
bi-predicted pictures. Each of the pictures comprises macroblocks that can be
further
3
CA 02738329 2011-05-02
divided into smaller blocks. The method entails encoding and decoding each
picture in
said stream of pictures in either frame mode or in field mode.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings illustrate various embodiments of the present
invention and are a part of the specification. Together with the following
description, the
drawings demonstrate and explain the principles of the present invention. The
illustrated
embodiments are examples of the present invention and do not limit the scope
of the
invention.
[0015] FIG. 1 illustrates an exemplary sequence of three types of pictures
that can
be used to implement the present invention, as defined by an exemplary video
coding
standard such as the MPEG-4 Part 10 AVC/H.264 standard.
[0016] FIG. 2 shows that each picture is preferably divided into slices
containing
macroblocks according to an embodiment of the present invention.
[0017] FIG. 3a shows that a macroblock can be further divided into a block
size
of 16 by 8 pixels according to an embodiment of the present invention.
[0018] FIG. 3b shows that a macroblock can be further divided into a block
size
of 8 by 16 pixels according to an embodiment of the present invention.
[0019] FIG. 3c shows that a macroblock can be further divided into a block
size
of 8 by 8 pixels according to an embodiment of the present invention.
[0020] FIG. 3d shows that a macroblock can be further divided into a block
size
of 8 by 4 pixels according to an embodiment of the present invention.
[0021] FIG. 3e shows that a macroblock can be further divided into a block
size
of 4 by 8 pixels according to an embodiment of the present invention.
[0022] FIG. 3f shows that a macroblock can be further divided into a block
size
of 4 by 4 pixels according to an embodiment of the present invention.
[0023] FIG. 4 shows a picture construction example using temporal prediction
with motion compensation that illustrates an embodiment of the present
invention.
[0024] FIG. 5 shows an exemplary stream of pictures which illustrates an
advantage of using multiple reference pictures in temporal prediction with
motion
compensation according to an embodiment of the present invention.
4
CA 02738329 2011-05-02
[0025] FIG. 6 illustrates according to an embodiment of the present invention
that
a unique reference frame number is assigned to each reference frame in the
frame buffer
according to its distance from the current picture that is being encoded in
frame mode.
[0026] FIG. 7a shows an exemplary reference field numbering configuration
where the reference fields of the same field parity as the current field are
given smaller
numbers than are their corresponding second fields according to an embodiment
of the
present invention.
[0027] FIG. 7b shows an exemplary reference field numbering configuration
where the current field is the second field of the picture that is to be
encoded as two
fields.
[0028] FIG. 8 shows an alternate reference field numbering configuration in
the
field buffer according to an embodiment of the present invention.
[0029] FIG. 9 illustrates a method of direct mode vector calculation where
both
the current macroblock and its co-located macroblock are in frame mode.
[0030] FIG. 10 illustrates a method of direct mode vector calculation where
both
the current macroblock and its co-located macroblock are in field mode.
[0031] FIG. 11 illustrates another method of direct mode vector calculation
where
both the current macroblock and its co-located macroblock are in field mode.
[0032] FIG. 12 illustrates a method of direct mode vector calculation where
the
current macroblock is in field mode and its co-located macroblock is in frame
mode.
[0033] FIG. 13 illustrates a method of direct mode vector calculation where
the
current macroblock is in frame mode and its co-located macroblock is in field
mode.
[0034] FIG: 14 shows a B picture with its two reference pictures in the
temporal
forward direction according to an embodiment of the present invention.
[0035] FIG. 15 shows a B picture with its two reference pictures in the
temporal
backward direction according to an embodiment of the present invention.
[0036] FIG. 16 shows a B picture with a forward reference picture in the
temporal
forward direction and a backward reference picture in the temporal backward
direction.
[0037) Throughout the drawings, identical reference numbers designate similar,
but not necessarily identical, elements.
CA 02738329 2011-12-06
DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION
[0038] The present invention provides a method of adaptive frame/field (AFF)
coding of digital video content comprising a stream of pictures at a picture
level. In AFF
coding at a picture level, each picture in a stream of pictures that is to be
encoded is encoded
in either frame mode or in field mode, regardless of the other picture's frame
or field coding
mode. If a picture is encoded in frame mode, the two fields that make up an
interlaced frame
are coded jointly. Conversely, if a picture is encoded in field mode, the two
fields that make
up an interlaced frame are coded separately. The encoder determines which type
of coding,
frame mode coding or field mode coding, is more advantageous for each picture
and chooses
that type of encoding for the picture. The exact method of choosing between
frame mode and
field mode is not critical to the present invention and will not be detailed
herein.
[0039] As noted above, the MPEG-4 Part 10 AVC/H.264 standard is a new standard
for encoding and compressing digital video content. The documents establishing
the MPEG-4
Part 10 AVC/H.264 standard include the "Joint Final Committee Draft (JFCD) of
Joint Video
Specification" issued by the Joint Video Team (JVT) on August 10, 2002. (ITU-T
Rec.
H.264 & ISO/IEC 14496-10 AVC). The JVT consists of experts from ISO or MPEG
and
ITU-T. Due to the public nature of the MPEG-4 Part 10 AVC/H.264 standard, the
present
specification will not attempt to document all the existing aspects of MPEG-4
Part 10
AVC/H.264 video coding, relying instead on the publicly available
specifications of the
standard.
[0040] Although this method of AFF encoding is compatible with and will be
explained using the MPEG-4 Part 10 AVC/H.264 standard guidelines, it can be
modified and
used as best serves a particular standard or application.
[0041] Using the drawings, the preferred embodiments of the present invention
will
now be explained.
[0042] FIG. I illustrates an exemplary sequence of three types of pictures
that can be
used to implement the present invention, as defined by an exemplary video
coding standard
such as the MPEG-4 Part 10 A VC/H.264 standard. As previously mentioned, the
encoder
encodes the pictures and the decoder decodes the pictures. The encoder or
decoder can be a
processor, application specific integrated circuit (ASIC), field
6
CA 02738329 2011-05-02
programmable gate array (FPGA), coder/decoder (CODEC), digital signal
processor
(DSP), or some other electronic device that is capable of encoding the stream
of pictures.
However, as used hereafter and in the appended claims, unless otherwise
specifically
denoted, the term "encoder" will be used to refer expansively to all
electronic devices
that encode digital video content comprising a stream of pictures. Also, as
used hereafter
and in the appended claims, unless otherwise specifically denoted, the term
"decoder"
will be used to refer expansively to all electronic devices that decode
digital video
content comprising a stream of pictures.
[0043] As shown in FIG. 1, there are preferably three types of pictures that
can be
used in the video coding method. Three types of pictures are defined to
support random
access to stored digital video content while exploring the maximum redundancy
reduction using temporal prediction with motion compensation. The three types
of
pictures are infra (I) pictures (100), predicted (P) pictures (102a,b), and bi-
predicted (B)
pictures (10la-d). An I picture (100) provides an access point for random
access to
stored digital video content and can be encoded only with slight compression.
Intra
pictures (100) are encoded without referring to reference pictures.
[0044] A predicted picture (102a,b) is encoded using an I, P, or B picture
that has
already been encoded as a reference picture. The reference picture can be in
either the
forward or backward temporal direction in relation to the P picture that is
being encoded.
The predicted pictures (102a,b) can be encoded with more compression than the
intra
pictures (100).
[0045] A bi-predicted picture (10la-d) is encoded using two temporal reference
pictures: a forward reference picture and a backward reference picture. The
forward
reference picture is sometimes called the past reference picture and the
backward
reference picture is sometimes called a future reference picture. An
embodiment of the
present invention is that the forward reference picture and backward reference
picture can
be in the same temporal direction in relation to the B picture that is being
encoded. Bi-
predicted pictures (10la-d) can be encoded with the most compression out of
the three
picture types.
[0046] Reference relationships (103) between the three picture types are
illustrated in FIG. 1. For example, the P picture (102a) can be encoded using
the encoded
I picture (100) as its reference picture. The B pictures (10la-b) can be
encoded using the
7
CA 02738329 2011-05-02
encoded I picture (100) and/or the encoded P picture (102a) as its reference
pictures, as
shown in FIG. 1. Under the principles of an embodiment of the present
invention,
encoded B pictures (lOla-d) can also be used as reference pictures for other B
pictures
that are to be encoded. For example, the B picture (101c) of FIG. 1 is shown
with two
other B pictures (10 lb and lOld) as its reference pictures.
[0047] The number and particular order of the I (100), B (101 a-d), and P
(102a,b)
pictures shown in FIG. 1 are given as an exemplary configuration of pictures,
but are not
necessary to implement the present invention. Any number of I, B, and P
pictures can be
used in any order to best serve a particular application. The MPEG-4 Part 10
AVC/H.264 standard does not impose any limit to the number of B pictures
between two
reference pictures nor does it limit the number of pictures between two I
pictures.
[0048] FIG. 2 shows that each picture (200) is preferably divided into slices
(202). A slice (202) contains a group of macroblocks (202). A macroblock (201)
is a
rectangular group of pixels. As shown in FIG. 2, a preferable macroblock (201)
size is
16 by 16 pixels.
[0049] FIGS. 3a-f show that a macroblock can be further divided into smaller
sized blocks. For example, as shown in FIGS. 3a-f, a macroblock can be further
divided
into block sizes of 16 by 8 pixels (FIG. 3a; 300), 8 by 16 pixels (FIG 3b;
301), 8 by 8
pixels (FIG. 3c; 302), 8 by 4 pixels (FIG. 3d; 303), 4 by 8 pixels (FIG. 3e;
304), or 4 by 4
pixels (FIG. 3f; 305).
[0050] FIG. 4 shows a picture construction example using temporal prediction
with motion compensation that illustrates an embodiment of the present
invention.
Temporal prediction with motion compensation assumes that a current picture,
picture N
(400), can be locally modeled as a translation of another picture, picture N-1
(401). The
picture N-1 (401) is the reference picture for the encoding of picture N (400)
and can be
in the forward or backwards temporal direction in relation to picture N (400).
[0051] As shown in FIG. 4, each picture is preferably divided into slices
containing macroblocks (201a,b). The picture N-1 (401) contains an image (403)
that is
to be shown in picture N (400). The image (403) will be in a different
temporal position
(402) in picture N (400) than it is in picture N-1 (401), as shown in FIG. 4.
The image
content of each macroblock (201 a) of picture N (400) is predicted from the
image content
of each corresponding macroblock (201b) of picture N-1 (401) by estimating the
required
8
CA 02738329 2011-05-02
amount of temporal motion of the image content of each macroblock (201b) of
picture N-
1 (401) for the image (403) to move to its new temporal position (402) in
picture N (400).
Instead of the original image (402), the difference (404) between the image
(402) and its
prediction (403) is actually encoded and transmitted..
[0052] For each image (402) in picture N (400), the temporal prediction can
often
be represented by motion vectors that represent the amount of temporal motion
required
for the image (403) to move to a new temporal position (402) in the picture N
(400). The
motion vectors (406) used for the temporal prediction with motion compensation
need to
be encoded and transmitted.
[0053] FIG. 4 shows that the image (402) in picture N (400) can be represented
by the difference (404) between the image and its prediction and the
associated motion
vectors (406). The exact method of encoding using the motion vectors can vary
as best
serves a particular application and can be easily implemented by someone who
is skilled
in the art.
[0054] FIG. 5 shows an exemplary stream of pictures which illustrates an
advantage of using multiple reference pictures in temporal prediction with
motion
compensation according to an embodiment of the present invention. The use of
multiple
reference pictures increases the likelihood that the picture N (400) to be
encoded with the
most compression possible. Pictures N-1 (401), N-2 (500), and N-3 (501) have
been
already encoded in this example. As shown in FIG. 5, an image (504) in picture
N-3
(501) is more similar to the image (402) in picture N (400) than are the
images (503, 502)
of pictures N-2 (500) and N-1 (401), respectively. The use of multiple
reference pictures
allows picture N (400) to be encoded using picture N-3 (501) as its reference
picture
instead of picture N-1 (401).
[0055] Picture level AFF coding of a stream of pictures will now be explained
in
more detail. A frame of an interlaced sequence contains two fields, the top
field and the
bottom field, which are interleaved and separated in time by a field period.
The field
period is half the time of a frame period. In picture level AFF coding, the
two fields of
an interlaced frame can be coded jointly or separately. If they are coded
jointly, frame
mode coding is used. Conversely, if the two fields are coded separately, field
mode
coding is used.
-9
CA 02738329 2011-05-02
[0056] Fixed frame/field coding, on the other hand, codes all the pictures in
a
stream of pictures in one mode only. That mode can be frame mode or it can be
field
mode. Picture level AFF is preferable to fixed frame/field coding because it
allows the
encoder to choose which mode, frame mode or field mode, to encode each picture
in the
stream of pictures based on the contents of the digital video material.
[0057] Frame mode coding uses pictures that have already been encoded as
reference frames. The reference frames can be any coded I, P, or B frame. The
reference
frames are stored in a frame buffer, which is part of the encoder. An
embodiment of the
present invention is that a unique reference frame number is assigned to each
reference
frame in the frame buffer according to its distance from the current picture
that is being
encoded in frame mode, as shown in the exemplary configuration of FIG. 6. For
example, as shown in FIG. 6, a current picture that is to be encoded as a
frame (600) has
a number of reference frames (0-5) in the frame buffer (601). Also shown in
FIG. 6 are
the corresponding fields (fl, f2) to the current frame (600) and the reference
frames (0-5).
The dotted lines labeled fl are first fields and the dotted lines labeled f2
are second fields.
A first field is the first field that is encoded in a picture of two fields.
Likewise, a second
field is the second field that is encoded in a picture of two fields. An
embodiment of the
present invention is that the first field can be either the top or bottom
field. In another
embodiment of the present invention, the second field can also be either the
top or bottom
field. The frames are represented by solid lines. As shown in FIG.6, the
reference frame
0 is temporally the closest reference frame to the current frame (600). The
reference
frame number increases the further the reference frame is temporally from the
current
frame (600).
[0058] Under principles of an embodiment of the present invention, a B picture
that is encoded as a frame can have multiple forward and backward reference
pictures.
Unique numbers are assigned to the forward and backward reference pictures.
[0059] In the temporal prediction with motion compensation algorithm, sub-pel
interpolation is performed on each of the pixels in a picture that is encoded
as a frame.
Padding can also be applied to reference pictures encoded as frames by
repeating the
pixels on the frame boundaries. Padding is sometimes desirable in the temporal
prediction with motion compensation algorithm. Loop filtering, or de-blocking
schemes,
CA 02738329 2011-05-02
can be applied to frame blocks to account for pixel value discontinuities at
the edges of
adjacent blocks.
[00601 According to another embodiment of the present invention, a macroblock
in a P picture can be skipped in AFF coding. If a macroblock is skipped, its
data is not
transmitted in the encoding of the picture. A skipped macroblock in a P
picture is
reconstructed by copying the co-located macroblock with motion compensation in
the
most recently coded I or P reference picture that has been encoded.
[00611 Field mode coding uses pictures that have already been encoded as
reference fields. The reference fields can be any coded I, P, or B fields. The
reference
fields are stored in a field buffer, which is part of the encoder. An
embodiment of the
present invention is that a unique reference field number is assigned to each
reference
field in the field buffer according to its distance from the current picture
that is being
encoded as two fields. FIG. 7a and FIG. 7b show exemplary reference field
numbering
configurations where the reference fields of the same field parity as the
current field are
given smaller numbers than are their corresponding second fields according to
an
embodiment of the present invention. Two fields have the same field parity if
they are
both top fields or if they are both bottom fields. In the examples of FIG. 7a
and FIG. 7b,
if the first field of the current picture that is to be encoded is a top
field, then the first
fields of the reference pictures are top fields as well. The second fields
would then be
bottom fields. The first fields can also be all bottom fields and the second
fields can all
be top fields.
[00621 As shown in FIG. 7a, a current picture that is to be encoded in field
mode
has a number of reference fields (0-10) in the field buffer (701). The dotted
lines labeled
fl are first fields and the dotted lines labeled f2 are second fields. The
corresponding
frames to the fields are also shown in FIG. 7a and are represented by solid
lines. As
shown in FIG. 7a, if the current field (700) is the first field of the picture
that is to be
encoded, the first field of the first picture in the field buffer (701) is
assigned the number
0, while the second field of the first picture in the field buffer (701) is
assigned the
number 1. The reference field numbers increase the further the reference
fields are
temporally from the current field (700). The first fields of the pictures in
the field buffer
have lower reference numbers than do their corresponding second fields.
11
CA 02738329 2011-05-02
[0063] FIG. 7b shows an exemplary reference field numbering configuration
where the current field (702) is the second field of the picture that is to be
encoded as two
fields. The dotted lines labeled fl are first fields and the dotted lines
labeled f2 are
second fields. The first field of the current picture has already been coded.
As shown in
FIG. 7b, because the current field (702) is a second field, the second field
of the first
picture in the field buffer (701) is assigned the number 0. The first coded
field of the
current picture is assigned the number 0. The reference field numbers increase
the
further the reference fields are temporally from the current field (702). The
second fields
of the pictures in the field buffer have lower reference numbers than do their
corresponding first fields.
[0064] FIG. 8 shows an alternate reference field numbering configuration in
the
field buffer according to an embodiment of the present invention. In this
configuration,
no favoring is given to fields of the same field parity as the current field.
For example, as
shown in FIG. 8, the current field (800) is a first field. The most recently
coded field of
the most recently coded picture in the field buffer is assigned the reference
number 0.
The reference field numbers increase the further the reference fields are
temporally from
the current field (800), regardless of their field parity.
[0065] According to another embodiment of the present invention, if field
coding
is selected by the encoder for a particular P picture, the encoder can use the
first field that
is encoded as a reference field for the encoding of the second field. If the
picture is a B
picture, the first field that is encoded can be used as one of the two
reference fields for
the encoding of the second field.
[0066] For adaptive bi-prediction (ABP), the two reference pictures can be
coded
in field mode. In this case, the temporal distances used in calculating the
scaled motion
vectors are in field interval. In ABP coding, both the reference pictures are
in the same
direction.
[0067] In the temporal prediction with motion compensation algorithm, sub-pel
interpolation is performed on each of the pixels in a picture that is encoded
in field mode.
Padding can also be applied to reference pictures encoded in field mode by
repeating the
pixels on the field boundaries. Padding is sometimes desirable in the temporal
prediction
with motion compensation algorithm. Loop filtering, or de-blocking schemes,
can be
12
CA 02738329 2011-05-02
applied to field blocks to account for pixel value discontinuities at the
edges of adjacent
blocks.
[0068] According to another embodiment of the present invention, a macroblock
in a P picture can be skipped in AFF coding. If a macroblock is skipped, its
data is not
transmitted in the encoding of the picture. A skipped macroblock in a P
picture is
reconstructed by copying the co-located macroblock with motion compensation in
the
most recently coded I or P reference field of the same field parity. Another
embodiment
is that the skipped macroblock in the P picture is reconstructed by copying
the co-located
macroblock in the most recently coded reference field, which can be of a
different field
parity.
[0069] Another embodiment of the present invention is direct mode coding for B
pictures. In direct mode coding, the forward and backward motion vectors for
the
macroblocks in a B picture are derived from the motion vectors used in the
corresponding, or co-located macroblocks of a backward reference picture. Co-
located
macroblocks in two pictures occupy the same geometric position in both of the
pictures.
The backward reference picture is sometimes referred to as a forward reference
picture,
even though according to an embodiment of the present invention, the backward
reference picture is not necessarily temporally ahead of the current picture
that is being
encoded.
[0070] Direct mode coding is advantageous over other methods of coding because
a macroblock can have up to 16 motion vectors and up to 4 reference frames in
inter
mode coding. Inter mode coding codes a macroblock using temporal prediction
with
motion compensation. If a macroblock is coded using inter coding, the MPEG-4
Part 10
AVC/H.264 standard allows each of the six smaller block sizes of FIGS. 3a-f
(16 by 8
pixels, 8 by 16 pixels, 8 by 8 pixels, 8 by 4 pixels, 4 by 8 pixels, and 4 by
4 pixels) to
have its own motion vector. A block size of 16 by 16 pixels can also have its
own motion
vector. The MPEG-4 Part 10 AVC/H.264 standard also allows block sizes of 16 by
16
pixels, 16 by 8 pixels, 8 by 16 pixels, and 8 by 8 pixels to have its own
reference frame.
Thus, a macroblock can have up to 16 motion vectors and up to 4 reference
frames. With
so many potential motion vectors, it is advantageous to derive the motion
vectors of a
macroblock that is to be encoded from motion vectors of the backward reference
picture's co-located macroblock that are already calculated. In direct mode
coding, the
13
CA 02738329 2011-05-02
forward and backward motion vectors of a macroblock that is to be encoded are
computed as the scaled versions of the forward motion vectors of the co-
located
macroblock in the backward reference picture.
[0071] In AFF coding at the picture level, a B picture and its backward
reference
picture can each be coded in frame mode or in field mode. Hence, in terms of
frame and
field coding modes, there can be four different combinations for a pair of
macroblocks in
the B picture and its co-located macroblock of the backward reference picture.
In case 1,
both the current macroblock and its co-located macroblock are in frame mode.
In case 2,
both the current macroblock and its co-located macroblock are in field mode.
In case 3,
the current macroblock is in field mode and its co-located macroblock is in
frame mode.
Finally, in case 4, the current macroblock is in frame mode and its co-located
macroblock
is in field mode. The method of direct mode motion vector calculation for the
macroblocks that are to be encoded is different in each of the four cases. The
four
methods of direct motion vector calculation for macroblocks in a B picture
will be
described in detail below.
[0072] The method of direct mode vector calculation in case 1 will be
described
in connection with FIG. 9. As shown in FIG. 9, a current B picture (900) is to
be
encoded in frame mode using a backward reference picture (901) that has been
encoded
in frame mode and a forward reference picture (902) as its reference pictures.
Frames in
FIG. 9 are represented with solid vertical lines and their corresponding
fields, fl and f2,
are represented with dashed vertical lines. According to an embodiment of the
present
invention, the backward reference picture (901) can be an I, P, or B picture
that has been
encoded in frame mode. Similarly, the forward reference picture (902) can also
be an
encoded I, P, or B picture.
[0073] As shown in FIG. 9, there is a block (903) in the current B picture
(900)
and its co-located block (904) in the backward reference picture (901). The
block (903)
and the co-located block (904) have equal pixel dimensions. These dimensions
can be 16
by 16 pixels, 16 by 8 pixels, 8 by 16 pixels, 8 by 8 pixels, 8 by 4 pixels, 4
by 8 pixels, or
4 by 4 pixels. According to an embodiment of the present invention, the
encoder derives
two motion vectors for the block (903) in the current B picture (900) that are
used in the
temporal prediction with motion compensation algorithm. One of the motion
vectors,
MVF, points to the forward reference picture (902). The other motion vector,
MVB,
14
CA 02738329 2011-05-02
points to the backward reference picture (901) that has been encoded in frame
mode. The
two motion vectors are calculated by:
(0074] MVF = TRB = MV / TRD
MVB = (TRB - TRD) = MV / TRD (Eqs. 1 and 2)
[0075] In Eqs. 1 and 2, TRB is the temporal distance, approximation of the
temporal distance, proportional distance to the temporal distance, or
proportional
approximation to the approximation of the temporal distance between the
current B
picture (900) that is to be encoded in frame mode and the forward reference
picture (902).
TRD is the temporal distance, approximation of the temporal distance,
proportional
distance to the temporal distance, or proportional approximation to the
approximation of
the temporal distance between the forward (902) and backward reference
pictures (901)
that have been encoded in frame mode. A preferable method of calculating the
temporal
distances between reference pictures will be explained below. MV is the motion
vector
that has already been calculated for the co-located block (904) in the
backward reference
picture (901) and points to the forward reference picture (902).
[0076] The method of direct mode vector calculation in case 2 will be
described
in connection with FIG. 10 and FIG. 11. As shown in FIG. 10 and FIG. 11, a
current B
picture (900) is to be encoded in field mode using a backward reference
picture (901) that
has been encoded in field mode and a forward reference picture (902) as its
reference
pictures. Frames in FIG. 10 and FIG. 11 are represented with solid vertical
lines and
their corresponding fields, fl and 12, are represented with dashed vertical
lines.
According to an embodiment of the present invention, the backward reference
picture
(901) can be an I, P, or B picture that has been encoded in field mode.
Similarly, the
forward reference picture (902) can also be an encoded I, P, or B picture.
[0077] As shown in FIG. 10, there is a block (905) in the first field of the
current
B picture (900). Its motion vectors are derived from the forward motion
vector, MV1 of
its co-located block (906) in the backward reference picture (901). According
to an
embodiment shown in FIG. 10, the co-located block (906) is in a field of the
same parity
as is the block (905) in the current B picture (900). The block (905) and the
co-located
block (906) have equal pixel dimensions. These dimensions can be 16 by 16
pixels, 16
by 8 pixels, 8 by 16 pixels, 8 by 8 pixels, 8 by 4 pixels, 4 by 8 pixels, or 4
by 4 pixels.
CA 02738329 2011-05-02
[0078] According to an embodiment of the present invention, the encoder
derives
two motion vectors for the block (905) in the current B picture (900) that are
used in the
temporal prediction with motion compensation algorithm. One of the motion
vectors,
MVF,1, points to the field in the forward reference picture (902) to which MV1
points.
The other motion vector, MVB,1, points to the field of the co-located block
(906) in the
backward reference picture (901). The two motion vectors are calculated by:
[0079] MVF,i = TRB i . MV I TRD t (Eqs. 3 and 4)
MVB, = (TRB; -TRDi)=MV ITRD!
[0080] In Eqs. 3 and 4, the subscript, i, is the field index. The first field
has a
field index of 1 and the second field's field index is 2. Thus, in the
exemplary scenario of
FIG. 10, the field index is 1 because the first field is being encoded. MVI is
the forward
motion vector of the co-located macroblock in field i of the backward
reference picture
(901). TRB,t is the temporal distance, approximation of the temporal distance,
proportional distance to the temporal distance, or proportional approximation
to the
approximation of the temporal distance between the i-th field of the current B
picture
(900) and the reference field pointed to by MV,. TRD,, is the temporal
distance,
approximation of the temporal distance, proportional distance to the temporal
distance, or
proportional approximation to the approximation of the temporal distance
between the i-
th field of the backward reference picture (901) and the reference field
pointed to by
MV,.
[0081] As shown in FIG. 10, there is another block (907) in the second field
of
the current B picture (900). It has a co-located block (908) in the second
field of the
backward reference picture (901). If the forward motion vector of the co-
located block
(908) points to a previously coded field in any picture other than its own
picture, the
calculation of the forward and backward motion vectors follow Eqs. 3 and 4,
with the
field index equal to 2.
[0082] However, according to an embodiment of the present invention, the
forward motion vector of the co-located block (908) in the second field of the
backward
reference picture (901) can also point to the first field of the same backward
reference
picture (901), as shown in FIG. 11. FIG. 11 shows that the co-located block
(908) has a
forward motion vector, MV2, that points to the first field of the backward
reference
16
CA 02738329 2011-05-02
picture (901). In this case, the two motion vectors for the current block
(907) are
calculated as follows:
[0083] MVF,2 = -TRB 2 = MV2 I TRD 2 (Eqs. 5 and 6)
MVB2 =-(TRB2+TRD2)=MV2/TRD2
[0084] In Eqs. 5 and 6, TRB,2 is the temporal distance, approximation of the
temporal distance, proportional distance to the temporal distance, or
proportional
approximation to the approximation of the temporal distance between the second
field of
the current B picture (900) and the reference field pointed to by MV2. TRD,2
is the
temporal distance, approximation of the temporal distance, proportional
distance to the
temporal distance, or proportional approximation to the approximation of the
temporal
distance between the second field of the backward reference picture (901) and
the
reference field pointed to by MV2. In this case, as shown in FIG. 11, both
motion vectors
point in the backward direction.
[0085] The method of direct mode vector calculation in case 3 will be
described
in connection with FIG. 12. As shown in FIG. 12, a current B picture (900) is
to be
encoded in field mode using a backward reference picture (901) that has been
encoded in
frame mode and a forward reference picture (902) as its reference pictures.
Frames in
FIG. 12 are represented with solid vertical lines and their corresponding
fields, fl and f2,
are represented with dashed vertical lines. According to an embodiment of the
present
invention, the backward reference picture (901) can be an I, P, or B picture
that has been
encoded in frame mode. Similarly, the forward reference picture (902) can also
be an
encoded I, P, or B picture.
[0086] As shown in FIG. 12, there is a block (905) in the first field of the
current
B picture (900). According to an embodiment shown in FIG. 12, the co-located
block
(904) is coded in frame mode. According to an embodiment of the present
invention, the
encoder derives two motion vectors for the block (905) in the current B
picture (900) that
are used in the temporal prediction with motion compensation algorithm. As
shown in
FIG. 12, one of the motion vectors, MVF,1, points to the field in the forward
reference
picture (902) that has the same parity as the current block's (905) field
parity. In the
example of FIG. 12, the current block (905) is in the first field of the
current B picture
(900). The other motion vector, MVB,1, points to the field of similar parity
in the
backward reference picture (901). The two motion vectors are calculated by:
17
CA 02738329 2011-05-02
[0087] MVF,i = TRB i MV / TRD (Eqs. 7 and 8)
MVBI = (TRB; - TRD) =MV!TRD
[0088] In Eqs. 7 and 8, MV is derived by dividing the frame-based forward
motion vector of the co-located block (904) by two in the vertical direction.
This
compensates for the fact that the co-located block (904) is in frame mode
while the
current block (905) is in field mode. The subscript, i, is the field index.
The first field
has a field index of 1 and the second field's field index is 2. Thus, in the
exemplary
scenario of FIG. 12, the field index is 1 because the first field is being
encoded. TRD is
the temporal distance, approximation of the temporal distance, proportional
distance to
the temporal distance, or proportional approximation to the approximation of
the
temporal distance between the i-th field of the backward reference picture
(901) and the i-
th field of the forward reference frame (902). TRB,; is the temporal distance,
approximation of the temporal distance, proportional distance to the temporal
distance, or
proportional approximation to the approximation of the temporal distance
between the i-
th field of the current B picture (900) and the i-th field of the reference
frame of the co-
located block (904) in the backward reference picture (901). The same
equations are
used to calculate the motion vectors for the block (907) in the second field
of the current
B picture (900).
[0089] The method of direct mode vector calculation in case 4 will be
described
in connection with FIG. 13. As shown in FIG. 13, a current B picture (900) is
to be
encoded in frame mode using a backward reference picture (901) that has been
encoded
in field mode and a forward reference picture (902) as its reference pictures.
Frames in
FIG. 13 are represented with solid vertical lines and their corresponding
fields, fl and f2,
are represented with dashed vertical lines. According to an embodiment of the
present
invention, the backward reference picture (901) can be an I, P, or B picture
that has been
encoded in field mode. Similarly, the forward reference picture (902) can also
be an
encoded I, P, or B picture.
[00901 As shown in FIG. 13, there is a block (903) in the current B picture
(900)
that is to be encoded as a frame. Its motion vectors are derived from the
forward motion
vector, MVI, of its co-located block (906) in the backward reference picture
(901).
According to an embodiment of the present invention, the encoder derives two
motion
vectors for the current block (903) in the current B picture (900) that are
used in the
18
CA 02738329 2011-05-02
temporal prediction with motion compensation algorithm. The two motion vectors
are
calculated as follows:
MV F TR V I TR
[0091] F B M1 DJ (Eqs. 9 and 10)
MVB = (TRB - TRD 1) - MV1 / TRD 1
[0092] In Eqs. 9 and 10, MVI is derived by doubling the field-based motion
vector of the co-located block (906) in the first field of the backward
reference picture
(901) in the vertical direction. TRB is the temporal distance, approximation
of the
temporal distance, proportional distance to the temporal distance, or
proportional
approximation to the approximation of the temporal distance between the
current B
picture (900) and the reference frame (902) with one of its fields pointed by
the forward
motion vector of the co-located block (906). In FIG. 13, this motion vector is
labeled
MVI. TRD,I is the temporal distance, approximation of the temporal distance,
proportional distance to the temporal distance, or proportional approximation
to the
approximation of the temporal distance between the first field of the backward
reference
picture (901) and the field in the forward reference picture (902) pointed by
the forward
motion vector of the co-located block (906).
[0093] Another embodiment of the present invention extends direct mode coding
to P pictures. In AFF coding at the picture level, a P picture and its forward
reference
picture can be coded in frame mode or in field mode. Hence, in terms of frame
and field
coding modes, there can be four different combinations for a pair of
macroblocks in the P
picture and its co-located macroblock of the forward reference picture. In
case 1, both
the current macroblock and its co-located macroblock are in frame mode. In
case 2, both
the current macroblock and its co-located macroblock are in field mode. In
case 3, the
current macroblock is in field mode and its co-located macroblock is in frame
mode.
Finally, in case 4, the current macroblock is in frame mode and its co-located
macroblock
is in field mode. Blocks in P pictures only have one motion vector, a forward
motion
vector. The method of direct mode motion vector calculation for the
macroblocks that
are to be encoded is different in each of the four cases. The four methods of
direct
motion vector calculation for macroblocks in a P picture will be described in
detail
below.
[0094] In case 1, both the current P picture and its forward reference picture
are
encoded in frame mode. The forward reference picture for a block in the
current P
19
CA 02738329 2011-05-02
picture has the same picture used by its co-located block in the forward
reference picture.
The forward motion vector, MVF, of the current block is the same as the
forward motion
vector of its co-located block.
[0095] In case 2, both the current P picture and its forward reference picture
are
encoded in field mode. The motion vector in direct mode coding of a block in a
field of
the current P picture is calculated from the forward motion vector of the co-
located block
in the field with the same parity in the forward reference picture. The
forward motion
vector, MVF,,, for the block in the i-th field of the current P picture is the
same as the
forward motion vector of its co-located block in the i-th field of the forward
reference
picture.
[0096] In case 3, the current P picture is in field mode and the forward
reference
picture is in frame mode. Since the co-located block of a block in one of the
fields of the
current P picture is frame coded, the forward motion vector of a block in one
of the fields
of the current P picture is derived by dividing the co-located block's motion
vector by
two in the vertical direction.
[0097] In case 4, the current P picture is in frame mode and the forward
reference
picture is in field mode. The co-located block in the first field of the
forward reference
picture is used in calculating the forward motion vector of the block in the
current P
picture that is in frame mode. The forward motion vector, MVF, of a block in
the current
P picture in frame mode is derived by doubling the field-based motion vector
of the co-
located block in the first field of the forward reference picture in the
vertical direction.
[0098] Another embodiment of the present invention is multi-frame
interpolative
prediction mode (MFIP). MFIP is a general frame interpolative prediction
framework.
As explained previously, a B picture that is encoded in frame mode or field
mode has two
reference pictures that are encoded in frame mode or field mode. The two
reference
pictures can be both forward reference pictures, as shown in FIG. 14. FIG. 14
shows a B
picture (140) that is to be encoded that has two reference pictures. One of
the reference
pictures is a forward reference picture (141) and the other is a backward
reference picture
(142). As shown in FIG. 14, they are both in the same temporal forward
direction. The
two reference pictures can also both be in the temporal backward direction, as
shown in
FIG. 15. In FIG. 15, the B picture (140) has both its forward reference
picture (141) and
its backward reference picture (142) in the temporal backward direction. FIG.
16 shows
CA 02738329 2011-05-02
another embodiment of the present invention. As shown in FIG. 16, the B
picture (140)
can have the forward reference picture (141) in the temporal forward direction
and the
backward reference picture (142) in the temporal backward direction.
[0099] In MFIP, a prediction signal is a linear interpolation of motion
compensated signals. The prediction signal (pred) in MPIF of a B picture is
calculated
as:
[00100] pred = w,ref, + w2ref2 + d (Eq. 11)
[00101] In Eq. 11, the variables ref and ref2 are the two reference pictures.
The
variables w, and w2 are weighting factors. The variable d is set to zero by
default. The
linear interpolation coefficients wl,w2,d can be determined explicitly for
each
macroblock. The reference picture, ref, , is the reference picture closer to
the B picture
in terms of temporal distance if both ref and ref2 are forward or backward
reference
pictures. For bi-directional reference pictures, ref and ref2 are the forward
and
backward reference pictures, respectively.
[00102] Both motion vectors of a MFIP macroblock are coded relative to each
other. The motion vector of ref2, MV 2 , is generated by adding an offset DMV
to the
scaled motion vector of ref MV, using the following equation:
[00103] MV2 = TR x M' + DMV (Eq. 12)
[00104] In Eq. 12, the variable DMV is the delta motion vector and is an
offset.
The variables TR, and TR2 are the temporal distances, approximation of the
temporal
distance, proportional distance to the temporal distance, or proportional
approximation to
the approximation of the temporal distance between the current picture and the
nearest
reference picture, ref, , and the farthest reference picture, ref2 ,
respectively.
[00105] In picture level AFF, a B picture can be coded as one B frame picture
or
two B field pictures. Rules for handling MFIP in field mode, with the current
B picture
that is to be encoded in field structure are given below:
[00106] Eq. 11 is used to generate prediction signal. However, ref, and ref2
are
the fields that are indexed by reference field numbers, ref idx fwd and ref
idx bwd.
21
CA 02738329 2011-05-02
The fields ref, and ref2 can be either top or bottom fields. The default
weighting factors,
w, and w2 , are (.5, .5, 0) and (2, -1, 0), respectively.
[00107] Equation 12 is used to generate MV 2 . Since both reference pictures
are in
field structure, TR, and TR2 are determined based on the temporal distances
between the
reference and the current fields.
[00108] Code number for reference field number, ref idx fwd and ref idx bwd,
in
MFIP mode follow the known and normal convention for field picture.
[00109] The temporal distance between pictures in AFF coding can be calculated
using a variable, temporal reference (TR) or by counting the picture numbers
and
calculating their differences. An embodiment of the present invention is that
TR is
incremented by 1 per field, and wrapped by a constant (for example, 256) for
picture
level AFF. TR is in field interval. Let n be the frame index or the frame
number. The
variable n is incremented by 1 per frame. If a frame with frame index n is
encoded in
frame mode, the TR of this frame is 2n. If a frame with frame index n is
encoded in field
mode, the TR of the first field of this frame is 2n and the TR of the second
field is 2n+1.
[00110] The preceding description has been presented only to illustrate and
describe embodiments of invention. It is not intended to be exhaustive or to
limit the
invention to any precise form disclosed. Many modifications and variations are
possible
in light of the above teaching.
[00111] The foregoing embodiments were chosen and described in order to
illustrate principles of the invention and some practical applications. The
preceding
description enables others skilled in the art to utilize the invention in
various
embodiments and with various modifications as are suited to the particular use
contemplated. It is intended that the scope of the invention be defined by the
following
claims.
22
