Note: Descriptions are shown in the official language in which they were submitted.
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
APPARATUS AND METHOD FOR VIDEO MOTION COMPENSATION
TECHNICAL FIELD
The present invention generally relates to the field of video processing and
to an
apparatus for video motion compensation, and specifically relates to a video
coder and
to a video decoder for supporting motion compensation to predict frames in a
video.
The present invention relates further to a method for coding and to a method
for
decoding a video stream using motion compensation. Finally, the present
invention
relates to a computer program having a program code for performing such a
method.
BACKGROUND
In the field of video processing, and in particular in the field of hybrid
video coding
and compression, it is known to use inter and intra picture prediction as well
as
transform coding. Such hybrid video coding technologies are used in known
video
compression standards like H.261, H.263, MPEG-1, 2, 4, H.264/AVC or HEVC.
Fig. 1 shows a video coder according to the state of the art. A video coder
100
comprises an input for receiving input pictures and an output for generating a
coded
video stream. The video coder 100 is adapted to apply prediction,
transformation,
quantization, and entropy coding to a video sequence. The transformation,
quantization, and entropy coding are carried out respectively by a transform
and
quantize unit 101 and by an entropy encoder 103. Between the transform and
quantize
unit 101 and the entropy encoder 103, a scan unit 102 is provided.
The video sequence corresponds to a plurality of input pictures, wherein each
picture is
divided into blocks of a certain size that are either intra or inter coded.
The blocks of
the first picture of a video sequence are intra coded by means of an intra
prediction
unit 109. An intra picture is coded using only the information within the same
picture,
so that it can be independently decoded and it can provide an entry point in
the bit
stream for random access.
1
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
For inter coded pictures, information from previous and/or future coded
pictures,
which are called reference pictures, are used to reduce the temporal
redundancy.
Reference pictures are processed by an inverse scan, dequantize, inverse
transform unit
104, by deblocking and SAO (sample adaptive offset) filters 105 and are stored
in a
frame buffer 106. Each block of an inter coded picture is predicted from a
block of the
same size in the reference pictures.
A motion estimation is applied by a motion estimation unit 107 to select
motion data,
such as for example a motion vector (MV), a mode decision, and reference
frames. The
motion estimation unit 107 comprises as input a picture to be inter coded and
a
reference picture. Motion data are selected based on certain cost function to
minimize
the difference between the original block and its predicted block. A coder
control unit
110 selects the motion information, such as motion vector or reference frame
index,
based on the rate distortion optimization procedure.
A motion compensation carried out by a motion compensation unit 108 uses the
motion data, which are coded and transmitted as side information to the
decoder, to
generate prediction signals or motion compensated frame signals. The
difference
between the original block and its prediction, which is called residual or
prediction
error, is then transformed. The transform coefficients are usually quantized,
scanned,
entropy coded and transmitted together with the motion data.
The decoder processing, which is part of the video coder 100, is shown in the
dashed
box 111 in Fig. 1. Fig. 2 shows a video decoder 200 according to the state of
the art.
The video decoder 200 comprises, similarly to the video coder 100, an entropy
decoder
203, an inverse scan, dequantize, inverse transform unit 204, a deblocking
filter and
SAO unit 205, a frame buffer 206, a motion compensation unit 208 and an intra
prediction unit 209.
Both the video coder 100 and video decoder 200 generate identical predictions.
The
residual data is reconstructed by the inverse scan, dequantize, inverse
transform unit
204. The approximated input picture is reconstructed block-wise by adding the
residual
to the prediction. The result of the addition may be further fed into loop
filters to
2
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
smooth out artifacts. The final reconstructed picture is then store in the
picture or
frame buffer 206, which can be used for the prediction of subsequence
pictures.
The motion compensation, as for example applied in the video coder 100 of Fig.
1 and
in the video decoder of Fig. 2, is one of the most important elements in
hybrid video
coding/decoding. The motion compensation utilizes temporal information to
predict
the current block and reduces thereby the redundancy of a video. The accuracy
of
prediction depends on the similarity of the content between the reference
block and the
current block. With the change of velocity of the camera or certain objects in
a scene,
which is typically expected in action and sports movies, motion blur varies
from frame
to frame leading to reduced prediction accuracy.
The general motion compensation method from video coding standards works well
with stationary objects or moving objects of constant velocity. However, its
accuracy
is limited in case of varying motion blur. Motion blur occurs in the direction
of object
motion if an object moves during the exposure time. An accelerated object
looks
blurrier in the current frame than in the previous reference frame while a
decelerated
object appears sharper. The changing extent of blur between successive frames
generally enlarges the prediction error. Thus it results in reduced
compression
efficiency and consequently in an increased data rate for the residual of
inter-predicted
blocks.
In this context, it is known to reduce the prediction error by filtering the
reference
frame for varying motion blur compensation. For example, it is known to
predefine a
plurality of blurring filters and to apply these blurring filters to a
reference frame
stored in the frame buffer 106. By iteratively testing the performance of each
filter in
terms of rate distortion optimization, the best blur filter is selected by the
video coder.
For allowing the video decoder to decode the generated bit stream, the
blurring filter
that has been selected by the video coder is signaled to the decoder as side
information
within the bit stream.
Nevertheless, even if such a technique can lead to a reduction of the
prediction error in
case of a blur variation between a reference frame and a current frame being
inter
coded, the signaling of the selected blurring filter from the video coder to
the video
3
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
decoder requires additional bits within the encoded bit stream. This
additional
signaling in turn decreases the number of bits that can be used for encoding
the
prediction error in the encoded bit stream.
SUMMARY
Having recognized the above-mentioned disadvantages and problems, the present
invention aims to improve the state of the art. In particular, the object of
the present
invention is to provide a video coder, a coding method, a video decoder, and a
decoding method for an improved coding and decoding of a video stream of
subsequent frames.
The present invention particularly intends to improve the quality of the inter
predictive
coding by improving the accuracy of a prediction signal. The invention also
intends to
reduce the number of signaling bits within the encoded bit stream.
The above-mentioned object of the present invention is achieved by the
solution
provided in the enclosed independent claims. Advantageous implementations of
the
present invention are further defined in the respective dependent claims.
A first aspect of the present invention provides a video coder for predictive
coding a
video stream of subsequent frames according to motion compensation into an
encoded
video bit stream. The video coder comprises a frame buffer adapted to store at
least
one reference frame of the video stream, said reference frame being different
from a
current frame of the video stream. The video coder comprises a motion
estimation unit
adapted to generate a motion vector by estimating motion between a reference
block of
the reference frame and a current block of the current frame. The video coder
comprises a filter selection unit adapted to select one of a plurality of
filters depending
on the orientation of the motion vector. The video coder comprises a filtering
unit
adapted to generate a filtered reference block by applying the selected filter
to the
reference block. The video coder comprises a motion compensation unit adapted
to
generate a filtered prediction block of the current block on the basis of the
current
block and the filtered reference block.
4
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
Thereby, while the prior art requires additional information to be coded in
the bit
stream for a video stream of subsequent frames, the present invention makes
use of the
motion vector, which is already part of the bit stream for the inter-
prediction support.
Therefore, no additional signalling is necessary for supporting the filtering
of the
reference frame/block. In the following, the features "frame" and "block" are
interchangeable since the filtering and the prediction can be applied to a
block of a
frame or even to the frame itself
In a first implementation form of the video coder according to the first
aspect, the
motion compensation unit is adapted to further generate a non-filtered
prediction block
of the current block on the basis of the current block and the reference
block, and to
output, from among the filtered prediction block and the non-filtered
prediction block,
the prediction block that minimizes a prediction error, said prediction error
being the
difference between the current block and the respective prediction block.
Thereby, a rate distortion optimization can be done so that the filtered or
the non-
filtered reference block is used for generating the encoded prediction error.
In a further implementation form of the video coder according to the first
aspect, the
motion estimation unit is adapted to generate a second motion vector by
estimating
motion between the filtered reference block and the current block. Depending
on a
comparison of the second motion vector and of motion vector between the
reference
block and the current block, the filtered reference block is taken into
consideration or
disregarded for generating the encoded video bit stream.
Thereby, the prediction of a block can be more accurate by the assessment of a
second
motion estimation.
In a further implementation form of the video coder according to the first
aspect, the
video coder comprises encoding means adapted to generate the encoded video bit
stream on the basis of the prediction error corresponding to the prediction
block
outputted by the motion compensation unit.
Thereby, the efficient predictive coding can be implemented.
5
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
In a further implementation form of the video coder according to the first
aspect, the
video coder comprises encoding means adapted to generate the encoded video bit
stream so that the encoded video bit stream comprises motion information and
filter
information. The motion information comprises the motion vector, and the
filter
information specifies whether the encoded video bit stream is based on the
prediction
error between the current block and the filtered prediction block or on the
prediction
error between the current block and the non-filtered prediction block.
Thereby, the filter information allows for the video decoder to rapidly
determine
whether or not the additional filtering of the reference block has to be done
for
performing the predictive decoding.
In a further implementation form of the video coder according to the first
aspect, the
encoding means are adapted to add in the encoded video bit stream the filter
information at a block level for each predictive coded current block, at a
frame level, at
a GOP level, at a PPS level or at an SPS level.
Thereby, it is possible to adaptively set the filter information to a desired
granularity so
that the signalling can be optimized.
In a further implementation form of the video coder according to the first
aspect, the
video coder comprises a transforming and quantizing unit adapted to transform
the
prediction error by means of a linear spatial transform so as to generate
transform
coefficients, and to quantize the transform coefficients, and a scanning unit
and an
entropy encoder respectively adapted to scan and entropy code the quantized
transform
coefficients so as to generate the encoded video bit stream.
Thereby, the number of bits for transmitting the prediction error in the video
bit stream
can be further reduced.
In a further implementation form of the video coder according to the first
aspect, the
filter selection unit is adapted to derive the angle of the motion vector with
respect to
6
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
the positive horizontal axis, and to select said one of a plurality of filters
depending on
the angle.
Thereby, the selection of a filter to be applied to the reference block is
simplified and
can be performed without any further signalling bits.
In a further implementation form of the video coder according to the first
aspect, the
video coder is a video coder for single-layer coding of the video stream.
In a further implementation form of the video coder according to the first
aspect, the
filter selection unit is adapted to select one of a plurality of blurring
and/or deblurring
filters depending on the orientation of the motion vector.
Thereby, it is possible to add or remove blur from the reference block in
cases of
motion acceleration or deceleration within the video stream.
A further aspect of the present invention provides a hybrid video encoder for
intra- and
inter-frame prediction comprising such a video coder.
A further aspect of the present invention provides a method for predictive
coding a
video stream of subsequent frames according to motion compensation into an
encoded
video bit stream. The method comprises: a step of storing at least one
reference frame
of the video stream, said reference frame being different from a current frame
of the
video stream, a motion estimation step comprising generating a motion vector
by
estimating motion between a reference block of the reference frame and a
current
block of the current frame, a step of selecting one of a plurality of filters
depending on
the orientation of the motion vector, a step of generating a filtered
reference block by
applying the selected filter to the reference block, a motion compensation
step
comprising generating a filtered prediction block of the current block on the
basis of
the current block and the filtered reference block.
A further aspect of the present invention provides a video decoder for
decoding an
encoded video bit stream obtained by predictive coding a video stream of
subsequent
frames according to motion compensation, wherein the encoded video bit stream
7
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
comprises a predictive encoded current frame and an encoded motion vector
reflecting
motion between a reference block of the reference frame and a current block of
the
current frame, said reference frame being different from the current frame.
The video
decoder comprises a decoder adapted to decode the encoded reference frame and
the
encoded motion vector so as to generate a reference frame and a motion vector,
a
frame buffer adapted to store the reference frame, a filter selection unit
adapted to
select one of a plurality of filters depending on the orientation of the
motion vector, a
filtering unit adapted to generate a filtered reference block by applying the
selected
filter to the reference block, a motion compensation unit adapted to generate
the
current block of the predictive encoded current frame on the basis of the
motion vector
and the filtered reference block.
Thereby, while the prior art requires additional information to be coded in
the bit
stream for a video stream of subsequent frames, the present invention makes
use of the
motion vector, which is already part of the bit stream for the inter-
prediction support.
Therefore, no additional signalling is necessary for supporting the filtering
of the
reference frame/block. Also, no information beyond the current frame is needed
to
select the filter and apply the selected filter to the reference frame/block.
In an implementation form of the video decoder according to the further
aspect, the
encoded video bit stream comprises a filter information related to the current
block.
The decoder is adapted to decode the filter information. The motion
compensation unit
is adapted to, depending on the value of the filter information, generate the
current
block of the predictive encoded current frame on the basis of the motion
vector and the
filtered reference block, or on the basis of the motion vector and the
reference block,
which corresponds to the non-filtered reference block i.e. to the reference
block that
has not been processed by the selected filter.
In an implementation form of the video decoder according to the further
aspect, the
filter selection unit is adapted to derive the angle of the motion vector such
as with
respect to the positive horizontal axis, and to select said one of a plurality
of filters
depending on the angle.
8
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
In an implementation form of the video decoder according to the further
aspect, the
video decoder is a video decoder for single-layer coding of the video stream.
In an implementation form of the video decoder according to the further
aspect, the
filter selection unit is adapted to select one of a plurality of blurring
and/or debluning
filters depending on the orientation of the motion vector.
A further aspect of the present invention provides a hybrid video decoder for
intra- and
inter-frame prediction comprising such a video decoder.
A further aspect of the present invention provides a method for decoding an
encoded
video bit stream obtained by predictive coding a video stream of subsequent
frames
according to motion compensation. The encoded video bit stream comprises at
least
one encoded reference frame, a predictive encoded current frame, said
reference frame
being different from the current frame, and an encoded motion vector
reflecting motion
between a reference block of the reference frame and a current block of the
current
frame. The method comprises decoding the encoded reference frame and the
encoded
motion vector so as to generate a reference frame and a motion vector, storing
the
reference frame, selecting one of a plurality of filters depending on the
orientation of
the motion vector, generating a filtered reference block by applying the
selected filter
to the reference block, motion compensation comprising generating the current
block
of the predictive encoded current frame on the basis of the motion vector and
the
filtered reference block.
A further aspect of the present invention provides a computer program having a
program code for performing such a coding and/or decoding method when the
computer program runs on a computing device.
It has to be noted that all devices, elements, units and means described in
the present
application could be implemented in the software or hardware elements or any
kind of
combination thereof All steps which are performed by the various entities
described in
the present application as well as the functionalities described to be
performed by the
various entities are intended to mean that the respective entity is adapted to
or
configured to perform the respective steps and functionalities. Even if, in
the following
9
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
description of specific embodiments, a specific functionality or step to be
full formed
by eternal entities not reflected in the description of a specific detailed
element of that
entity which performs that specific step or functionality, it should be clear
for a skilled
person that these methods and functionalities can be implemented in respective
software or hardware elements, or any kind of combination thereof
BRIEF DESCRIPTION OF DRAWINGS
The above aspects and implementation forms of the present invention will be
explained in the following description of specific embodiments in relation to
the
enclosed drawings, in which
Fig. 1 shows a video coder according to the state of the art.
Fig. 2 shows a video decoder according to the state of the art.
Fig. 3 shows a video coder according to an embodiment of the present
invention.
Fig. 4 shows a video decoder according to an embodiment of the present
invention.
Fig. 5(a) shows an embodiment of motion blur compensation according to the
present
invention.
Fig. 5(b) shows an embodiment of motion blur compensation according to the
present
invention.
Fig. 6 shows examples of motion vectors obtained by the embodiment of Fig.
5(a).
Fig. 7 shows an embodiment of a filter according to the present invention.
Fig. 8 shows an embodiment of a filter according to the present invention.
Fig. 9 shows an embodiment of filter coefficients according to the present
invention.
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
DETAILED DESCRIPTION OF EMBODIMENTS
Fig. 3 shows a video coder according to an embodiment of the present
invention, and
particularly a hybrid video coder 300 for predictive coding a video stream of
subsequent frames according to motion compensation into an encoded video bit
stream.
The video coder 300 comprises particularly a frame buffer 306, a motion
estimation
unit 307, a filter selection unit 313, a filtering unit 314 and a motion
compensation unit
308.
The frame buffer 306 is adapted to store at least one reference frame of the
video
stream. Said reference frame is different from a current frame of the video
stream. In
the context of the invention, the current frame is a frame of the video stream
that is
currently encoded. The current frame is encoded using an inter coding
technique, i.e.
the current frame is predicted from one or a plurality of reference frames.
The
reference frame can be a previous frame, i.e. a frame that is located prior to
the current
frame within the video stream of subsequent frames. Alternatively if forward
prediction is used, the reference frame can be a future frame, i.e. a frame
that is located
after the current frame. In case of a plurality of reference frames, at least
one can be a
previous frame and at least one of them can be a future frame. A reference
frame can
be intra coded, i.e. can be coded without using any further frame and without
any
dependence on other frames, so that it can be independently decoded and it can
serve
as entry point for random video access.
The motion estimation unit 307 is adapted to generate a motion vector by
estimating
motion between a reference block of the reference frame and a current block of
the
current frame. Fig. 3 accordingly shows that the motion estimation unit 307
comprises
an input for the reference frame stored in the frame buffer 306, and an input
for an
input picture that corresponds to the current frame.
The motion vector is generated by using a block motion estimation. The
reference
block and the current block are preferably a respective area or sub-area of
the reference
frame and the current frame. Such a block may have a regular, rectangular
shape or an
11
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
irregular shape. Alternatively, the blocks can have the same size as the
frames. Both
the current block and the reference block have the same size. The size of the
blocks is
defined by means of a block mode. A block can correspond to a coding unit (CU)
that
is a basic coding structure of the video sequence of a pre-defined size,
containing a
part of a frame, e.g. 64x64 pixels.
The current block of the current frame being inter coded is predicted from the
reference block of the reference frame. The encoding process for the inter
frame
prediction comprises choosing motion data that comprise a reference index and
the
motion vector to be applied for predicting the current block. The reference
index is a
index of the reference block and/or of the reference frame.
The filter selection unit 313 is adapted to select one of a plurality of
filters depending
on the orientation of the motion vector. The filtering unit 314 is adapted to
generate a
filtered reference block by applying the selected filter to the reference
block. The
motion compensation unit 308 is adapted to generate a filtered prediction
block of the
current block on the basis of the current block and the filtered reference
block.
Fig. 3 shows accordingly that the motion estimation unit 307 transmits the
motion
vector to the filter selection unit 313. The output of the filter selection
unit 313
identifies the selected filter. The filtering unit 314 then applies the filter
selected by the
filter selection unit 313 to the reference block of the reference frame stored
in the
frame buffer 306, so as to generate a filtered reference block transmitted to
the motion
compensation unit 308. The motion compensation unit 308 then generates the
filtered
prediction block on the basis of the current block of the current frame and
the filtered
reference block. Preferably, the filtered prediction block is generated on the
basis of
the current block, the motion vector, and the filtered reference block.
Particularly, for the hybrid video coding, said motion estimation 307 is
performed
during encoding to find the motion vector pointing to the best prediction unit
in a
reference frame. Motion compensation 308 uses motion data, which are coded and
transmitted as side information to the decoder, to generate prediction
signals. The
prediction signals refer to prediction blocks that can be combined to obtain a
prediction
frame.
12
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
The video coder 300 of Fig. 3 comprises further units similarly to the video
coder 100
of Fig. 1. Particularly, the video coder 300 comprises similar units that are
a transform
and quantize unit 301, a scan unit 302, an entropy encoder 103, an inverse
scan,
dequantize, inverse transform unit 304, a deblocking and SAO filters unit 305.
The
video coder 300 comprises further similar units that are the motion estimation
unit 307,
the motion compensation unit 308, the intra prediction 309 that is responsible
for the
intra prediction of for example the reference frame, and the coder control
310.
In the hybrid video coder 300 shown in Fig. 3, based on the motion vector
generated
during motion estimation 307 for e.g. a block or coding unit, a pre-defined
filter is
selected 313 according to the orientation of said motion vector, and
particularly
according to the angle of said motion vector with respect to the horizontal
axis. The
angle is calculated based on the motion vector of the current block. Then, the
selected
filter is applied to the reference frame stored in the frame buffer 306.
In the video encoder 300, the motion compensation unit 308 is adapted to
further
generate a non-filtered prediction block of the current block on the basis of
the current
block and the reference block, and particularly on the basis of the current
block, the
motion vector and the reference block. The reference block taken into
consideration
here is the non-filtered reference block of the reference frame stored in the
frame
buffer 306. The motion compensation unit 308 is moreover adapted to output,
from
among the filtered prediction block and the non-filtered prediction block, the
prediction block that minimizes a prediction error, said prediction error
being the
difference between the current block and the respective prediction block.
This means that the rate distortion performance of the filtered prediction
block and of
the non-filtered prediction block will be compared. The option which presents
the
better rate distortion performance is then chosen. This comparison and this
choice can
be carried out by the coder control 310. The coder control 310 can select the
motion
information, such as motion vector and reference index, and the filter
information like
the blurring flag, based on the rate distortion optimization procedure.
13
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
The entropy encoder 303 is adapted to generate the encoded video bit stream so
that
the encoded video bit stream comprises motion information and filter
information. The
motion information comprises the motion vector selected by the unit, and
advantageously also the reference index that is a index to identify the
reference block
and/or reference frame used for predictive coding of the current block and/or
current
frame. The filter information specifies whether the encoded video bit stream
is based
on the prediction error between the current block and the filtered prediction
block or
on the prediction error between the current block and the non-filtered
prediction block.
Particularly, the filter information specifies whether the encoded bit stream
has been
generated based on the filtered prediction block and the non-filtered
prediction block.
The filter selection unit 313 is adapted to select one of a plurality of
blurring or
deblurring filters depending on the orientation of the motion vector. The
filters can be
pre-determined filters that reside on both the coder 300 and a corresponding
decoder as
the decoder 400 shown in Fig. 4. Alternatively, the definition of the filters
can be
transmitted from the coder 300 to the decoder 400 by corresponding filter
signalling in
the bit stream. This means that either fixed filters are used for all video
sequences,
such that there is no need to send the pre-defined filters, or customized
filters are
defined for e.g. difference categories of videos. In the latter case, the
filters can be sent
through an SET (supplemental enhancement information) message to the decoder.
The filter information transmitted within the bit stream can be a filter flag.
For
example, a blurring flag can be defined for each block which has been coded
with inter
prediction i.e. according to an inter mode. The flag at block level can be
used to
indicate whether a blurring or deblurring filter has been used for a current
block. In
addition, a blurring flag added in a slice header can be used to indicate
whether a
blurring/deblurring filter has been used for the current slice. Such a flag in
SPS
(sequence parameter set) can be used to indicate whether blurring/deblurring
filter has
been used for a coded video sequence. Such a flag in PPS (picture parameter
set) can
be used to indicate whether blurring/deblurring filter has been used for a
coded picture.
Alternatively, the blurring flag can be used for indicating whether a blurring
filter has
been used for the current block and a deblurring flag can be used for
indicating
whether a deblurring filter has been used for the current block.
14
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
In addition to the known standard coding methods that are intra-frame and
inter-frame
prediction (incl. skip), the present invention proposes a further mode that is
the motion
blur compensation mode for hybrid video coding. This motion blur compensation
can
be considered as a complement of inter mode (incl. skip) of hybrid video
coding, such
as HEVC, during the coding of a certain block. The invention proposes to
create
temporal reference frames by filtering 314 the reconstructed frame stored in
the frame
buffer 306 with a filter of angle 0 that is derived from the motion vector
obtained by
the motion estimation 307.
These filtered reference frames and/or reference blocks are then used for
motion blur
compensation. In case the blurring filter is applied to the reference frame,
the filtered
reference frame and the original reference frame stored in the frame buffer
306 share
the same reference index. Namely, a reference index will refer to either a
reconstructed
frame in the general motion estimation in hybrid video coding, or a blurred
temporal
frame used for the motion estimation of the proposed motion blur compensation.
The motion estimation of the motion blur compensation according to the present
invention is executed after the traditional motion estimation of hybrid video
coding,
such as motion estimation in HEVC.
The blurring flag can be defined for each block or coding unit that is coded
with inter
mode (incl. skip) for all partition sizes (2Nx2N, 2NxN, 2NxNu, 2NxNd, ...).
The
decoder 400 is able to determine whether the reference frame is blurred during
reconstruction of a CU and to replicate the blurring of reference frame using
the angle
0 derived from the encoded MV. The blurring flag is considered within the rate
distortion (RD) optimization.
The flag at block or coding unit level can be used to indicate whether a
blurring/debluning filter has been used for a coding unit. In addition
thereto, a flag in
SP S can be used to indicate whether blurring/debluning filter has been used
for a
coded video sequence. A flag in PPS can also be used to indicate whether
blurring/debluning filter has been used for a coded picture.
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
Fig. 4 shows a video decoder according to an embodiment of the present
invention.
This video decoder 400 corresponds to the video coder 300 in that it comprises
the
elements of the video coder 300 located in the dashed box 311 in Fig. 3. The
decoder
400 is adapted to decode the bit stream generated by the video coder 300, and
both the
decoder 400 and the coder generate identical predictions.
The video decoder 400 is a decoder for decoding an encoded video bit stream
obtained
by predictive coding a video stream of subsequent frames according to motion
compensation. The encoded video bit stream comprises a predictive encoded
current
frame, which corresponds to the prediction error, and an encoded motion vector
reflecting motion between a reference block of the reference frame and a
current block
of the current frame, said reference frame being different from the current
frame.
The video decoder 400 comprises a decoder 403, a frame buffer 406, a filter
selection
unit 413, a filtering unit 414, and a motion compensation unit 408. The
decoder 403 is
adapted to decode the encoded reference frame and the encoded motion vector so
as to
generate a reference frame and a motion vector. The frame buffer 406 is
adapted to
store the reference frame. The filter selection unit 413 is adapted to select
one of a
plurality of filters depending on the orientation of the motion vector. The
filtering unit
414 is adapted to generate a filtered reference block by applying the selected
filter to
the reference block. The motion compensation unit 408 is adapted to generate
the
current block of the predictive encoded current frame on the basis of the
motion vector
and the filtered reference block.
Further on, the encoded video bit stream comprises a filter information
related to the
current block. The decoder 403 is adapted to decode the filter information,
and the
motion compensation unit 408 is adapted to, depending on the value of the
filter
information, generate the current block of the predictive encoded current
frame on the
basis of the motion vector and the filtered reference block, or on the basis
of the
motion vector and the reference block. Said reference block in fact
corresponds to the
non-filtered reference block i.e. to the reference block that has not been
processed by
the selected filter.
16
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
The filter selection unit 413 is adapted to derive the angle of the motion
vector with
respect to the positive horizontal axis, and to select said one of a plurality
of filters
depending on the angle.
Fig. 4 shows that the coded video stream, i.e. the bit stream, is fed to the
entropy
decoder 403 that decodes the reference frame and the prediction error for the
current
block/frame via the inverse scan, dequantize, inverse transform unit 404 and
the
deblocking and SAO filters 405. The reference frame is stored in the frame
buffer 406.
On the other hand, the decoder 403 transmits the motion information and filter
information, which comprise respectively the motion vector and the
blurring/deblurring flag, to the filter selection unit 413 and to the motion
compensation
unit 408.
Filter information, such as the blurring/deblurring flag, can be defined for
each block
or coding unit which has been coded with inter mode (incl. skip) for all
partition sizes
including e.g. 2Nx2N, 2NxN, 2NxNu, 2NxNd, ... . The blurring/deblurring flag
might
be present in slice header, PPS or SPS. The decoder 400 is able to determine
whether
the reference frame is blurred during reconstruction of a block and to
replicate the
blurring of reference frame using the angle 0 derived from the encoded motion
vector.
In order to make sure that decoded frames in the decoder 400 are identical to
the
reconstructed ones, the same filter is applied to the current block by the
decoder 400
and by the coder 300, the filter being determined by using the orientation of
the
decoded motion vector. Then motion blurring filter is applied to the reference
frame/block. Motion compensation utilizes the blurring filtered signal
depending on
the filter information, and particularly if the blurring/deblurring flag
indicates that a
filter, e.g. blurring filter, has been applied in the encoder 300.
Two embodiments of the proposed motion compensation i.e. motion blur
compensation at the encoder are explained in the followings.
Fig. 5(b) shows an embodiment of motion blur compensation according to the
present
invention. This embodiment comprises one motion estimation in that, when
encoding
500 the current block or coding unit (CU), a motion estimation 501 is
performed by the
motion estimation unit 307 to generate a motion vector by estimating the
motion
17
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
between the reference block and the current block. During a filter generation
503, the
filter selection unit 313 selects the filter corresponding to the orientation
of the motion
vector, i.e. to the angle 0 of the motion vector with respect to e.g. the
horizontal. The
range [0, ir] for the angle 0 is subdivided into sub-ranges [0, Oil, [Of, 02]
... [On-i, On],
[On, 70, with n being an integer value, and each sub-range is associated with
one pre-
defined filter.
Once the filter is generated or selected, this filter is applied to the
reference block or
frame for generating a filtered reference block/frame like e.g. a blurred
reference
block/frame 504. Fig. 5(b) shows that the filtered reference block/frame is
generated
for a filter flag, i.e. a blurring flag, being true 502, while the reference
block/frame is
not filtered for a filter flag, i.e. a blurring flag, being false 505. The
motion
compensation 506 is applied to both the filtered reference block/frame and to
the non-
filtered reference block/frame. Based on a following rate distortion
optimization 507, it
is decided if the filtered or the non-filtered reference block is used for
generating the
prediction error.
Fig. 5(a) shows a further embodiment of motion blur compensation according to
the
present invention. In contrast to the embodiment of Fig. 5(b), the blurred
reference
block/frame 504 is not sent to the motion compensation 506 but is used for
carrying
out a second motion estimation.
Accordingly, the motion estimation unit 307 is adapted to carry out a second
motion
estimation 510 on the basis of the filtered reference block 504, and thus to
generate a
second motion vector by estimating motion between the filtered reference block
504
and the current block. A comparison 511 is then carried out between the second
motion
vector generated by the second motion estimation 510, and the first motion
vector,
generated by the first motion estimation 501, between the reference block and
the
current block. Depending on this comparison 511, the filtered reference block
is taken
into consideration or disregarded for generating the encoded video bit stream.
So, the embodiment of Fig. 5(a) comprises two rounds of motion estimation 501,
510,
the second motion estimation 510 being performed based on the filtered
reference
block/frame. The filtering/blurring method is thereby ignored 512 in case the
first and
18
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
second motion vectors are not located in the same defined angular sector
according to
a sector-based algorithm.
During the coding 501 of a certain block or coding unit, one element of the
sector-
based algorithm in the encoding process is the validation check of the
predicted motion
vector, which corresponds to the second motion vector, after a second motion
estimation 510 is made based on the filtered/blurred reference block/frame.
The validation check has following three criteria:
1) the first motion vector and second motion vector are in same sector.
2) the second motion vector is non-zero vector.
3) the motion vector is inside defined search range, i.e. 64 in horizontal
and vertical
directions.
Fig. 6 shows examples of motion vectors obtained by the embodiment of Fig.
5(a)
using the sector-based algorithm. Fig. 6 shows four different angular sectors
being [0,
R/4], [R/4, R/2], [R/2, 3R/4], and [37r/4, 7r]. The first motion vector
obtained by the first
motion estimation 501 is labeled "1. MV" and is located in the first sector.
According to a first example, the second motion estimation 510 is then applied
and
results in a second motion vector labeled "valid 2. MV". Since the second
motion
vector is located in the same sector as the first motion vector, the blurring
flag is true
513. This means that a motion compensation based on the filtered reference
block is
possible. In this case, a motion compensation 506 is performed for both the
filtered
reference block and the non-filtered reference block from the frame buffer. A
following rate distortion optimization is carried out to select either the
filtered or the
non-filtered version.
According to a second example, the second motion vector labeled "invalid 2.
MV" is
located in a different sector than the sector of the first motion vector. As a
consequence, the filtering is ignored 512 and a motion compensation is applied
only
for the non-filtered reference block.
19
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
In other words, after the blurring operation 504, the blurring flag is only
set to true 513
in case the first motion vector and second motion vector are located in the
same
defined sector. Otherwise, the blurring flag is set to false 512. In contrast,
in the
embodiment of Fig. 5(b) no second round motion estimation is needed. The
blurring
flag is set to true 502 if blurring 504 is performed. These embodiments are
coder-
related embodiments on how to decide whether blurring filter should be used.
Coming back to the embodiment of Fig. 5(a), the first criterion preferably
means that a
representative direction is defined for each predefined sector. The
representative
direction preferably lies at the middle of the sector. For example, Fig. 6
shows the
representative direction for the first sector, said direction being labeled
"Rep. MV".
The blurring filters are generated based on those representative directions.
The number
of sectors can be given according to different requirement of accuracy and the
same
number of corresponding blurring filters is calculated before coding starts.
If the first and second motion vectors belong to the same sector, the same
representative direction as well as the same blurring filter is used by
encoder and
decoder. The decoder only has to find out the direction carried by the
transmitted
motion vector and does the exact filtering to have the same reconstructed
block or
coding unit.
In the example of Fig. 6, each filter covers an angle of R/4=45 if the number
of four
sectors is chosen. Since the first motion vector falls in first sector, the
representative
motion vector points to the direction in the middle of the sector. If the
second motion
vector "valid 2. MV" falls in the same sector, the value true is given by the
validation
check and the decoder finds the same representative motion vector.
Preferably, the second motion vector is then transmitted to the decoder. This
means
that the video coder comprises encoding means 303 adapted to generate the
encoded
video bit stream so that the encoded video bit stream comprises motion
information,
wherein the motion information comprises the second motion vector.
If the second motion vector "invalid 2. MV" leads to another representative
motion
vector for the second vector, the value false will be given by the validation
check.
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
Advantageously, a second criterion consists in that the second motion vector
is a non-
zero vector. This criterion is based on the fact that motion blur occurs if
there is a
relative motion between objects and cameras during exposure. A zero motion
vector
means that no motion is found for the block or coding unit, thus there should
be no
blur and the validation check gives false as a result. No filter is then
applied 314 to the
reference block. This second criterion can be considered in addition to the
first
criterion or alone.
Finally, according to the third criterion, the second motion vector should be
in the
search range of motion estimation for the validation check to be true. This
third
criterion can be considered in addition to the first and/or second criterion.
With the rise of the number of sectors, the direction of blurring will be more
accurate
while the validation rate of the second motion vector decreases, which may
lead to a
drop in the coding gain and efficiency. There is hence a trade-off between
filter
accuracy and validation rate.
Compared to the embodiment of Fig. 5(a), the encoding structure of the
embodiment
of Fig. 5(b) is simplified by applying the same motion vector to motion
compensation
(MC) based on filtered reference instead of a second motion estimation and a
validation check based on sector. The embodiment of Fig. 5(b) thus can reduce
the run
time. Indeed this embodiment spares not only one additional motion estimation,
but
also the region of blurring. In place of blurring the whole area covered by
standard
motion estimation, it is only requested to blur the block or coding unit on
the reference
frame pointed by the motion vector and its surrounding which will be used by
interpolation later. For the same reason as above in the embodiment of Fig.
5(a), also
here a zero motion vector is excluded from motion blur compensation.
The embodiment of Fig. 5(a) introduces a second motion estimation on filtered
reference frames and a sector-based validation check to make sure that encoder
and
decoder share an identical blurring direction. In the embodiment of Fig. 5(a),
the
prediction of a block or coding unit becomes more accurate with an increase of
the
number of filters, i.e. with an increased number of sectors, since the
generated filters
21
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
are more precise in directions. On the other hand, a higher number of filters
means that
the range of valid angles for the second motion vector decreases and the
motion blur
compensation is ignored more often. The use of 9 filters is a good compromise.
In the followings, an embodiment of blurring filters that can be used for
motion blur
compensation will be shown.
A commonly used linear uniform motion blur point spread function (P SF) in
continuous time domain is described with a line segment L and an angle 0 with
respect
to the horizontal axis, as given in the following formula:
1
h(x,y) = -1,' \ix
1
0, 2+y 2 <
_ 2, x = tan
otherwise (1)
wherein L is proportional to motion speed and exposure duration, 0 indicates
the
motion direction and (x,y) is the location in Cartesian coordinate system.
This function
is e.g. known from "Photo consistent motion blur modeling for realistic image
synthesis", H. Lin and C. Chang, PSIVT 2006.
The discrete version of equation (1) is acquired by considering a bright spot
traversing
across the sensors covered by the line segment during the exposure time with
constant
velocity, as known from "Parametric Blur Estimation for Blind Restoration of
Natural
Images: Linear Motion and Out-of-Focus", J. Oliveira, M. Figueiredo, and J.
Bioucas-
Dias, Image Processing, IEEE Transactions on Image Processing, 2014. Each
coefficient of the blur kernel is proportional to the time spent on each
sensor element.
With the assumption of a constant motion, the filter coefficients are given by
the
normalized length of the intersection of the line segment with each pixel in
the grid, as
illustrated by Fig. 7. Said Fig. 7 shows a discrete version of linear motion
blur with
0=arctan 4/5.
In case of varying motion blur, a filtered reference may improve the coding
performance. It is assumed that motion blur as well as the change of motion
blur can
be described by equation (1). Motion can be considered constant for the
exposure time
of a single frame. Since the time interval between two nearby frames is only
0.02
22
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
seconds for a 50 fps video sequence, we suggest that the change of motion blur
extent
is small in most cases. Therefore we employ a fixed extent of 3 pels to
describe the
phenomenon of variation in blurring, i.e., L = 3. Hence the two dimensional
blurring
filter for the luminance component has a dimension of 3x3.
The other degree of freedom for a blurring filter is the angle 0. It is
derived from the
motion vector used in hybrid video coding, such as HEVC. Hence a standard
hybrid
video coding motion search is executed before the blurring filter is
established. The
blurred reference frame is generated by using the directions of the motion
vectors.
The general discrete version of the 3x3 filter has 5 different coefficients a,
b, c, d and e
due to symmetry and is calculated according to an arbitrary angle 0 using Fig.
8. The
suggested filter is a low-pass filter. As an example, filter coefficients for
0 = 15 and 0
= 60 are listed in Fig. 9.
In an embodiment, we will only blur the luminance component of the reference
by
using such a 3x3 filter if a test sequence has the 4:2:0 format. Chrominance
pixels lie
at every other pixel position of the luminance component [6]. With blur extent
of only
3 pels, three neighboring luminance pixels but only 1 chrominance pixel have
to be
considered. A neighbored chrominance pixel is out of the influence range of
blurring,
hence no filter is required. In case of a sequence of 4:4:4 chroma subsampling
, the
same filter for luminance component will be applied for chrominance
components.
The motion blur model proposed and the proposed blurring filter are exemples.
Other
blurring filter can be apply according to the proposed invention as well, in
which the
motion vector of the current coding unit/block is used to select pre-defined
filters for
coding efficiency improvement in hybrid video coding.
The present invention has been described in conjunction with various
embodiments as
examples as well as implementations. However, other variations can be
understood and
effected by those persons skilled in the art and practicing the claimed
invention, from
the studies of the drawings, this disclosure and the independent claims. In
the claims as
well as in the description the word "comprising" does not exclude other
elements or
steps and the indefinite article "a" or "an" does not exclude a plurality. A
single
23
CA 02986613 2017-11-21
WO 2016/184527
PCT/EP2015/061278
element or other unit may fulfill the functions of several entities or items
recited in the
claims. The mere fact that certain measures are recited in the mutual
different
dependent claims does not indicate that a combination of these measures cannot
be
used in an advantageous implementation.
24
