Note : Les descriptions sont présentées dans la langue officielle dans laquelle elles ont été soumises.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
1
Audio decoder having a bandwidth extension module with an energy
adjusting module
Description
SBR (Spectral Band Replication), like other bandwidth extension techniques,
is meant to encode and decode spectral high band parts of audio signals on
top of a core coder stage. SBR is standardized in [IS009] and used jointly
with AAC in the MPEG-4 Profile HE-AAC, which is employed in various ap-
io plication standards, e. g. 3GPP [3GP12a], DAB+ [EBU10] and DRM
[EBU12].
State of the art SBR decoding in conjunction with AAC is described in
[IS009, section 4.6.18].
Fig. 1 illustrates the state of the art SBR decoder which comprises an analy-
sis and a synthesis filterbank, SBR data decoding an HF generator and an
HF adjuster:
= In the state-of-the-art SBR decoding, the output of the core coder is a low-
pass filtered representation of the original signal. It is the input xpcm_in
to the
QMF analysis filterbank of the SBR decoder.
= The output of this filterbank xQMF_ ana is handed over to the HF
generator,
where the patching takes place. Patching basically is a replication of the low-
band spectrum up into the high-bands.
= The patched spectrum XHF_patched is now given to the HF adjuster,
together
with the spectral information of the high-bands (envelopes), obtained from
the SBR data decoding. Envelope information will be Huffman decoded, then
differentially decoded and finally de-quantized in order to obtain the
envelope
data (see Fig. 2). The obtained envelope data is a set of scale factors which
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
2
covers a certain amount of time, e. g. a full frame or parts of it. The HF ad-
juster properly adjusts the energies of the patched high-bands in order to
match as good as possible with the original high-band energies at encoder
side for every band k. Equation 1 and Fig. 2 clarify this:
gsbr [k] = ERef [k] I EEsfAvg II
EAdj [kj = EEst [k] x gsbr [k] (1)
where
ERef [k] denotes the energy for one band k, being transmitted in encoded form
in the SBR bitstream;
EEst [k] denotes the energy from one high-band k, patched by the HF genera-
tor;
EEstAvg [I] denotes the averaged high-band energy inside of one scale factor
band I, being defined as a range of bands between a start band lestart and a
stop band kistop:
x,kistop
EEstAvg ts"Ti hlt=ki tart EEst (k) (2)
EAdj [lc] denotes the energy from one high-band k, adjusted by the HF adjust-
er, using gainsbr;
gsbr[k] denotes one gain factor, resulting from the division shown in equation
(1).
= The Synthesis QMF filterbank decodes the processed QMF,samples
xHF_adj to PCM audio
xpcm_out.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
3
If the reconstructed spectrum has a lack of noise, which was present in the
original high-bands but not patched by the HF Generator, there is the possi-
bility to add some additional noise with a certain noise floor Q for each
band k.
E? I Cr q fond
Q[k] ¨ ' õ (3)
,
rfitni r t II
Moreover, state of the art SBR allows for moving SBR frame borders within
certain limits and multiple envelopes per frame.
lo
SBR decoding in conjunction with CELP/HVXC is described in [EBU12, sec-
tion 5.6.2.21 The CELP/HVXC+SBR decoder in DRM is closely related to
state of the art SBR decoding in HEAAC, described in section 1.1.1. Basical-
ly, Fig. 1 applies.
Decoding of envelope information is adapted to spectral properties of
speech-like signals, as described in [EBU12, section 5.6.2.2.41
In regular AMR-WB decoding, the high-band excitation is obtained by gener-
ating white noise uHel(n). The power of the high-band excitation is set equal
to the power of the lower band excitation u2(n),
which means that
G3
r.fo
utin2(7?)= unm(11) (4)
7 v7
11(k =
Finally the high-band excitation is found by
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
4
=
where gi.u3 is a gain factor.
In the 23.85 kbit/s mode, gm is decoded from the received gain index (side
information).
In the 6.60, 8.85, 12.65, 14.25, 15.85, 18.25, 19.85 and 23.05 kbit/s modes,
gHB is estimated using voicing information bounded by 10.1, 1.0]. First, the
tilt
of synthesis eta is found
Ga
E Ahp(n) = ,4,,,,(11 ¨ 1)
1-0
= (6)
i=0
where ghp is the high-pass filtered lower band speech synthesis 402,8(n) with
cut-off frequency of 400Hz. gHB is then found by
WSJ> =PST' ( (7)
where gsp =1-etiit is the gain for the speech signal, gBG = 1.25 gsp is the
gain
for the background noise signal, and wsp is a weighting function set to 1,
when voice activity detection (VAD) is ON, and 0 when VAD is OFF. gHB is
bounded between [0.1, 1.0]. In case of voiced segments where less energy is
present at high frequencies, etut approaches 1 resulting in a lower gain gHB.
This reduces the energy of the generated noise in case of voiced segments.
Then the high-band LP synthesis filter AHB (z) is derived from the weighted
low-band LP synthesis filter:
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
= i(-I--
where A(z) is the interpolated LP synthesis filter. A(z)has been computed
5 analyzing the signal with the sampling rate of 12.8 kHz but it is now
used for
a 16 kHz signal. This means that the band 5.1-5.6 kHz in the 12.8 kHz do-
main will be mapped to 6.4-7.0 kHz in the 16 kHz domain.
uHB (n) is then filtered through AHB (z). The output of this high-band
synthesis
sHB (n) is filtered through a band-pass FIR filter HHB (z), which has the pass-
band from 6 to 7 kHz. Finally, sHB is added to synthesized speech to produce
the synthesized output speech signal.
In AMR-WB+ the HF signal is composed out of the frequency components
above (fs/4) of the input signal. To represent the HF signal at.a low rate, a
bandwidth extension (BWE) approach is employed. In BWE, energy infor-
mation is sent to the decoder in the form of spectral envelope and frame en-
ergy, but the fine structure of the signal is extrapolated at the decoder from
the received (decoded) excitation signal in the LF signal.
The spectrum of the down sampled signal sHF can be seen as a folded ver-
sion of the high-frequency band prior to down-sampling. An LP analysis is
performed on sHF (n) to obtain a set of coefficients, which model the spectral
envelope of this signal. Typically, fewer parameters are necessary than
in the LF signal. Here, a filter of order 8 is used. The LP coefficients are
then
transformed into ISP representation and quantized for transmission.
The synthesis of the HF signal implements a kind of bandwidth extension
(BWE) mechanism and uses some data from the LF decoder. It is an evolu-
tion of the BWE mechanism used in the AMR-WB speech decoder (see
above). The HF decoder is detailed in Fig. 3.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
6
The HF signal is synthesized in 2 steps:
1. Calculation of the HF excitation;
2. Computation of the HF signal from the HF excitation.
The HF excitation is obtained by shaping the LF excitation signal in time-
domain with scalar factors (or gains) on a 64-sample subframe basis. This
HF excitation is post-processed to reduce the "buzziness" of the output, and
113 then filtered by an HF linear-predictive synthesis filter 1/AHF (z).
The result is
further post-processed to smooth energy variations. For further information
please refer to [3GP09].
The packet-loss concealment in SBR in conjunction with MC is specified in
3GPP TS 26.402 [3GP12a, section 5.21 and was subsequently reused in
DRM [EBU12, section 5.6.3.1] and DAB [EBUl 0, section A2].
In case of a frame loss, the number of envelops per frame is set to one and
the last valid received envelope data is reused and decreased in energy by a
constant ratio for every concealed frame.
The resulting envelope data are then fed into the normal decoding process
where the HF adjuster uses them to calculate the gains, which are used for
adjusting the patched highbands out of the HF generator. The rest of SBR
decoding takes place as usual.
Moreover, the coded noise floor delta values are being set to zero which lets
the delta decoded noise floor remain static. At the end of the decoding pro-
cess, this means that the energy of the noise floor follows the energy of the
HF signal.
Furthermore, the flags for adding sines are cleared.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
7
State of the art SBR concealment takes also care of recovery. It attends for a
smooth transition from the concealed signal to the correctly decoded signal in
terms of energy gaps that may result from mismatched frame borders.
State of the art SBR concealment in conjunction with CELP/HVXC is de-
scribed in [EBU12, section 5.6.3.21 and briefly outlined in the following:
Whenever a corrupted frame has been detected, a predetermined set of data
values is applied to the SBR decoder. This yields "a static highband spectral
envelope at a low relative playback level, exhibiting a roll-off towards the
higher frequencies." [EBU12, section 5.6.3.2]. Here, SBR concealment in-
serts some kind of comfort noise, which has no dedicated fading in SBR do-
main. This prevents the listener's ears from potentially loud audio bursts and
keeps the impression of a constant bandwidth.
State of the art concealment of the BWE of G.718 is described in [ITU08,
7.11.1.7.1] and briefly outlined as follows:
In the low delay mode, which is exclusively available for layer 1 and 2, the
concealment of the high-frequency band 6000 - 7000 Hz is performed exact-
ly in the same way as when no frame erasures occur. The clean-channel de-
coder operation for layers 1, 2 and 3 is as follows: a blind bandwidth exten-
sion is applied. The spectrum in the range 6400-7000Hz is filled up with a
white noise signal, properly scaled in the excitation domain (energy of the
high-band must match the low band energy). It is then synthesized with a
filter derived by weighting from the same LP synthesis filter as used in the
12.8 kHz domain. For layers 4 and 5 no bandwidth extension is performed,
since those layers cover the full band up to 8 kHz.
In the default operation a low complexity processing is performed to recon-
struct the high-frequency band of the synthesized signal at 16 kHz sampling
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
8
frequency. First, the scaled high-frequency band excitation, u"HE3(n), is
linear-
ly attenuated throughout the frame as
nn = 4117(n) !hit 171). for n 4 .............. 319 9
where the frame length is 320 samples and gatt (n) is an attenuation factor
which is given by
Nu( n) = 1.0 n 1.0 . for o = 0..... 319 (10)
3")0
In the equation above, gp is the average pitch gain. It is the same gain as
used during concealment of the adaptive codebook. Then, the memory of the
band-pass filter in the frequency range 6000 - 7000Hz is attenuated using
gatt (n), as derived in equation 10, to prevent any discontinuities. Finally,
the
high-frequency excitation signal, u" (n), is filtered through the synthesis
filter.
The synthesized signal is then added to the concealed synthesis at a 16 kHz
sampling frequency.
State of the art concealment of blind bandwidth extension in AMR-WB is out-
lined in [3GP12b, 6.2.4] and briefly summarized here:
When a frame is lost or partly lost, the high-band gain parameter is not re-
ceived and an estimation for the high-band gain is used instead. This means
that in case of bad/lost speech frames, the high-band reconstruction operates
in the same way for all the different modes.
In case a frame is lost, the high-band LP synthesis filter is derived like
usual
from the LPC coefficients from the core band. The only exception is that the
LPC coefficients have not been decoded from the bitstream, but were ex-
trapolated using the regular AMR-WB concealment approach.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
9
State of the art concealment of bandwidth extension in AMR-WB+ is outlined
in [3GP09, 6.2] and briefly summarized here:
In the case of a packet loss, the control data which are internal to the HF de-
coder are generated from the bad frame indicator vector BFI = (bfiO, bfil,
bfi2, bfi3). These data are bf, iisfhf, BFIGAIN, and the number of subframes
for
ISF interpolation. The nature of these data is defined in more details below:
bf iisfht. is a binary flag indicating the loss of the ISF parameters. As the
ISF
parameters for the HF signal are always transmitted in the first packet (con-
taining the first subframe) being either HF20, 40 or 80, the loss flag is
always
set to the bfi indicator of the first subframe (bfi0). The same holds true for
the
indication of lost HF gains. If the first packet/subframe of the current mode
is
lost (HF20, 40 or 80) the gain is lost and needs to be concealed.
The concealment of the HF ISF vectors is very similar to the ISF conceal-
ment for the core 1SFs. The main idea is to reuse the last good ISF vector,
but shift it towards the mean ISF vector (where the mean ISF vector is offline
trained):
ish[i] = 0.9 = isfitil + 0.1 = nicau is f h f [11 (11)
The BWE gains (floe. . = , finb-1) are estimated according to the following
source code (in the code: g", 4-1 gain_q[i]; 2.807458 is a decoder constant).
r use the past gains slightly shifted towards the means *I
*past_q = (0.91*(*past_q + 20.0f)) - 20.0f;
for (1=0; 1<4; i++) {
gain_q[i] = *past_q + 2.807458f;
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
tmp = 0.0;
for (i=0; i<4; i++)
tmp += gain_q[i];
5
*past_q = 0.25rtmp - 2.807458f;
In order to derive the "gains to match the magnitude at fs/4" the same algo-
rithm as in clean channel decoding is performed, but with the exception that
10 the ISFs for the HF and/or the LF part may already be concealed. All
follow-
ing steps like linearldB interpolation, summation and application of gains are
the same as in the clean channel case.
To derive the excitation, the same procedure is applied as in a correctly re-
ceived frame, where the lower band excitation is used after:
= it was randomized
= it was amplified in the time-domain with subframe gains
= it was shaped in the frequency domain with an LP filter
= the energy was smoothed over time
Then the synthesis is performed according to figure 3.
AES convention paper 6789 : Schneider, Krauss and Ehret [SKE06] describe
a concealment technique which reuses the last valid SBR envelope data, If
more than one SBR frame is lost, a fadeout is applied. "The basic principle is
to simply lock the last known valid SBR envelope values until SBR pro-
cessing may be continued with newly transmitted data. In addition a fade-out
is performed if more than one SBR frame is not decodable."
AES convention paper 6962: Sang-Uk Ryu and Kenneth Rose [RR06] de-
scribe a concealment technique which estimates the parametric information,
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
11
utilizing SBR data from the previous and the next frame. High band enve-
lopes are adaptively estimated from energy evolution in the surrounding
frames.
The packet-loss concealment concepts may produce a perceptually degrad-
ed audio signal during packet loss.
It's an objective of the present invention to provide an audio decoder and a
method having an improved packet-loss concealment concept.
lo
This object may be achieved by an audio decoder configured to produce an
audio signal from a bitstream containing audio frames, the audio decoder
cornprising:
a core band decoding module configured to derive a directly decoded core
band audio signal from the bitstream;
a bandwidth extension module configured to derive a parametrically decoded
bandwidth extension audio signal from the core band audio signal and from
the bitstream, wherein the bandwidth extension audio signal is based on a
frequency domain signal having at least one frequency band; and
a combiner configured to combine the core band audio signal and the band-
width extension audio signal so as to produce the audio signal;
wherein the bandwidth extension module comprises an energy adjusting
module being configured in such way that in a current audio frame in which
an audio frame loss occurs, an adjusted signal energy for the current audio
frame for the at least one frequency band is set
based on a current gain factor for the current audio frame, wherein the cur-
rent gain factor is derived from a gain factor from a previous audio frame or
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
12
from the bitstream, and
based on an estimated signal energy for the at least one frequency band,
wherein the estimated signal energy is derived from a spectrum of the current
audio frame of the core band audio signal.
The audio decoder according to the invention links the bandwidth extension
module to the core band decoding module in terms of energy or, in other
words, assures that the bandwidth extension module follows the core band
decoding module energy-wise during concealment, no matter what the core
band decoding module does.
The innovation with this approach is that - in concealment case - the high
band generation is not strictly adapted to envelope energies anymore. With
the technique of gain locking, the high band energies are adapted to the low
band energies during concealment and hence are no more relying only on
the transmitted data in the last good frame. This proceeding takes up the
idea to use low band information for high band reconstruction.
zo With this approach, no additional data (e .g. fadeout factor) needs to
be
transferred from the core coder to the bandwidth extension coder. This
makes the technique easily applicable to any coder with bandwidth exten-
sion, especially to SBR, where gain calculation already is performed inher-
ently (equation 1).
The concealment of the inventive audio decoder takes into consideration the
fading slope of the core band decoding module. This leads to intended be-
havior of the fadeout as a whole:
Situations in which the energies of the frequency bands of the core band de-
coding module fade out slower than the energies of the frequency bands of
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
13
the bandwidth extension module, which would become perceivable and
cause the unlovely impression of a band limited signal, are avoided.
Furthermore, situations in which the energies in the frequency bands of the
core band decoding module fade out faster than the energies of the frequen-
cy bands of the bandwidth extension module, which would introduce artifacts
because frequency bands of the bandwidth extension module are amplified
too much, compared to the frequency bands of the core band decoding mod-
ule, are avoided as well.
lo
In contrast to a non-fading decoder having a bandwidth extension with prede-
fined energy levels (as for example a CELP/HVXC+SBR decoder), which
preserves only the spectral tilt of a certain signal type, works the inventive
audio decoder independently from the spectral characteristics of the signals,
so that a perceptually decoded degradation of the audio signal is avoided.
The proposed technique could be used with any bandwidth extension (BWE)
method on top of a core band decoding module (core coder in the following).
Most of the bandwidth extension technique is based on the gain per band
between the original energy levels and the energy levels obtained after copy-
ing the core spectrum. The proposed technique does not work on the ener-
gies of the previous audio frame, as the state of the art does, but on the
gains of the previous audio frame.
When an audio frame is lost or unreadable (or in other words, if an audio
frame loss occurs) the gains from the last good frame are fed into the normal
decoding process of the core band decoding module, which adjusts the en-
ergies of the frequency bands of the bandwidth extension module (see equa-
tion 1). This forms the concealment. Any fadeout, being applied on the core
band decoding module by a core band decoding module concealment, will be
automatically applied to the energies of the frequency bands of the band-
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
14
width extension module by locking the energy ratio between the low and the
high band.
The frequency domain signal having at least one frequency band may be, for
example, an algebraic code-excited linear prediction excitation signal
(ACELP excitation signal).
In some embodiments the bandwidth extension module comprises gain factor
providing module configured to forward the current gain factor at least in the
io current audio frame in which the audio frame loss occurs to the energy
ad-
justing module.
In a preferred embodiment the gain factor providing module is configured in
such way that in the current audio frame in which the audio frame loss occurs
the current gain factor is the gain factor of the previous audio frame.
This embodiment completely deactivates the fadeout contained in the band-
width extension decoding module by only locking the gains derived for the
last envelope in the last good frame:
[1.1 = [k] t 2 )
= EF.:4 fliiroc
wherein EAdj [k] denotes the energy from one frequency band k of the band-
width extension module, adjusted to express the original energy distribution
as good as possible; g,, N, gbwe [k] denotes the gain factor of the current
frame; and gLnw-1,1 [k] denotes the gain factor of the previous frame.
In other preferred embodiment the gain factor providing module is configured
in such way that in the current audio frame in which the frame loss occurs the
current gain factor is calculated from the gain factor of the previous audio
frame and from a signal class of the previous audio frame.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
This embodiment uses a signal classifier to compute the gains based on the
past gains and also adaptively on the signal class of the previously received
frame:
5
4 f
1 (9;:',',::=;k1-cj, = ) 1:3)
E.141 [k: = EF.q[k].' 95z, r.k)
wherein f (grhnLil , c-11) denotes a function, depending on the gain factor
g [bn, o f the previous audio frame and the signal class cs[V]of the previous
io audio frame. Signal classes may refer to classes of speech sounds such
as:
obstruent (with subclasses: stop, affricative, fricative), sonorant (this sub-
classes: nasal, flap approximant, vowel), lateral, trill.
In a preferred embodiment the gain factor providing module is configured to
15 calculate a number of subsequent audio frames in which audio frame
losses
occur and configured to execute a gain factor lowering procedure in case the
number of subsequent audio frames in which audio frame losses occur ex-
ceeds a predefined number.
If a fricative occurs immediately before a burst frame loss (multiple frame
losses in subsequent audio frames), the inherent default fadeout of the core
band decoding module may be too slow to assure a pleasant and natural
sound in combination with gain locking. The perceived result Of this issue
may be a prolonged fricative with too much energy in the frequency bands of
the bandwidth extension module. For this reason a check for multiple frame
losses may be performed. If this check is positive a gain factor lowering pro-
cedure may be executed.
In a preferred embodiment the gain factor lowering procedure comprises the
step of lowering the current gain factor by dividing the current gain factor
by a
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
16
first figure in case the current gain factor exceeds a first threshold. By
these
features on gains that exceed a the first threshold (which may be determined
empirically) are lowered.
In a preferred embodiment the gain factor lowering procedure comprises the
step of lowering the current gain factor by dividing the current gain factor
by a
second figure which is large than the first figure in case the current gain
fac-
tor exceeds a second threshold which is larger than the first threshold. These
features ensure that extremely high gains decrease even faster. All gains
io exceeding the second threshold will be decreased faster.
In some embodiments the gain factor lowering procedure comprises the step
of setting the current gain factor to the first threshold in case the current
threshold after lowering is below the first threshold. By these features the
decreased gains are prevented to fall below the first threshold.
An example can be seen within the pseudo code 1:
/*limit gain in case of multiple frameloss*/
#DEFINE BWE GAINDEC 10
if (previousFrameErrorFlag && (gain[k] > BWE_GAINDEC) ) (
/* gains exceeding the first threshold 50 times will be decreased faster */
if (gain[k] > 50* BWE_GAINDEC ) {
gain[k] 1= 6;
else {
gain[k] /= 4;
r prevent gains from falling below BWE_GAINDEC */
if (gain[k] < BWE_GAINDEC) (
gain[k] = BWE_GAINDEC:
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
17
wherein previousFrameErrorFlag is a flag, which indicates if a multiple frame
loss is present, BWE_GAINDEC denotes the first threshold, 50*
BWE _ GAINDEC denotes the second threshold and gain[k] denotes the cur-
rent gain factor for the frequency band k.
In some embodiments the bandwidth extension module comprises a noise
io generator module configured to add noise to the at least one frequency
band,
wherein in the current audio frame in which the audio frame loss occurs a
ratio of the signal energy to the noise energy of the at least on frequency
band of the previous audio frame is used to calculate the noise energy of the
current audio frame.
In case there is a noisefloor feature (i. e. additional noise components to re-
tain noisiness of the original signal) implemented in the bandwidth extension,
it is necessary to adopt the idea of gain locking also towards the noise
floor.
To achieve this, the noise floor energy levels of non-concealed frames are
converted to a noise ratio, taking into account the energy of the frequency
bands of the bandwidth extension module. The ratio is saved to a buffer and
will be the base for the noise level in the concealment case. The main ad-
vantage is the better coupling of the noise floor to the core coder energy due
to a calculation of the ratio prev_noise[k].
The pseudo code 2 shows this:
for (k=bands)
if i(frameErrorFlag)
prev_noise[k] = nrgHighband[k] / noiseLevel[k];
else {
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
18
noiseLevel[k] = nrgHighband[k] / prev_noise[k];
wherein frameErrorFlag is a flag indicating if a frame loss is present and
prev_noise[k] is the ratio between the energy nrgHighband[k] of the frequen-
cy band k and the noise level noiseLevel[k] of the frequency band k.
In a preferred embodiment the audio decoder comprises a spectrum analyz-
ing module configured to establish the spectrum of the current audio frame of
the core band audio signal and to derive the estimated signal. energy for the
current frame for the at least one frequency band from the spectrum of the
current audio frame of the core band audio signal.
In some embodiments the gain factor providing module is configured in such
way that, in case that a current audio frame, in which an audio frame loss
does not occur, subsequently follows on a previous audio frame, in which an
audio frame loss occurs, the gain factor received for the current audio frame
is used for the current frame, if a delay between audio frames of the band-
width extension module with respect to the audio frames of the core band
decoding module is smaller than a delay threshold, whereas the gain factor
from the previous audio frame is used for the current frame, if the delay be-
tween audio frames of the bandwidth extension module with respect to the
audio frames of the core band decoding module is bigger than the delay
threshold.
On top of the concealment, in the bandwidth extension module special atten-
tion needs to be paid to the framing. Audio frames of the bandwidth exten-
sion module and audio frames of the core band decoding module are often
not exactly aligned but could have a certain delay. So it may happen that one
lost packet contains bandwidth extension data being delayed, relative to the
core signal contained in the same packet.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
19
The result in this case is that the first good packet after a loss may contain
extension data to create parts of the frequency bands of the bandwidth ex-
tension module of the previous core band decoding module audio frame,
which was already concealed in the decoder.
For this reason, the framing needs to be considered during recovery, de-
pending on the respective properties of the core and decoding module and
bandwidth extension module. This could mean to treat the first audio frame or
parts of it in the bandwidth extension module as erroneous and not to apply
the newest gains at once but to keep the locked gains from the first audio
frame for one additional frame.
Whether or not to keep the locked gains for the first good frame depends on
the delay. Experimental application to codecs with different delays showed
different benefit for codecs with different delays. For codecs with quite
small
delays (e. g. 1ms), it is better to use the newest gains for the first good
audio
frame.
In a preferred embodiment the bandwidth extension module comprises a sig-
nal generator module configured to create a raw frequency domain signal
having at least on frequency band, which is forwarded to the energy adjusting
module, based on the core band audio signal and the bitstream.
In a preferred embodiment the bandwidth extension module comprises a sig-
nal synthesis module configured to produce the bandwidth extension audio
signal from the frequency domain signal.
The object of the invention may be achieved by a method for producing an
audio signal from a bitstream containing audio frames. The method compris-
es the steps of:
,
deriving a directly decoded core band audio signal from the bitstream;
deriving a parametrically decoded bandwidth extension audio signal from the
core band audio
signal and from the bitstream, wherein the bandwidth extension audio signal is
based on a fre-
5 quency domain signal having at least one frequency band; and
combining the core band audio signal and the bandwidth extension audio signal
so as to pro-
duce the audio signal;
10 wherein in a current audio frame in which an audio frame loss occurs, an
adjusted signal en-
ergy for the current audio frame for the at least one frequency band is set
based on a current gain factor for the current audio frame, wherein the
current gain factor is
derived from a gain factor from a previous audio frame or from the bitstream,
and
based on an estimated signal energy for the at least one frequency band,
wherein the esti-
mated signal energy is derived from a spectrum of the current audio frame of
the core band
audio signal.
The object of the invention may further be achieved by a computer program for
performing,
when running on a computer or a processor, the method described above.
Brief description of the drawings:
Fig. 1 illustrates an SBR decoder according to prior art;
Fig. 2 illustrates the mode of operation of the SBR decoder according
to prior art;
Fig. 3 illustrates a bandwidth extension module in AMR-WB;
Fig. 4 illustrates an embodiment of an audio decoder according to the
invention in a
schematic view; and
CA 2915001 2018-03-15
,
21
Fig. 5 illustrates the framing of an embodiment of an audio decoder
according to the
invention.
Fig. 4 illustrates an embodiment of an audio decoder 1 according to the
invention in a sche-
matic view. The audio decoder 1 is configured to produce an audio signal AS
from a bitstream
BS containing audio frames AF. The audio decoder 1 comprises:
a core band decoding module to configured to derive a directly decoded core
band audio sig-
nal CBS from the bitstream BS;
a bandwidth extension module 2 configured to derive a parametrically decoded
bandwidth ex-
tension audio signal BES from the core band audio signal CBS and from the
bitstream BS,
wherein the bandwidth extension audio signal BES is based on a frequency
domain signal
FDS having at least one frequency band FB; and
a combiner 4 configured to combine the core band audio signal CBS and the
bandwidth exten-
sion audio signal BES so as to produce the audio signal AS;
wherein the bandwidth extension module 3 comprises an energy adjusting module
5 being
configured in such way that in a current audio frame AF2 in which an audio
frame loss AFL oc-
curs, an adjusted signal energy for the current audio frame AF2 for the at
least one frequency
band FB is set
based on a current gain factor CGF for the current audio frame AF2, wherein
the current gain
factor CGF is derived from a gain factor from a previous audio frame AF1 or
from the bitstream
BS, and
based on an estimated signal energy EE for the at least one frequency band FB,
wherein the
estimated signal energy EE is derived from a spectrum of the current audio
frame AF2 of the
core band audio signal CBS.
CA 2915001 2018-03-15
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
22
The audio decoder 1 according to the invention links the bandwidth extension
module 3 to the core band decoding module to in terms of energy or, in other
words, assures that the bandwidth extension module 3 follows the core band
decoding module 2 energy-wise during concealment, no matter what the core
band decoding module 2 does.
The innovation with this approach is that - in concealment case - the high
band generation is not strictly adapted to envelope energies anymore. With
io the technique of gain locking, the high band energies are adapted to the
low
band energies during concealment and hence are no more relying only on
the transmitted data in the last good frame AF1. This proceeding takes up the
idea to use low band information for high band reconstruction.
With this approach, no additional data (e .g. fadeout factor) needs to be
transferred from the core coder 2 to the bandwidth extension coder 3. This
makes the technique easily applicable to any coder 1 with bandwidth exten-
sion 3, especially to SBR, where gain calculation already is performed inher-
ently (equation 1).
The concealment of the inventive audio decoder 1 takes into consideration
the fading slope of the core band decoding module 2. This leads to intended
behavior of the fadeout as a whole:
Situations in which the energies of the frequency bands FB of the core band
decoding module 2 fade out slower than the energies of the frequency bands
FB of the bandwidth extension module 3, which would become perceivable
and cause the unlovely impression of a band limited signal, are avoided.
Furthermore, situations in which the energies in the frequency bands FB of
the core band decoding module 2 fade out faster than the energies of the
frequency bands FB of the bandwidth extension module 3, which would in-
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
23
troduce artifacts because frequency bands FB of the bandwidth extension
module 3 are amplified too much, compared to the frequency bands FB of
the core band decoding module 2, are avoided as well.
In contrast to a non-fading decoder having a bandwidth extension with prede-
fined energy levels (as for example a CELP/HVXC+SBR decoder), which
preserves only the spectral tilt of a certain signal type, the inventive audio
decoder 1 works independently from the spectral characteristics of the sig-
nals, so that a perceptually decoded degradation of the audio signal AS is
avoided.
The proposed technique could be used with any bandwidth extension (BWE)
method on top of a core band decoding module 2 (core coder in the follow-
ing). Most of the bandwidth extension technique is based on the gain per
band between the original energy levels and the energy levels obtained after
copying the core spectrum. The proposed technique does not work on the
energies of the previous audio frame, as the state of the art does, but on the
gains of the previous audio frame AF1.
When an audio frame AF2 is lost or unreadable (or in other words, if an audio
frame loss AFL occurs) the gains from the last good frame are fed into the
normal decoding process of the core band decoding module 2, which adjusts
the energies of the frequency bands FB of the bandwidth extension module 3
(see equation 1). This forms the concealment. Any fadeout, being applied on
the core band decoding module 2 by a core band decoding module conceal-
ment, will be automatically applied to the energies of the frequency bands FB
of the bandwidth extension module 3 by locking the energy ratio between the
low and the high band.
In some embodiments the bandwidth extension module 3 comprises gain
factor providing module 6 configured to forward the current gain factor CGF
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
24
at least in the current audio frame AF2 in which the audio frame loss AFL
occurs to the energy adjusting module 5.
In a preferred embodiment the gain factor providing module 6 is configured in
such way that in the current audio frame AF2 in which the audio frame loss
AFL occurs the current gain factor CGF is the gain factor of the previous au-
dio frame AF1.
This embodiment completely deactivates the fadeout contained in the band-
ict width extension decoding module 3 by only locking the gains derived for
the
last envelope in the last good frame:
In other preferred embodiment the gain factor providing module 6 is config-
ured in such way that in the current audio frame AF2 in which the frame loss
AFL occurs the current gain factor she CGS is calculated from the gain factor
of the previous audio frame and from a signal class of the previous audio
frame.
This embodiment uses a signal classifier to compute the gains GCS based
on the past gains and also adaptively on the signal class of the previously
received frame AF1. Signal classes may refer to classes of speech sounds
such as: obstruent (with subclasses: stop, affricative, fricative), sonorant
(this
subclasses: nasal, flap approximant, vowel), lateral, trill.
In a preferred embodiment the gain factor providing module 6 is configured to
calculate a number of subsequent audio frames in which audio frame losses
AFL occur and configured to execute a gain factor lowering procedure in
case the number of subsequent audio frames in which audio frame losses
AFL occur exceeds a predefined number.
If a fricative occurs immediately before a burst frame loss (multiple frame
losses AFL in subsequent audio frames AF), the inherent default fadeout of
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
the core band decoding module 2 may be too slow to assure a pleasant and
natural sound in combination with gain locking. The perceived result of this
issue may be a prolonged fricative with too much energy in the frequency
bands FB of the bandwidth extension module 3. For this reason a check for
5 .. multiple frame losses AFL may be performed. If this check is positive a
gain
factor lowering procedure may be executed.
In a preferred embodiment the gain factor lowering procedure comprises the
step of lowering the current gain factor by dividing the current gain factor
by a
10 first figure in case the current gain factor exceeds a first threshold.
By these
features on gains that exceed the first threshold (which may be determined
empirically) are lowered.
In a preferred embodiment the gain factor lowering procedure comprises the
15 step of lowering the current gain factor by dividing the current gain
factor by a
second figure which is large than the first figure in case the current gain
fac-
tor exceeds a second threshold which is larger than the first threshold. These
features ensure that extremely high gains decrease even faster. All gains
exceeding the second threshold will be decreased faster.
In some embodiments the gain factor lowering procedure comprises the step
of setting the current gain factor to the first threshold in case the current
threshold after lowering is below the first threshold. By these features the
decreased gains are prevented to fall below the first threshold.
In some embodiments the bandwidth extension module 3 comprises a noise
generator module 7 configured to add noise NOI to the at least one frequen-
cy band FB, wherein in the current audio frame AF2 in which the audio frame
loss AFL occurs a ratio of the signal energy to the noise energy of the at
least
on frequency band FB of the previous audio frame AF1 is used to calculate
the noise energy of the current audio frame AF2.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
26
In case there is a noisefloor feature (i. e. additional noise components to re-
tain noisiness of the original signal) implemented in the bandwidth extension
3, it is necessary to adopt the idea of gain locking also towards the noise
floor. To achieve this, the noise floor energy levels of non-concealed frames
are converted to a noise ratio, taking into account the energy of the frequen-
cy bands of the bandwidth extension module. The ratio is saved to a buffer
and will be the base for the noise level in the concealment case. The main
advantage is the better coupling of the noise floor to the core coder energy
due to a calculation of the ratio.
-10
In a preferred embodiment the audio decoder 1comprises a spectrum analyz-
ing module 8 configured to establish the spectrum of the current audio frame
AF2 of the core band audio signal CBS and to derive the estimated signal
energy EE for the current frame AF2 for the at least one frequency band FB
from the spectrum of the current audio frame AF2 of the core band audio sig-
nal CBS.
In a preferred embodiment the bandwidth extension module 3 comprises a
signal generator module 9 configured to create a raw frequency domain sig-
nal RFS having at least on frequency band FB, which is forwarded to the en-
ergy adjusting module 5, based on the core band audio signal CBS and the
bitstream BS.
In a preferred embodiment the bandwidth extension module 3 comprises a
signal synthesis module 10 configured to produce the bandwidth extension
audio signal BES from the frequency domain signal FDS.
Fig. 5 illustrates the framing of an embodiment of an audio decoder 1 accord-
ing to the invention.
In some embodiments the gain factor providing module 6 is configured in
such way that, in case that a current audio frame AF2, in which an audio
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
27
frame loss AFL does not occur, subsequently follows on a previous audio
frame AF1, in which an audio frame loss AFL occurs, the gain factor received
for the current audio frame AF2 is used for the current frame AF2, if a delay
DEL between audio frames AF of the bandwidth extension module 3 with
.. respect to the audio frames AF' of the core band decoding module 2 is
smaller than a delay threshold, wheras the gain factor from the previous au-
dio frame AF1 is used for the current frame AF 2, if the delay DEL between
audio frames AF of the bandwidth extension module 3 with respect to the
audio frames AF of the core band decoding module 3 is bigger than the de-
io lay threshold.
On top of the concealment, in the bandwidth extension module 3 special at-
tention needs to be paid to the framing. Audio frames AF of the bandwidth
extension module and audio frames AF' of the core band decoding module 3
are often not exactly aligned but could have a certain delay DEL. So it may
happen that one lost packet contains bandwidth extension data being de-
layed, relative to the core signal contained in the same packet.
The result in this case is that the first good packet after a loss may contain
zo extension data to create parts of the frequency bands FB of the
bandwidth
extension module 3 of the previous core band decoding module audio frame
AF', which was already concealed in the decoder 2.
For this reason, the framing needs to be considered during recovery, de
.. pending on the respective properties of the core decoding module and band-
width extension module. This could mean to treat the first audio frame or
parts of it in the bandwidth extension module 3 as erroneous and not to apply
the newest gain factor at once but to keep the locked gains from the first au-
dio frame for one additional frame.
Whether or not to keep the locked gains for the first good frame depends on
the delay. Experimental application to codecs with different delays showed
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
28
different benefit for codecs with different delays. For codecs with quite
small
delays (e. g. 1ms), it is better to use the newest gain factors for the first
good
audio frame.
Although some aspects have been described in the context of an apparatus,
it is clear that these aspects also represent a description of the correspond-
ing method, where a block or device corresponds to a method step or a fea-
ture of a method step. Analogously, aspects described in the context of a
method step also represent a description of a corresponding block or item or
feature of a corresponding apparatus. Some or all of the method steps may
be executed by (or using) a hardware apparatus, like for example, a micro-
processor, a programmable computer or an electronic circuit. In some em-
bodiments, some one or more of the most important method steps may be
executed by such an apparatus.
Depending on certain implementation requirements, embodiments of the in-
vention can be implemented in hardware or in software. The implementation
can be performed using a non-transitory storage medium such as a digital
storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM,
a PROM, and EPROM, an EEPROM or a FLASH memory, having electroni-
cally readable control signals stored thereon, which cooperate (or are capa-
ble of cooperating) with a programmable computer system such that the re-
spective method is performed. Therefore, the digital storage medium may be
computer readable.
Some embodiments according to the invention comprise a data carrier hav-
ing electronically readable control signals, which are capable of cooperating
with a programmable computer system, such that one of the methods de-
scribed herein is performed.
Generally, embodiments of the present invention can be implemented as a
computer program product with a program code, the program code being
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
29
operative for performing one of the methods when the computer program
product runs on a computer. The program code may, for example, be stored
on a machine readable carrier.
Other embodiments comprise the computer program for performing one of
the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a com-
puter program having a program code for performing one of the methods de-
l() scribed herein, when the computer program runs on a computer.
A further embodiment of the inventive method is, therefore, a data carrier (or
a digital storage medium, or a computer-readable medium) comprising, rec-
orded thereon, the computer program for performing one of the methods de-
scribed herein. The data carrier, the digital storage medium or the recorded
medium are typically tangible and/or non-transitionary.
A further embodiment of the invention method is, therefore, a data stream or
a sequence of signals representing the computer program for performing one
of the methods described herein. The data stream or the sequence of signals
may, for example, be configured to be transferred via a data communication
connection, for example, via the internet.
A further embodiment comprises a processing means, for example, a corn-
puter or a programmable logic device, configured to, or adapted to, perform
one of the methods described herein.
A further embodiment comprises a computer having installed thereon the
computer program for performing one of the methods described herein.
A further embodiment according to the invention comprises an apparatus or a
system configured to transfer (for example, electronically or optically) a
corn-
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
puter program for performing one of the methods described herein to a re-
ceiver. The receiver may, for example, be a computer, a mobile device, a
memory device or the like. The apparatus or system may, for example, com-
prise a file server for transferring the computer program to the receiver.
5
In some embodiments, a programmable logic device (for example, a field
programmable gate array) may be used to perform some or all of the func-
tionalities of the methods described herein. In some embodiments, a field
programmable gate array may cooperate with a microprocessor in order to
10 perform one of the methods described herein. Generally, the methods are
preferably performed by any hardware apparatus.
The above described embodiments are merely illustrative for the principles of
the present invention. It is understood that modifications and variations of
the
15 arrangements and the details described herein will be apparent to others
skilled in the art. It is the intent, therefore, to be limited only by the
scope of
the impending patent claims and not by the specific details presented by way
of description and explanation of the embodiments herein.
20 Reference signs:
1 audio decoder
2 core band decoding module
3 bandwidth extension module
25 4 combiner
5 energy adjusting module
6 gain factor providing module
7 noise generator module
8 spectrum analyzing module
30 9 signal generator module
10 signal synthesis module
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
31
AS audio signal
BS bitstream
AF audio frame
CBS core band audio signal
BES bandwidth extension audio signal
FDS frequency domain signal
FB frequency band
AFL audio frame loss
CGF current gain factor
EE estimated signal energy
NO1 noise
DEL delay
RFS raw frequency domain signal
References:
[3GP09] 3GPP; Technical Specification Group Services and System As-
pects, Extended adaptive multi-rate - wideband (AMR-WB+) co-
dec, 3GPP TS 26.290, 3rd Generation Partnership Project, 2009.
[3GP12a] General audio codec audio processing functions; Enhanced aac-
Plus general audio codec; additional decoder tools (release 11),
3GPP TS 26.402, 3rd Generation Partnership Project, Sep 2012.
(3GP121DI Speech codec speech processing functions; adaptive multi-rate -
wideband (AMRWB) speech codec; error concealment of errone-
ous or lost frames, 3GPP TS 26.191, 3rd Generation Partnership
Project, Sep 2012.
3o [EBU10] EBU/ETSI JTC Broadcast, Digital audio broadcasting (DAB);
transport of advanced audio coding (AAC) audio, ETSI TS 102
563, European Broadcasting Union, May 2010.
CA 02915001 2015-12-10
WO 2014/202701
PCT/EP2014/062902
32
[EBU12] Digital radio mondiale (DRM); system specification, ETSI ES 201
980, ETSI, Jun 2012.
[IS0091 ISO/1EC JTC1/SC29/WG11, Information technology ¨ coding of
audio-visual objects ¨ part 3: Audio, ISO/IEC IS 14496-3, Interna-
tional Organization for Standardization, 2009.
[ITU08] ITU-T, G.718: Frame error robust narrow-band and wideband em-
bedded variable bit-rate coding of speech and audio from 8-32
kbit/s, Recommendation ITU-T G.718, Telecommunication Stand-
ardization Sector of ITU, Jun 2008.
[RR061 Sang-Uk Ryu and Kenneth Rose, Frame loss concealment for au-
dio decoders employing spectral band replication, Convention Pa-
per 6962, Electrical and Computer Engineering, University of Cali-
fornia, Oct 2006, AES.
[SKE06] Andreas Schneider, Kurt Krauss, and Andreas Ehret, Evaluation of
real-time transport protocol configurations using aacplus, Conven-
tion paper 6789, AES, May 2006, Presented at the 120th Conven-
tion 2006 May 20-23.
