Note: Descriptions are shown in the official language in which they were submitted.
WO 2019/068638 PCT/EP2018/076641
Apparatus, method and computer program for encoding, decoding, scene pro-
cessing and other procedures related to DirAC based spatial audio coding
6 Field of the Invention
The present invention is related to audio signal processing and particularly
to audio signal
processing of audio descriptions of audio scenes.
Introduction and state-of-the-art:
Transmitting an audio scene in three dimensions requires handling multiple
channels
which usually engenders a large amount of data to transmit. Moreover 3D sound
can be
represented in different ways: traditional channel-based sound where each
transmission
channel is associated with a loudspeaker position; sound carried through audio
objects,
which may be positioned in three dimensions independently of loudspeaker
positions; and
scene-based (or Arnbisonics), where the audio scene is represented by a set of
coefficient
signals that are the linear weights of spatially orthogonal basis functions,
e.g., spherical
harmonics. In contrast to channel-based representation, scene-based
representation is
independent of a specific loudspeaker set-up, and can be reproduced on any
loudspeaker
set-ups at the expense of an extra rendering process at the decoder.
For each of these forrnats, dedicated coding schemes were developed for
efficiently stor-
ing or transmitting at low bit-rates the audio signals. For example, MPEG
surround is a
parametric coding scheme for channel-based surround sound, while MPEG Spatial
Audio
Object Coding (SAOC) is a parametric coding method dedicated to object-based
audio. A
parametric coding technique for high order of Ambisonics was also provided in
the recent
standard MPEG-Fi phase 2.
In this context, where all three representations of the audio scene, channel-
based, object
based and scene-based audio, are used and need to be supported, there is a
need to
design a universal scheme allowing an efficient parametric coding of ail three
3D audio
representations. Moreover there is a need to be able to encode, transmit and
reproduce
complex audio scenes composed of a mixture of the different audio
representations.
Directional Audio Coding (DirAC) technique [11 is an efficient approach to the
analysis and
reproduction of spatial sound. DirAC uses a perceptually motivated
representation of the
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
2
sound field based on direction of arrival (DOA) and diffuseness measured per
frequency
band. It is built upon the assumption that at one time instant and at one
critical band, the
spatial resolution of auditory system is limited to decoding one cue for
direction and an-
other for inter-aural coherence. The spatial sound is then represented in
frequency do-
main by cross-fading two streams: a non-directional diffuse stream and a
directional non
diffuse stream.
DirAC was originally intended for recorded B-format sound but could also serve
as a
common format for mixing different audio formats. DirAC was aiready extended
for pro-
cessing the conventional surround sound format 5.1 in [3]. It was also
proposed to merge
multiple DirAC streams in [4]. Moreover, DirAC we extended to also support
microphone
inputs other than B-format [6].
However, a universal concept is missing to make DirAC a universal
representation of au-
dio scenes in 3D which also is able to support the notion of audio objects.
Few considerations were previously done for handling audio objects in DirAC.
DirAC was
employed in [5] as an acoustic front end for the Spatial Audio Coder, SAOC, as
a blind
source separation for extracting several talkers from a mixture of sources. It
was, howev-
er, not envisioned to use DirAC itself as the spatial audio coding scheme and
to process
directly audio objects along with their rnetadata and to potentially combine
them together
and with other audio representations.
it is an object of the present invention to provide an improved concept of
handling and
processing audio scenes and audio scene descriptions.
This object is achieved by an apparatus for generating a description of a
combined audio
scene of claim 1, a method of generating a description of a combined audio
scene of
claim 14, or a related computer program of claim 15.
Furthermore, this object is achieved by an apparatus for performing a
synthesis of a plu-
rality of audio scenes of claim 16, a method for performing a synthesis of a
plurality of
audio scenes of claim 20, or a related computer program in accordance with
claim 21.
This object is furthermore achieved by an audio data converter of claim 22, a
method for
performing an audio data conversion of claim 23, or a related computer program
of claim
29.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
3
Furthermore, this object is achieved by an audio scene encoder of claim 30, a
method of
encoding an audio scene of claim 34, or a related computer program of claim
35.
Furthermore, this object is achieved by an apparatus for performing a
synthesis of audio
data of claim 36, a method for performing a synthesis of audio data of claim
40, or a relat-
ed computer program of claim 41,
Embodiments of the invention relate to a universal parametric coding scheme
for 3D audio
scene built around the Directional Audio Coding paradigm (DirAC), a
perceptually-
motivated technique for spatial audio processing. Originally DirAC was
designed to ana-
lyze a B-format recording of the audio scene. The present invention aims to
extend its
ability to process efficiently any spatial audio formats such as channel-based
audio, Ambi-
sonics, audio objects, or a mix of them
DirAC reproduction can easily be generated for arbitrary loudspeaker layouts
and head-
phones. The present invention also extends this ability to output additionally
Arnbisonics,
audio objects or a mix of a format. More importantly the invention enables the
possibility
for the user to manipulate audio objects and to achieve, for example, dialogue
enhance-
ment at the decoder end.
Context: Svstern overvie.w of a DirAC apatial Audio Coder
In the following, an overview of a novel spatial audio coding system based on
DirAC de-
signed for Immersive Voice and Audio Services (IVAS) is presented. The
objective of such
a system is to be able to handle different spatial audio formats representing
the audio
scene and to code them at low bit-rates and to reproduce the original audio
scene as
faithfully as possible after transmission.
The system can accept as input different representations of audio scenes. The
input audio
scene can be captured by multi-channel signals aimed to be reproduced at the
different
loudspeaker positions, auditory objects along with metadata describing the
positions of
the objects over time, or a first-order or higher-ufder Airibisonics format
representing the
sound field at the listener or reference position.
Preferably the system is based on 3GPP Enhanced Voice Services (EVS) since the
solu-
tion is expected to operate with low latency to enable conversational services
on mobile
networks,
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
4
Fig. 9 is the encoder side of the DirAC-based spatial audio coding supporting
different
audio formats. As shown in Fig. 9, the encoder (WAS encoder) is capable of
supporting
different audio formats presented to the system separately or at the same
time. Audio
signals can be acoustic in nature, picked up by microphones, or electrical in
nature, which
are supposed to be transmitted to the loudspeakers. Supported audio formats
can be mul-
ti-channel signal, first-order and higher-order Ambisonics components, and
audio objects.
A complex audio scene can also be described by combining different input
formats. All
audio formats are then transmitted to the DirAC analysis 1B0, which extracts a
parametric
representation of the complete audio scene. A direction of arrival and a
diffuseness
measured per time-frequency unit form the parameters. The DirAC analysis is
followed by
a spatial metadata encoder 190, which quantizes and encodes DirAC parameters
to ob-
tain a low bit-rate parametric representation.
Along with the parameters, a down-mix signal derived 160 from the different
sources or
audio input signals is coded for transmission by a conventional audio core-
coder 170. In
this case an EVS-based audio coder is adopted for coding the down-mix signal.
The
down-mix signal consists of different channels, called transport channels: the
signal can
be e.g. the four coefficient signals composing a B-format signal, a stereo
pair or a mono-
phonic down-mix depending of the targeted bit-rate. The coded spatial
parameters and
the coded audio bitstream are multiplexed before being transmitted over the
communica-
tion channel.
Fig. 10 is a decoder of the DirAC-based spatial audio coding delivering
different audio
formats. In the decoder, shown in Fig. 10, the transport channels are decoded
by the
core-decoder 1020, while the DirAC metadata is first decoded 1060 before being
con-
veyed with the decoded transport channels to the DirAC synthesis 220, 240. At
this stage
(1040), different options can be considered. It can be requested to play the
audio scene
directly on any loudspeaker or headphone configurations as is usually possible
in a con-
ventional DirAC system (MC in Fig. 10). In addition, it can also be requested
to render the
scene to Ambisonics format for other further manipulations, such as rotation,
reflection or
movement of the scene (FONFIOA in Fig. 10). Finally, the decoder can deliver
the indi-
vidual objects as they were presented at the encoder side (Objects in Fig.
10).
Audio objects could also be restituted but it is more interesting for the
listener to adjust the
rendered mix by interactive manipulation of the objects. Typical object
manipulations are
adjustment of level, equalization or spatial location of the object. Object-
based dialogue
enhancement becomes, for example, a possibility given by this interactivity
feature. Final-
ly, it is possible to output the original formats as they were presented at
the encoder input.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
In this case, it could be a mix of audio channels and objects or Ambisonics
and objects. In
order to achieve separate transmission of multi-channels and Ambisonics
components,
several instances of the descnbed system Gould be used.
5 The present invention is advantageous in that, particularly in accordance
with the first as-
pect, a framework is established in order to combine different scene
descriptions into a
combined audio scene by way of a common format, that allows to combine the
different
audio scene descriptions.
This common format may, for example, be the B-format or may be the
pressure/velocity
signal representation format, or can, preferably, also be the DirAC parameter
representa-
tion format.
This format is a compact format that, additionally, allows a significant
amount of user in-
teraction on the one hand and that is, on the other hand, useful with respect
to a required
bitrate for representing an audio signal.
In accordance with a further aspect of the present invention, a synthesis of a
plurality of
audio scenes can be advantageously performed by combing two or more different
DirAC
descriptions. Both these different DirAC descriptions can be processed by
combining the
scenes in the parameter domain or, alternatively, by separately rendering each
audio sce-
ne and by then combining the audio scenes that have been rendered from the
individual
DirAC descriptions in the spectral domain or, alternatively, already in the
time domain.
This procedure allows for a very efficient and nevertheless high quality
processing of dif-
ferent audio scenes that are to be combined into a single scene representation
and, par-
ticularly, a single time domain audio signal.
A further aspect of the invention is advantageous in that a particularly
useful audio data
converted for converting object metadata into DirAC rnetadata is derived where
this audio
data converter can be used in the framework of the first, the second or the
third aspect or
can also be applied independent from each other. The audio data converter
allows effi-
ciently converting audio object data, for example, a waveform signal for an
audio object,
and corresponding position data, typically, with respect to time for
representing a certain
trajectory of an audio object within a reproduction setting up into a very
useful and com-
pact audio scene description, and, particularly, the DirAC audio scene
description format.
While a typical audio object description with an audio object waveform signal
and an audio
object position metaciata is related to a particular reproduction setup or,
generally, is reiat-
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
6
ad to a certain reproduction coordinate system, the DirAC description is
particularly useful
in that it is related to a listener or microphone position and is completely
free of any limita-
tions with respect to a loudspeaker setup or a reproduction setup.
Thus, the DirAC description generated from audio object metadata signals
additionally
allows for a very useful and compact and high quality combination of audio
objects differ-
ent from other audio object combination technologies such as spatial audio
object coding
or amplitude panning of objects in a reproduction setup.
An audio scene encoder in accordance with a further aspect of the present
invention is
particularly useful in providing a combined representation of an audio scene
having DirAC
metadata and, additionally, an audio object with audio object metadata.
Particularly, in this situation, it is particularly useful and advantageous
for a high interactiv-
in order to generate a combined metadata description that has DirAC metadata
on the
one hand and, in parallel, object metadata on the other hand. Thus, in this
aspect, the
object metadata is not combined with the DirAC metadata, but is converted into
DirAC-like
metadata so that the object metadata comprises at direction or, additionally,
a distance
and/or a diffuseness of the individual object together with the object signal.
Thus, the ob-
ject signal is converted into a DirAC-like representation so that a very
flexible handling of
a DirAC representation for a first audio scene and an additional object within
this first au-
dio scene is allowed and made possible. Thus, for example, specific objects
can be very
selectively processed due to the fact that their corresponding transport
channel on the one
hand and DirAC-style parameters on the other hand are still available.
in accordance with a further aspect of the invention, an apparatus or method
for perform-
ing a synthesis of audio data are particularly useful in that a manipulator is
provided for
manipulating a DirAC description of one or more audio objects, a DirAC
description of the
multichannel signal or a DirAC description of first order Ambisonics signals
or higher Am-
bisonics signals. And, the manipulated DirAC description is then synthesized
using a Di-
rAC synthesizer.
This aspect has the particular advantage that any specific manipulations with
respect to
any audio signals are very usefully and efficiently performed in the DirAC
domain, i.e., by
manipulating either the transport channel of the DirAC description or by
alternatively ma-
nipulating the parametric data of the DirAC description. This modification is
substantially
more efficient and more practical to perform in the DirAC domain compared to
the ma-
nipulation in other domains. Particularly, position-dependent weighting
operations as pre-
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
7
ferred manipulation operations can be particularly performed in the DirAC
domain. Thus,
in a specihc embodiment, a conversion of a corresponding signal representation
in the
DirAC domain and, then, performing the manipulation within the DirAC domain is
a partic-
ularly useful application scenario for modern audio scene processing and
manipulation.
Preferred embodiments are subsequently discussed with respect to their
accompanying
drawings, in which:
Fig. la is a block diagram of a preferred implementation of an
apparatus or method
for generating a description of a combined audio scene in accordance with
a first aspect of the invention;
Fig. lb is an implementation of the generation of a combined audio
scene, where
the common format is the pressure/velocity representation;
Fig. lc is a preferred implementation of the generation of a combined
audio scene,
where the DirAC parameters and the DirAC description is the common for-
mat;
Fig. id is a preferred implementation of the combiner in Fig. lc
illustrating two dif-
ferent alternatives for the implementation of the combiner of DirAC parame-
ters of different audio scenes or audio scene descriptions;
Fig. le is a preferred implementation of the generation of a combined
audio scene
where the common format is the B-forrnat as an example for an Ambisonics
representation;
Fig. If is an illustration of an audio object/DirAC converter useful
in the context of,
of example, Fig. lc or 1d or useful in the context of the third aspect
relating
to a metadata converter;
Fig. lg is an exemplary illustration of a 5.1 multichannel signal into
a DirAC de-
scription;
Fig_ lh is a further illustration the conversion of a multichannel format
into the Di-
rAC format in the context of an encoder and a decoder side;
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
8
F. 2a illustrates an embodiment of an apparatus or method for
performing a syn-
thesis of a plurality of audio scenes in accordance with a second aspect of
the present invention;
Fig. 2b illustrates, a preferred implementation of the DirAC synthesizer of
Fig. 2a;
Fig. 2c illustrates a further implementation of the DirAC synthesizer
with a combi-
nation of rendered signals;
Fig. 2d illustrates an implementation of a selective manipulator either
connected
before the scene combiner 221 of Fig. 2b or before the combiner 225 of
Fig. 20;
Fig, 3a is a preferred implementation of an apparatus or method for
performing and
audio data conversion in accordance with a third aspect of the present in-
vention;
Fig. 3b is a preferred implementation of the metadata converter also
illustrated in
Fig. if;
Fig. 3c is a flowchart for performing a further implementation of a
audio data con-
version via the pressure/velocity domain;
Fig. 3d illustrates a flowchart for performing a combination within
the DirAC do-
main;
Fig. 3e illustrates a preferred implementation for combining different
DirAC descrip-
tions, for example as illustrated in Fig. id with respect to the first aspect
of
the present invention;
Fig. 3f illustrates the conversion of an object position data into a
DirAC parametric
representation];
Fig. 4a illustrates a preferred implementation of an audio scene
encoder in accord-
ance with a fourth aspect of the present invention for generating a com-
bined metadata description comprising the DirAC metadata and the object
metadata;
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
9
Fig. 4b illustrates a preferred embodiment with respect to the fourth
aspect of the
present invention;
Fig. 5a illustrates a preferred implementation of an apparatus for
performing a syn-
thesis of audio data or a corresponding method in accordance with a fifth
aspect of the present invention;
Fig. 5b illustrates a preferred implementation of the DirAC
synthesizer of Fig. 5a;
Fig. Sc illustrates a further alternative of the procedure of the
manipulator of Fig.
5a;
Fig. 5d illustrates a further procedure for the implementation of the
Fig. 5a manipu-
lator;
Fig. 6 illustrates an audio signal converter for generating, from a
mono-signal and
a direction of arrival information, Le., from an exemplary DirAC description,
where the diffuseness is, for example, set to zero, a B-format representa-
tion comprising an omnidirectional component and directional components
in X, Y and Z directions;
Fig. 7a illustrates an implementation of a DirAC analysis of a 8-
Format microphone
Si gnal;
Fig. 7b illustrates an implementation of a DirAC synthesis in accordance
with a
known procedure;
Fig. 8 illustrates a flowchart for illustrating further embodiments
of, particularly, the
Fig. la embodiment;
Fig. 9 is the encoder side of the DirAC-based spatial audio coding
supporting dif-
ferent audio formats,
Fig. 10 is a decoder of the DirAC-based spatial audio coding
delivering different
audio formats;
Fig. 11 is a system overview of the DirAC-based encoder/decoder
combining dif-
ferent input formats in a combined B-format;
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
Fig. 12 is a system overview of the DirAC-based encoder/decoder
combining in the
pressure/velocity domain;
5 Fig. 13 is a system overview of the DirAC-based encoder/decoder
combining dif-
ferent input formats in the DirAC domain with the possibility of object ma-
nipulation at the decoder side;
Fig. 14 is a system overview of the DirAC-based encoder/decoder
combining dil-
1 0 ferent input formats at the decoder-side through a DirAC metadata
combin-
er;
Fig. 15 is a system overview of the DirAC-based encoder/decoder
combining dif-
ferent input formats at the decoder-side in the DirAC synthesis; and
Fig_ 16a-f illustrates several representations of useful audio formats in
the context of
the first to fifth aspects of the present invention.
Fig. la illustrates a preferred embodiment of an apparatus for generating a
description of
a combined audio scene. The apparatus comprises an input interface 100 for
receiving a
first description of a first scene in a first format and a second description
of a second sce-
ne in a second format, wherein the second format is different from the first
format. The
format can be any audio scene format such as any of the formats or scene
descriptions
illustrated from Figs. 16a to 16f_
Fig. 16a, for example, illustrates an object description consisting,
typically, of a (encoded)
object 1 waveform signal such as a mono-channel and corresponding metadE.Ita
related to
the position of object 1, where this is information is typically given for
each time frame or a
group of time frames, and which the object 1 waveforms signal is encoded.
Corresponding
representations for a second or further object can be included as illustrated
in Fig. 16a.
Another alternative can be an object description consisting of an object
downmix being a
mono-signal, a stereo-signal with two channels or a signal with three or more
channels
and related object rnetadata such as object energies, correlation information
per
time/frequency bin and, optionally, the object positions_ However, the object
positions can
also be given at the decoder side as typical rendering information and,
therefore, can be
modified by a user. The format in Fig. 16b can, for example, be implemented as
the well-
known SACK: (spatial audio object coding) format,
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
11
Another description of a scene is illustrated in Fig. 16c as a multichannel
description hav-
ing an encoded or non-encoded representation of a first channel, a second
channel, a
third channel, a fourth channel, or a fifth channel, where the first channel
can be the left
channel L, the second channel can be the right channel R, the third channel
can be the
center channel C, the fourth channel can be the left surround channel LS and
the fifth
channel can be the right surround channel RS. Naturally, the multichannel
signal can have
a smaller or higher number of channels such as only two channels for a stereo
channel or
six channels for a 5.1 format or eight channels for a 1.1 format, etc.
A more efficient representation of a multichannel signal is illustrated in
Fig. 16d, where the
channel downmix such as a mono downmix, or stereo downmix or a downmix with
more
than two channels is associated with parametric side information as channel
metadata for,
typically, each time andior frequency bin. Such a parametric representation
can, for ex-
ample, be implemented in accordance with the MPEG surround standard.
Another representation of an audio scene can, for example, be the B-format
consisting of
an omnidirectional signal W, and directional components X, Y, Z as shown in
Fig, 16e.
This would be a first order or FoA signal. A higher order Ambisonics signal,
i.e., an HoA
signal can have additional components as is known in the art.
The Fig. 16e representation is, in contrast to the Fig. 16c and Fig. 16d
representation a
representation that is non-dependent on a certain loudspeaker set up, but
describes a
sound field as experienced at a certain (microphone or listener) position,
Another such sound field description is the DirAC format as, for example,
illustrated in Fig.
16f. The DirAC format typically comprises a DirAC downmix signal which is a
mono or
stereo or whatever downmix signal or transport signal arid corresponding
parametric side
information. This parametric side information is, for example, a direction of
arrival infor-
mation per time/frequency bin and, optionally, diffuseness information per
time/frequency
The input into the input interface 100 of Fig. la can be, for example, in any
one of those
formats illustrated with respect to Fig. 16a to Fig. 161. The input interface
100 forwards the
corresponding format descriptions to a format converter 120. The format
converter 120 is
configured for converting the first description into a common format arid for
converting the
second description into the same common format, when the second format is
different
from the common format. When, however, the second format is already in the
common
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
12
format, then the format converter only convers the first description into the
common for-
mat, since the first description is in a format different from the common
format.
Thus, at the output of the format converter or, generally, at the input of a
format combiner,
6 there does exist a representation of the first scene in the common format
and the repre-
sentation of the second scene in the same common format. Due to the fact that
both de-
scriptions are now included in one and the same common format, the format
combiner
can now combine the first description and the second description to obtain a
combined
audio scene.
In accordance with an embodiment illustrated in Fig. le, the format converter
120 is con-
figured to convert the first description into a first B-format signal as, for
example, illustrat-
ed at 127 in Fig. 1e and to compute the B-format representation for the second
description
as illustrated in Fig. le at 128.
Then, the format combiner 140 is implemented as a component signal adder
illustrated at
146a for the W component. adder, 146b for the X component adder, illustrated
at 146c for
the Y component adder and illustrated at 146d for the Z component adder.
Thus, in the Fig. le embodiment, the combined audio scene can be a B-format
represen-
tation and the B-format signals can then operate as the transport channels and
can then
be encoded via a transport channel encoder 170 of Fig. la. Thus, the combined
audio
scene with respect to B-format signal can be directly input into the encoder
170 of Fig. la
to generate an encoded B-format signal that could then be output via the
output interface
200. In this case, any spatial metadata are not required, but, at the price of
an encoded
representation of four audio signals, i.e., the omnidirectional component W
and the direc-
tional components X, Y, Z.
Alternatively, the common format is the pressure/velocity format as
illustrated in Fig. 1b.
To this end, the format converter 120 comprises a time/frequency analyzer 121
for the first
audio scene and the time/frequency analyzer 122 for the second audio scene or,
general-
ly, the audio scene with number N, where N is an integer number.
Then, for each such spectral representation generated by the spectral
converters 121,
122, pressure and velocity are computed as illustrated at 123 and 124, and,
the format
combiner then is configured to calculate a summed pressure signal on the one
hand by
summing the corresponding pressure signals generated by the blocks 123, 124.
And, ad-
ditionally, an individual velocity signal is calculated as well by each cif
the blocks 123, 124
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
13
and the velocity signals can be added together in order to obtain a combined
pres-
sure/velocity signal.
Depending on the implementation, the procedures in blocks 142, 143 does not
necessarily
have to be performed. Instead, the combined or "summed" pressure signal and
the com-
bined or "summed" velocity signal can be encoded in an analogy as illustrated
in Fig. le of
the B-format signal and this pressure/velocity representation could be encoded
while once
again via that encoder 170 of Fig. la and could then be transmitted to the
decoder without
any additional side information with respect to spatial parameters, since the
combined
pressure/velocity representation already includes the necessary spatial
information for
obtaining a finally rendered high quality sound field on a decoder-side .
In an embodiment, however, it is preferred to perform a DirAC analysis to the
pres-
sure/velocity representation generated by block 141. To this end, the
intensity vector 142
is calculated and, in block 143, the DirAC parameters from the intensity
vector is calculat-
ed, and, then, the combined DirAC parameters are obtained as a parametric
representa-
tion of the combined audio scene. To this end, the DirAC analyzer 180 of Fig.
la is im-
plemented to perform the functionality of block 142 and 143 of Fig. lb. And,
preferably,
the DirAC data is additionally subjected to a rnetaciata encoding operation in
metadata
encoder 190. The metadete encoder 190 typically comprises a quantizer and
entropy
coder in order to reduce the bitrate required for the transmission of the
DirAC parameters.
Together with the encoded DirAC parameters, an encoded transport channel is
also
transmitted. The encoded transport channel is generated by the transport
channel genera-
tor 160 or Fig. la that can, for example, be implemented as illustrated in
Fig. lb by a first
downrnix generator 161 for generating a downrnix from the first audio scene
and a N-th
downrnix generator 162 for generating a downrnix from the N-th audio scene.
Then: the downmix channels are combined in combiner 163 typically by a
straightforward
addition and the combined downrnix signal is then the transport channel that
is encoded
by the encoder 170 of Fig. la. The combined downmix can, for example, be a
stereo pair,
i.e., a first channel arid a second channel of a stereo representation or can
be a mono
channel, i.e., a single channel signal.
in accordance with a further embodiment illustrated in Fig. 1 c, a format
conversion in the
format converter 120 is done to directly convert each of the input audio
formats into the
DirAC format as the common format. To this end, the format converter 120 once
again
forms a time-frequency conversion or a time/frequency analysis in
corresponding blocks
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
14
121 for the first scene and block 122 for a second or further scene. Then.
DirAC parame-
ters are derived from the spectral representations of the corresponding audio
scenes illus-
trated at 125 and 126. The result of the procedure in blocks 125 and 126 ale
DirAC pa-
rameters consisting of energy information per time/frequency tile, a direction
of arrival
information eD0A per time/frequency tile and a diffuseness information i for
each
time/frequency tile. Then, the format combiner 140 is configured to perform a
combination
directly in the DirAC parameter domain in order to generate combined DirAC
parameters
Lp for the diffuseness and ee0A for the direction of arrival. Particularly,
the energy infor-
mation Ei and EN are required by the combiner 144 but are not part of the
final combined
parametric representation generated by the format combiner 140.
Thus, comparing Fig. lc to Fig. le reveals that, when the format combiner 140
already
performs a combination in the DirAC parameter domain, the DirAC analyzer 180
is not
necessary and not implemented_ Instead, the output of the format combiner 140
being the
output of block 144 in Fig. 1 c is directly forwarded to the metadata encoder
190 of Fig_ la
and from there into the output interface 200 so that the encoded spatial
metadata and,
particularly, the encoded combined DirAC parameters are included in the
encoded output
signal output by the output interface 200.
Furthermore, the transport channel generator 160 of Fig. 12 may receive,
already from the
input interface 100, a waveform signal representation for the first scene and
the waveform
signal representation for the second scene. These representations are input
into the
downmix generator blocks 161, 162 and the results are added in block 163 to
obtain a
combined downmix as illustrated with respect to Fig, lb.
Fig. Id illustrates a similar representation with respect to Fig. 1c_ However,
in Fig. id, the
audio object waveform is input into the time/frequency representation
converter 121 for
audio object 1 and 122 for audio object N. Additionally, the metadata are
input, together
with the spectral representation into the DirAC parameter calculators 125, 126
as illustrat-
ad also in Fig_ lc.
However, Fig. id provides a more detailed representation with respect to how
preferred
implementations of the combiner 144 operate. In a first alternative, the
combiner performs
an energy-weighted addition of the individual diffuseness for each individual
object or
scene and, a corresponding energy-weighted calculation of a combined DoA for
each
time/frequency tile is performed as illustrated in the lower equation of
alternative 1.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
However, other implementations can be performed as well. Particularly, another
very effi-
cient calculation is set the diffuseness to zero for the combined DirAC
metadata and to
select, as the direction of arrival for each time/frequency tile the direction
of arrival calcu-
lated from a certain audio object that has the highest energy within the
specific
5 time/frequency tile. Preferably, the procedure in Fig. id is more
appropriate when the in-
put into the input interface are individual audio objects correspondingly
represented a
waveform or mono-signal for each object and corresponding metadata such as
position
information illustrated with respect to Fig. 16a or 16b.
10 However, in the Fig. 1 c embodiment, the audio scene may be any other of
the representa-
tions illustrated in Fig. 16c, 16d, 16e or 16f. Then, there can be metadata or
not, i.e., the
metadata in Fig. 1 c is optional. Then, however, a typically useful
diffuseness is calculated
for a certain scene description such as an Ambisonics scene description in
Fig. 16e and,
then, the first alternative of the way how the parameters are combined is
preferred over
15 the second alternative of Fig. id. Therefore, in accordance with the
invention, the format
converter 120 is configured to convert a high order Ambisonics or a first
order Ambisonics
format into the B-format, wherein the high order Ambisonics format is
truncated before
being converted into the B-format.
In a further embodiment, the format converter is configured to project an
object or a chan-
nel on spherical harmonics at the reference position to obtain projected
signals, and
wherein the format combiner is configured to combine the projection signals to
obtain B-
format coefficients, wherein the object or the channel is located in space at
a specified
position and has an optional individual distance from a reference position.
This procedure
particularly works well for the conversion of object signals or multichannel
signals into first
order or high order Ambisonics signals.
In a further alternative, the format converter 120 is configured to perform a
DirAC analysis
comprising a time-frequency analysis of B-format components and a
determination of
pressure and velocity vectors and where the format combiner is then configured
to corn-
bine different pressure/velocity vectors and where the format combiner further
comprises
the DirAC analyzer 180 for deriving DirAC metadata from the combined
pressure/velocity-
data,
In a further alternative embodiment, the format converter is configured to
extract the Di-
rAC parameters directly from the object metadata of an audio object format as
the first or
second format, where the pressure vector for the DirAC representation is the
object wave
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
16
form signal and the direction is derived from the object position in space or
the diffuseness
is directly given in the object metadata or is set to a default value such as
the zero value.
In a further embodiment, the format converter is configured to convert the
DirAC parame-
tee's derived from the object data format into pressure/velocity data and the
format com-
biner is configured to combine the pressure/velocity data with
pressure/velocity data de-
rived from different description of one or more different audio objects.
However, in a preferred implementation illustrated with respect to Fig. lc and
Id, the for-
mat combiner is configured to directly combine the DirAC parameters derived by
the for-
mat converter 120 so that the combined audio scene generated by block 140 of
Fig. la is
already the final result and a DirAC analyzer 180 illustrated in Fig. la is
not necessary,
since the data output by the format combiner 140 is already in the DirAC
format.
In a further implementation, the format converter 120 already comprises a
DirAC analyzer
for first order Ambisonics or a high order Ambisonics input format or a
multichannel signal
format. Furthermore, the format converter comprises a metadata converter for
converting
the object metadata into DirAC metadata, and such a metadata converter is, for
example,
illustrated in Fig. If at 150 that once again operates on the time/frequency
analysis in
block 121 and calculates the energy per band per time frame illustrated at
147, the direc-
tion of arrival illustrated at block 148 of Fig. if and the diffuseness
illustrated at block 149
of Fig. if. And, the metadata are combined by the combiner 144 for combining
the individ-
ual DirAC metadata streams, preferably by a weighted addition as illustrated
exemplarily
by one of the two alternatives of the Fig. id embodiment.
Multichannel channel signals can be directly converted to B-format. The
obtained B-format
can be then processed by a conventional DirAC. Fig. lg illustrates a
conversion 127 to B-
format and a subsequent DirAC processing 180.
Reference [3] outlines ways to perform the conversion from multi-channel
signal to B-
format. In principle, converting multi-channel audio signals to B-format is
simple, virtual
loudspeakers are defined to be at different positions of the loudspeaker
layout. Fur exam-
ple for 5,0 layout, loudspeakers are positioned on the horizontal plane at
azimuth angles
41-30 and 44-110 degrees. A virtual B-format microphone is then defined to be
in the cen-
ter of the loudspeakers, and a virtual recording is performed. Hence, the \Pk/
channel is
created by summing all loudspeaker channels of the 5.0 audio file. The process
for getting
1/\./ and other B-format coefficients can be then summarized:
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
17
k
W wisi
. 2
i= 3
X = WiSt (COS(0i) cos((pi))
t=
Y=
wfsi (sin(0) cos())
i-
Z = WiSi (Si n (T))
where st are the multichannel signals located in the space at the loudspeaker
positions
defined by the azimuth angle Oi and elevation angle cp i, of each loudspeaker
and m are
weights function of the distance. If the distance is not available or simply
ignored, then
1. Though, this simple technique is limited since it is an irreversible
process. Moreo-
ver since the loudspeaker are usually distributed non-uniformly, there is also
a bias in the
estimation done by a subsequent DirAC analysis towards the direction with the
highest
loudspeaker density. For example in 5,1 layout, there will be a bias towards
the front since
there are more loudspeakers in the front than in the back.
To address this issue, a further technique was proposed in [31 for processing
5.1 multi-
channel signal with DirAC.:, The final coding scheme will then look as
illustrated in Fig lh
showing the B-format converter 127, the DirAC analyzer 180 as generally
described with
respect to element 180 in Fig. 1, and the other elements 190, 1000, 160, 170,
1020.
and/or 220, 240.
In a further embodiment, the output interface 200 is configured to add, to the
combined
format, a separate object description for an audio object, where the object
description
comprises at least one of a direction, a distance, a diffuseness or any other
object attrib-
ute, where this object has a single direction throughout all frequency bands
and is either
static or moving slower than a velocity threshold.
This feature is furthermore elaborated in more detail vvith respect to the
fourth aspect of
the present invention discussed with respect to Fig. 4a and Fig. 4b,
ist.EncpilLng,,Alterna,tive..: Pornbininctand processing..different. audio
rtpresentations
through B-format or eguivalent..reoresentation.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
18
A first realization of the envisioned encoder can be achieved by converting
all input format
into a combined B-format as it is depicted in Fig. 11.
Fig. 11: System overview of the DirAC-based encoder/decoder combining
different input
formats in a combined B-format
Since DirAC is originally designed for analyzing a B-format signal, the system
converts the
different audio formats to a combined B-format signal. The formats are first
individually
converted 120 into a B-format signal before being combined together by summing
their B-
format components VV,X,Y,Z. First Order Ambisonics (FOR) components can be
normal-
ized and re-ordered to a B-format. Assuming FOR is in ACIXI/1\13D format, the
four signals
of the B-format input are obtained by:
1/1/ = Yo /
). _ ==2 i I
73 1
2
Y ,4---- --- Y-1 "13 1
.1-
Where Yriõ denotes the Ambisonics component of order / and index in. ¨1 5; m <
1. Since
FOA components are fully contained in higher order Arnbisonics format, HOR
format
needs only to be truncated before being converted into B-format.
Since objects and channels have determined positions in the space, it is
possible to pro-
ject each individual object and channel on spherical Harmonics (SR) at the
center position
such as recording or reference position. The sum of the projections allows
combining dif-
ferent objects and multiple channels in a single B-format and can be then
processed by
the DirAC analysis. The B-format coefficients (W,X,Y,Z) are then given by:
k I-
W ===-=- >I] 2: wici
=2 '
.,-1 \
k
X = 7 w-s1(cos(91) cos(ipi))
44 '
,-i
k
Y = I wis,(sin(9i) cos(ipi))
.E..-3.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
19
iL Z ==- Ems, (si n ((PO)
where si are independent signals located in the space at positions defined by
the azimuth
angle 6i arid elevation angle (pi, and wi are weights function of the
distance. if the dis-
tance is not available or simply ignored, then wi = 1. For example, the
independent sig-
nals can correspond to audio objects that are located at the given position or
the signal
associated with a loudspeaker channel at the specified position.
In applications where an Ambisonics representation of orders higher than first
order is
desired, the Ambisonics coefficients generation presented above for first
order is extend-
ed by additionally considering higher-order components.
The transport channel generator 160 can directly receive the multichannel
signal, objects
waveform signals, and the higher order Ambisonics components. The transport
channel
generator will reduce the number of input channels to transmit by downmixing
them. The
channels can be mixed together as in MPEG surround in a mono or stereo
downmix,
while object waveform signals can be summed up in a passive way into a mono
downmix.
In addition, from the higher order Ambisonics, it is possible to extract a
lower order repre-
sentation or to create by beamforming a stereo downmix or any other sectioning
of the
space. If the downrnixes obtained from the different input format are
compatible with each
other, they can be combined together by a simple addition operation.
Alternatively, the transport channel generator 160 can receive the same
combined B-
format as that conveyed to the DirAC analysis. In this case, a subset of the
components or
the result of a beamforrning (or other processing) form the transport channels
to be coded
and transmitted to the decoder. In the proposed system, a conventional audio
coding is
required which can be based on, but is not limited to, tho standard 3GPP EVS
codec.
3GPP EVS is the preferred coded choice because of its ability to code either
speech or
music signals at low bit-rates with high quality while requiring a relatively
low delay ena-
bling real-time communications.
At a very low bit-rate, the number of channels to transmit needs to be limited
to one and
therefore only the omnidirectional microphone signal µN of the B-format is
transmitted. If
bit-rate allows, the number of transport channels can be increased by
selecting a subset
of the B-format components. Alternatively, the B--format signals can be
combined into a
beamformer 160 steered to specific partitions of the space. As an example two
cardinids
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
can be designed to point at opposite directions, for example to the left and
the right of the
spatial scene:
= Nifiw + Y
11? = Y
5 These two stereo channels L and R can be then efficiently coded 170 by a
joint stereo
coding. The two signals will be then adequately exploited by the DirAC
Synthesis at the
decoder side for rendering the sound scene. Other beamforming can be
envisioned, for
example a virtual cardloid microphone can be pointed toward any directions of
given azi-
muth 0 and elevation co:
C cos (9) cos(v) X sin(0) cos(co) Y sin(v)Z
Further ways of forming transmission channels can be envisioned that carry
more spatial
information than a single monophonic transmission channel would do
Alternatively, the 4 coefficients of the B-format can be directly transmitted.
In that case the
DirAC metadata can be extracted directly at the decoder side, without the need
of trans-
mitting extra information for the spatial metadata.
Fig.12 shows another alternative method for combining the different input
formats. Fig. 12
also is a system overview of the DirAC-based encoder/decoder combining in Pres-
sure/velocity domain.
Both multichannel signal and Ambisonics components are input to a DirAC
analysis 123,
124. For each input format a DirAC analysis is performed consisting of a time-
frequency
analysis of the B-format components vt '01), x'(n), (n), z' (a) and the
determination of
the pressure and velocity vectors:
k) = W(k)
(n, k) = X1(kin)e, + Yl(k,n)ey Zi(k,n)e,
where is the index of the input and, k and n time and frequency indices of the
time-
frequency tile, and ex, ey, e, represent the Cartesian unit vectors.
f)(71,k) and Li(n,k) are necessary to compute the DirAC parameters, namely DOA
and
diffuseness. The DirAC metadata combiner can exploit that N sources which play
together
result in a linear combination of their pressures and particle velocities that
would be
measured when they are played alone. The combined quantities are then derived
by:
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
21
P(n, k) =11)i (n) k)
U(n, k) = W (n, k)
The combined DirAC parameters are computed 143 through the computation of the
corn-
bined intensity vector:
1 (k, n) = 19itP(k,
where () denotes complex conjugation. The diffuseness of the combined sound
field is
given by:
1 - E (k
cE{E(k,n))
where 41 denotes the temporal averaging operator, c the speed of sound and L'
(k, n) the
sound field energy given by:
Po
E (k , n) = (I (lc , n)jiz IP(k,n)12
4 poc-
The direction of arrival (DOA) is expressed by means of the unit vector
CD0A(k, rt), defined
as
(k, n)
CD0A(k,n) =
11 i.(k,n)11
If an audio object is input, the DirAC parameters can be directly extracted
from the object
metaciata while the pressure vector P?(Ic,n) is the object essence (waveform)
signal. More
precisely, the direction is straightforwardly derived from the object position
in the space,
while the diffuseness is directly given in the object metadata or - if not
available ¨ can be
set by default to zero. From the DirAC parameters the pressure and the
velocity vectors
are directly given by:
k n) (k,n)Pi (k, n)
O(k,n) --1- PE (k, n). eboA(k,n)
i 1.7r
Po c
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
22
The combination of objects or the combination of an object with different
input formats is
then obtained by summing the pressure and velocity vectors as explained
previously.
.. in summary, the combination of different input contributions (Ambisonics,
channels, ob-
jects) is performed in the pressure / velocity domain and the result is then
subsequently
converted into direction / diffuseness DirAC parameters. Operating in
pressure/velocity
domain is the theoretically equivalent to operate in 6-format. The main
benefit of this al-
ternative compared to the previous one is the possibility to optimize the
DirAC analysis
according to each input format as it is proposed in [31 for surround format
5.1.
The main drawback of such a fusion in a combined B-format or pressure/velocity
domain
is that the conversion happening at the front-end of the processing chain is
already a bot-
tleneck for the whole coding system. Indeed, converting audio representations
from high-
er-order Ambisonics, objects or channels to a (first-order) B-format signal
engenders al-
ready a great loss of spatial resolution which cannot be recovered afterwards.
2st Encodiag Alternative: combination and processing in DirAC domain
To circumvent the limitations of converting all input formats into a combined
B-format sig-
nal, the present alternative proposes to derive the DirAC parameters directly
from the orig-
inal format and then to combine them subsequently in the DirAC parameter
domain. The
general overview of such a system is given in Fig. 13. Fig. 13 is a system
overview of the
DirAC-based encoder/decoder combining different input formats in DirAC domain
with the
possibility of object manipulation at the decoder side.
In the following, we can also consider individual channels of a multichannel
signal as an
audio object input for the coding system. The object metaclata is then static
over time and
represent the loudspeaker position and distance related to listener position.
The objective of this alternative solution is to avoid the systematic
combination of the dif-
ferent input formats into to a combined B-format or equivalent representation.
The aim is
to compute the DirAC parameters before combining them. The method avoids then
any
biases in the direction and diffuseness estimation due to the combination.
Moreover, it can
optimally exploit the characteristics of each audio representation during the
DirAC analy-
sis or while delormining the DirAC parameters.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
23
The combination of the DirAC metadata occurs after determining 125, 126, 126a
for each
input format the DirAC parameters, diffuseness, direction as well as the
pressure con-
tained in the transmitted transport channels. The DirAC analysis can estimate
the parame-
ters from an intermediate B-format, obtained by converting the input format as
explained
previously. Alternatively, DirAC parameters can be advantageously estimated
without go-
ing through B-format but directly from the input format, which might further
improve the
estimation accuracy. For example in [7], it is proposed to estimate the
diffuseness direct
from higher order Ambisonics. In case of audio objects, a simple metadata
convertor 150
in Fig. 15 can extract from the object metadata direction and diffuseness for
each object,
The combination 144 of the several Dirac metadata streams into a single
combined DirAC
metadata stream can be achieved as proposed in plj. For some content it is
much better
to directly estimate the DirAC parameters from the original format rather than
converting it
to a combined B-format first before performing a DirAC analysis. Indeed, the
parameters,
direction and diffuseness, can be biased when going to a B-format [3] or when
combining
the different sources. Moreover, this alternative allows a
Another simpler alternative can average the parameters of the different
sources by
weighting them according to their energies:
0(k, n) = ¨77-777-77-7- El (k, a) (k,n)
1c,
-1
1
'104(k n) = (k,n))Ei n) e
0A(k. V(k.3171)L9(k
For each object there is the possibility to still send its own direction and
optionally dis-
tance, diffuseness or any other relevant object attributes as part of the
transmitted bit-
stream from the encoder to the decoder (see e.g., Figs, 4a, 4b). This extra
side-
information will enrich the combined DirAC metadata and will allow the decoder
to resti-
tute and or manipulate the object separately. Since an object has a single
direction
throughout all frequency bands and can be considered either static or slowly
moving, the
extra information requires to be updated less frequently than other DirAC
parameters and
will engender only very low additional bit-rate.
At the decoder side, directional filtering can be performed as educated in [5)
for rnanipulat-
ing objects. Directional filtering is based upon a shaft-time spectral
attenuation technique.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
24
It is performed in the Spectral domain by a zero-phase gain function, which
depends upon
the direction of the objects. The direction can be contained in the bitstrearn
if directions of
objects were transmitted as side-information. Otherwise, the direction could
also be given
interactively by the user.
3rd _Alternative.: combination. at decoder side
Alternatively, the combination can be performed at the decoder side. Fig, 14
is a system
overview of the DirAC-based encoder/decoder combining different input formats
at de-
coder side through a DirAC metadata combiner. In Fig. 14, the DirAC-based
coding
scheme works at higher bit rates than previously but allows for the
transmission of individ-
ual DirAC metadata. The different DirAC rnetadata streams are combined 144 as
for ex-
ample proposed in [1] in the decoder before the DirAC synthesis 220, 240. The
DirAC
rnetadata combiner 144 can also obtain the position of an individual object
for subsequent
manipulation of the object in DirAC analysis.
Fig. 15 is a system overview of the DirAC-based encoder/decoder combining
different
input formats at decoder side in DirAC synthesis. If bit-rate allows, the
system can fur-
ther be enhanced as proposed in Fig. 15 by sending for each input component
(FONI--10A, MC, Object.) its own downrnix signal along with its associated
DirAC metada-
ta. Still, the different DirAC streams share a common DirAC synthesis 220, 240
at the de-
coder to reduce complexity.
Fig. 2a illustrates a concept for performing a synthesis of a plurality of
audio scenes in
accordance with a further, second aspect of the present invention. An
apparatus illustrat-
ed in Fig. 2a comprises an input interface 100 for receiving a first DirAC
description of a
first scene and for receiving a second DirAC description of a second scene and
one or
more transport channels.
Furthermore, a DirAC synthesizer 220 is provided for synthesizing the
plurality of audio
scenes in a spectral domain to obtain a spectral domain audio signal
representing the
plurality of audio scenes. Furthermore, a spectrum-time converter 214 is
provided that
converts the spectral domain audio signal into a time domain in order to
output a time do-
main audio signal that can be output by speakers, for example. In this case,
the DirAC
synthesizer is configured to perform rendering of loudspeaker output signal.
Alternatively,
the audio signal could be a stereo signal that can be output to a headphone.
Again, alter-
natively, the audio signal output by the spectrum-time converter 214 can be a
B-format
sound field description. All these signals, i.e., loudspeaker signals for more
than two
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
channels, headphone signals or sound field descriptions are time domain signal
for further
processing such as outputting by speakers or headphones or tor transmission or
storage
in the case of sound field descriptions such as first order Ambisonies signals
or higher
order Arnbisonics signals.
5
Furthermore, the Fig, 2a device additionally comprises a user interface 260
for controlling
the DirAC synthesizer 220 in the spectral domain. Additionally, one or more
transport
channels can be provided to the input interface 100 that are to be used
together with the
first and second DirAC descriptions that are, in this case, parametric
descriptions provid-
10 trig, for each time/frequency tile, a direction of arrival information
and, optionally, addition-
ally a diffuseness information.
Typically, the two different DirAC descriptions input into the interface 100
in Fig. 2a de-
scribe two different audio scenes. In this case, the DirAC synthesizer 220 is
configured to
15 perform a combination of these audio scenes. One alternative of the
combination is illus-
trated in Fig. 2b. Here, a scene combiner 221 is configured to combine the two
DirAC de-
scription in the parametric domain, i.e., the parameters are combined to
obtain combined
direction of arrival (DoA) parameters and optionally diffuseness parameters at
the output
of block 221. This data is then introduced into to the DirAC renderer 222 that
receives,
20 additionally, the one or more transport channels in order to channels in
order to obtain the
spectral domain audio signal 222. The combination of the DirAC parametric data
is pref-
erably performed as illustrated in Fig_ id and, as is described with respect
to this figure
and, particularly, with respect to the first alternative.
25 Should at least one of the two descriptions input into the scene
combiner 221 include dif-
fuseness values of zero or no diffuseness values at all, then, additionally,
the second al-
ternative can be applied as well as discussed in the context of Fig. -Id.
Another alternative is illustrated in Fig_ 2c. in this procedure, the
individual DirAC descrip-
tions are rendered by means of a first DirAC renderer 223 for the first
description and a
second DirAC renderer 224 for the second description and at the output of
blocks 223 and
224, a first and the second spectral domain audio signal are available, and
these first and
second spectral domain audio signals are combined within the combiner 225 to
obtain, at
the output of the combiner 225, a spectral domain combination signal.
Exemplarily, the first DirAC renderer 223 and the second DirAC renderer 224
are config-
ured to generate a stereo signal having a left channel L and a right channel
R. Then, the
combiner 225 is configured to combine the left channel from block 223 and the
left chan-
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
26
nel from block 224 to obtain a combined left channel. Additionally, the right
channel from
block 223 is added with the right channel from block 224, and the result is a
combined
right channel at the output of block 225.
For individual channels of a multichannel signal, the analogous procedure is
performed,
i.e., the individual channels are individually added, so that always the same
channel from
a DirAC renderer 223 is added to the corresponding same channel of the other
DirAC
renderer and so on. The same procedure is also performed for, for example. B-
format or
higher order Ambisonics signals. When, for example, the first DirAC renderer
223 outputs
signals W, X, Y, Z signals, and the second DirAC renderer 224 outputs a
similar format,
then the combiner combines the two omnidirectional signals to obtain a
combined omnidi-
rectional signal W, and the same procedure is performed also for the
corresponding com-
ponents in order to finally obtain a X, Y and a Z combined component.
Furthermore, as already outlined with respect to Fig. 2a, the input interface
is configured
to receive extra audio object metadata for an audio object. This audio object
can already
be included in the first or the second DirAC description or is separate from
the first and the
second DirAC description. In this case, the DirAC synthesizer 220 is
configured to selec-
tively manipulate the extra audio object metadata or object data related to
this extra audio
.. object metadata to, for example, perform a directional filtering based on
the extra audio
object metadata or based on user-given direction information obtained from the
user inter-
face 260. Alternatively or additionally, and as illustrated in Fig. 2d, the
DirAC synthesizer
220 is configured for performing, in the spectral domain, a zero-phase gain
function, the
zero-phase gain function depending upon a direction of an audio object,
wherein the di-
rection is contained in a bit stream if directions of objects are transmitted
as side infoi-
rnation, or wherein the direction of is received from the user interface 260.
The extra audio
object metadata input into the interface 100 as an optional feature in Fig. 2a
reflects the
possibility to still send, for each individual object its own direction and
optionally distance,
diffuseness and any other relevant object attributes as part of the
transmitted bit stream
from the encoder to the decoder. Thus, the extra audio object metadata may
related to an
object already included in the first DirAC description or in the second DirAC
description or
is an additional object not included in the first DirAC description and in the
second 04-AC
description already.
However, it is preferred to have the extra audio object metadata already in a
DirAC-style,
i.e., a direction of arrival information and, optionally, a diffuseness
information although
typical audio objects have a diffusion of zero, i.e., or concentrated to their
actual position
resulting in a concentrated and specific direction of arrival that is constant
over all fre-
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
27
quency bands and that is, with respect to the frame rate, either static or
slowly moving.
Thus, since such an object has a single direction throughout all frequency
bands and can
be considered either static or slowly moving, the extra information requires
to be updated
less frequently than other DirAC parameters and will, therefore, incur only
very low addi-
tiona.1 bitrato. Exemplarily, while the first and the second DirAC
descriptions have DoA
data and diffuseness data for each spectral band and for each frame, the extra
audio ob-
ject metadata only requires a single DoA data for all frequency bands and this
data only
for every second frame or, preferably, every third, fourth, fifth or even
every tenth frame in
the preferred embodiment.
Furthermore, with respect to directional filtering performed in the DirAC
synthesizer 220
that is typically included within a decoder on a decoder side of an
encoder/decoder sys-
tem, the DirAC synthesizer can, in the Fig. 2b alternative, perform the
directional filtering
within the parameter domain before the scene combination or again perform the
direction-
at filtering subsequent to the scene combination. However, in this case, the
directional
filtering is applied to the combined scene rather than the individual
descriptions.
Furthermore, in case an audio object is not included in the first or the
second description,
but is included by its own audio object rnetadata, the directional filtering
as illustrated by
the selective manipulator can be selectively applied only the extra audio
object, for which
the extra audio object rnetaciata exists without effecting the first or the
second DirAC de-
scription or the combined DirAC description. For the audio object itself,
there either exists
a separate transport channel representing the object waveform signal or the
object wave-
forms signal is included in the downmixed transport channel,
A selective manipulation as illustrated, for example, in Fig. 2b may, for
example, proceed
in such a way that a certain direction of arrival is given by the direction of
audio object
introduced in Fig. 2d included in the bit stream as side information or
received from a user
interface. Then, based on the user-given direction or control information, the
user may, for
example, outline that, from a certain direction, the audio data is to be
enhanced or is to be
attenuated. Thus, the object (metadata) for the object under consideration is
amplified or
attenuated.
In the case of actual waveform data as the object data introduced into the
selective ma-
nioulator 226 from the left in Fig. 2d, the audio data would be actually
attenuated or en-
hanced depending on the control information. However, in the case of object
data having,
in addition to direction of arrival and optionally diffuseness or distance, a
further energy
information, then the energy information for the object would be reduced in
the case of a
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
28
required attenuation for the object or the energy information would be
increased in the
case of a required amplification of the object data.
Thus, the directional filtering is based upon a short-time spectral
attenuation technique,
and it is performed it the spectral domain by a zero-phase gain function which
depends
upon the direction of the objects. The direction can be contained in the bit
stream if direc-
tions of objects were transmitted as side-information. Otherwise, the
direction could also
be given interactively by the user. Naturally, the same procedure cannot only
be applied to
the individual object given and reflected by the extra audio object metadata
typically pro-
vided by DoA data for all frequency bands and DoA data with a low update ratio
with re-
spect to the frame rate and also given by the energy information for the
object, but the
directional filtering can also be applied to the first DirAC description
independent from the
second DirAC description or vice versa or can be also applied to the combined
DirAC de-
scription as the case may be.
Furthermore, it is to be noted that the feature with respect to the extra
audio object data
can also be applied in the first aspect of the present invention illustrated
with respect to
Figs. la to if. Then, the input interface 100 of Fig. la additionally receives
the extra audio
object data as discussed with respect to Fig. 2a, and the format combiner may
be imple-
mented as the DirAC synthesizer in the spectral domain 220 controlled by a
user interface
260.
Furthermore, the second aspect of the present invention as illustrated in Fig.
2 is different
from the first aspect in that the input interface receives already two DirAC
descriptions,
i.e., descriptions of a sound field that are in the same format and,
therefote, fur the second
aspect, the format converter 120 of the first aspect is not necessarily
required,
On the other hand, when the input into the format combiner 140 of Fig. in
consists of two
DirAC descriptions, then the format combiner 140 can be implemented as
discussed with
respect to the second aspect illustrated in Fig. 2a, or, alternatively, the
Fig. 2a devices
220, 240, can be implemented as discussed with respect to the format combiner
140 of
Fig, la of the first aspect,
Fig. 3a illustrates an audio data converter comprising an input interface 100
for receiving
an object description of an audio object having audio object metadata.
Furthermore, the
input interface 100 is followed by a metadata converter 150 also corresponding
to the
metadata converters 125, 126 discussed with respect to the first aspect of the
present
invention for converting the audio object metadata into DirAC metadata. The
output of the
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
29
Fig. 3a audio converter is constituted by an output interface 300 for
transmitting or storing
the DirAC metadata. The input interface 100 may, additionally receive a
waveform signal
as illustrated by the second arrow input into the interface 100. Furthermore,
the output
interface 300 may be implemented to introduce, typically an encoded
representation of the
waveform signal into the output signal output by block 300. If the audio data
converter is
configured to only convert a single object description including metadata,
then the output
interface 300 also provides a DirAC description of this single audio object
together with
the typically encoded waveform signal as the DirAC transport channel.
Particularly, the audio object metadata has an object position, and the DirAC
metadata
has a direction of arrival with respect to a reference position derived from
the object posi-
tion. Particularly, the metadata converter 150, 126, 126 is configured to
convert DirAC
parameters derived from the object data format into pressure/velocity data,
and the
metadata converter is configured to apply a DirAC analysis to this
pressure/velocity data
as, for example, illustrated by the flowchart of Fig. 3c consisting of block
302, 304, 306.
For this purpose, the DirAC parameters output by block 306 have a better
quality than the
DirAC parameters derived from the object metadata obtained by block 302, i.e.,
are en-
hanced DirAC parameters. Fig. 3b illustrates the conversion of a position for
an object into
the direction of arrival with respect to a reference position for the specific
object.
Fig_ 3f illustrates a schematic diagram for explaining the functionality of
the metadata con-
verter 150. The metadata converter 150 receives the position of the object
indicated by
vector P in a coordinate system. Furthermore, the reference position, to which
the DirAC
metadata are to be related is given by vector R in the same coordinate system.
Thus, the
direction of arrival vector DoA extends from the tip of vector R to the tip of
vector 13. Thus,
the actual DoA vector is obtained by subtracting the reference position R
vector from the
object position P vector.
In order to have a normalized DoA information indicated by the vector DoA, the
vector
difference is divided by the magnitude or length of the vector DoA.
Furthermore, and
should this be necessary and intended, the length of the DoA vector can also
be included
into the metadata generated by the metadata converter 150 so that,
additionally, the dis-
tance of the object from the reference point is also included in the metadata
so that a se-
lective manipulation of this object can also be performed based on the
distance of the
object from the reference position. Particularly, the extract direction block
148 of Fig. 11
may also operate as discussed with respect to Fig. 31, although other
alternatives for cal-
culating the DoA information and, optionally, the distance information can be
applied as
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
well. Furthermore, as already discussed with respect to Fig. 3a, blocks 125
and 126 illus-
trated in Fig. lc or Id may operate in the similar way as discussed with
respect to Fig. 3f.
Furthermore, the Fig. 3a device may be configured to receive a plurality of
audio object
5 descriptions, and the metadata converter is configured to convert each
metadata descrip-
tion directly into a DirAC description and, then, the metadata converter is
configured to
combine the individual DirAC metadata descriptions to obtain a combined DirAC
descrip-
tion as the DirAC metadata illustrated in Fig. 3a. In one embodiment, the
combination is
performed by calculating 320 a weighting factor for a first direction of
arrival using a first
10 energy and by calculating 322 a weighting factor for a second direction
of arrival using a
second energy, where the direction of arrival is processed by blocks 320, 332
related to
the same time/frequency bin. Then, in block 324, a weighted addition is
performed as also
discussed with respect to item 144 in Fig. Id. Thus, the procedure illustrated
in Fig. 3a
represents an embodiment of the first alternative Fig. id.
However, with respect to the second alternative, the procedure would be that
all diffuse-
ness are set to zero or to a small value and, for a time/frequency bin, all
different direction
of arrival values that are given for this time/frequency bin are considered
and the largest
direction of arrival value is selected to be the combined direction of arrival
value for this
time/frequency bin. In other embodiments, one could also select the second to
largest
value provided that the energy information for these two direction of arrival
values are not
so different. The direction of arrival value is selected whose energy is
either the largest
energy among the energies from the different contribution for this time
frequency bin or
the second or the third highest energy.
Thus, the third aspect as described with respect to Figs. 3a to 3f are
different from the first
aspect in that the third aspect is also useful for the conversion of a single
object descrip-
tion into a DirAC metadata. Alternatively, the input interface 100 may receive
several ob-
ject descriptions that are in the same object/metadata format. Thus, any
format converter
as discussed with respect to the first aspect in Fig. la is not required.
Thus, the Fig_ 3a
embodiment may be useful in the context of receiving two different object
descriptions
using different object waveform signals and different object metadata as the
first scene
description and the second description as input into the format combiner 140,
and the
output of the metadata converter 150, 125, 126 or 148 may be a DirAC
representation
with DirAC metadata and, therefore, the DirAC analyzer 180 of Fig. 1 is also
net required.
However, the other elements with respect to the transport channel generator
160 corre-
sponding to the downmixer 163 of Fig. 3a can be used in the context of the
third aspect as
well as the transport channel encoder 170, the metadata encoder 190 and, in
this context,
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
31
the output interface 300 of Fig, 3a corresponds to the output interface 200 of
Fig. la.
Hence, all corresponding descriptions given with respect to the first aspect
also apply to
the third aspect as well.
Figs. 4a, 4b illustrate a fourth aspect of the present invention in the
context of an appa-
ratus for performing a synthesis of audio data. Particularly, the apparatus
has an input
interface 100 for receiving a DirAC description of an audio scene having DirAC
metadata
and additionally for receiving an object signal having object rnetadata. This
audio scene
encoder illustrated in Fig. 4b additionally comprises the metadata generator
400 for gen-
crating a combined metadata description comprising the DirAC metadata on the
one hand
and the object metadata on the other hand. The DirAC metadata comprises the
direction
of arrival for individual time/frequency tiles and the object metadata
comprises a direction
or additionally a distance or a diffuseness of an individual object.
Particularly, the input interface 100 is configured to receive, additionally,
a transport signal
associated with the DirAC description of the audio scene as illustrated in
Fig. 4b, and the
input interface is additionally configured for receiving an object waveform
signal associat-
ed with the object signal. Therefore, the scene encoder further comprises a
transport sig-
nal encoder for encoding the transport signal and the object waveform signal,
and the
transport encoder 170 may correspond to the encoder 170 of Fig. la.
Particularly, the metadata generator 140 that generates the combined metadata
may be
configured as discussed with respect to the first aspect, the second aspect or
the third
aspect. And, in a preferred embodiment, the metadata generator 400 is
configured to
generate, for the object metadata, a single broadband direction per time,
i.e., for a certain
time frame, and the metadata generator is configured to refresh the single
broadband di-
rection per time less frequently than the DirAC metadata.
The procedure discussed with respect to Fig. 4b allows to have combined
metadata that
has metadata for a full DirAC description and that has, in addition, metadata
for an addi-
tional audio object, but in the DirAC format so that a very useful DirAC
rendering can be
performed by, at the same time, a selective directional filtering or
modification as already
discussed with respect to the second aspect can be performed.
.. Thus, the fourth aspect of the present invention and, particularly, the
metadata generator
400 represents a specific format converter where the common format is the
DirAC format,
and the input is a DirAC description for the first scene in the first format
discussed with
respect to Fig. la and the second scene is a single or a combined such as SACO
object
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
32
signal. Hence, the output of the format converter 120 represents the output of
the metada-
ta generator 400 but, in contrast to an actual specific combination of the
metadata by one
of the two alternatives, for example, as discussed with respect to Fig. Id,
the object
metadata is included in the output signal, Le., the 'combined metadata"
separate from the
metadata for the DirAC description to allow a selective modification for the
object data.
Thus, the "direction/distance/diffuseness" indicated at item 2 at the right
hand side of Fig.
4a corresponds to the extra audio object metadata input into the input
interface 100 of Fig.
2a, but, in the embodiment of Ho. 4a, for a single DirAC description only.
Thus, in a
sense, one could say that Fig. 2a represents a decoder-side implementation of
the en-
coder illustrated in Fig. 4a, 4b with the provision that the decoder side of
Fig. 2a device
receives only a single DirAC description and the object metadata generated by
the
rnetadata generator 2100 within the same bit stream as the "extra audio object
metadata".
Thus, a completely different modification of the extra object data can be
performed when
the encoded transport signal has a separate representation of the object
waveform signal
separate from the DirAC transport stream. And, however, the transport encoder
170
downmixes both data, Le., the transport channel for the DirAC description and
the wave-
form signal from the object, then the separation will be less perfect, but by
means of addi-
tional object energy information, even a separation from a combined downrnix
channel
and a selective modification of the object with respect to the DirAC
description is availa-
ble.
Fig. 5a to 5d represent a further of fifth aspect of the invention in the
context of an appa-
ratus for performing a synthesis of audio data. To this end, an input
interface 100 is pro-
vided for receiving a DirAC description of one or more audio objects and/or a
DirAC de-
scription of a multi-channel signal and/or a DirAC description of a first
order Ambisonics
signal and/or a higher order Ambisonics signal, wherein the DirAC description
comprises
position information of the one or more objects or a side information for the
first order Am-
bisonics signals or the high order Ambisonics signals or a position
information for the mul-
ti-channel signal as side information or from a user interface.
Particularly, a manipulator 500 is configured for manipulating the DirAC
description of the
one or more audio objects, the DirAC description of the multi-channel signal,
the DirAC
description of the first order Ambisonics signals or the DirAC description of
the high order
Ambisonics signals to obtain a manipulated DirAC description. In order to
synthesize this
manipulated DirAC description, a DirAC synthesizer 220, 240 is configured for
synthesiz-
ing this manipulated DirAC description to obtain synthesized audio data.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
33
In a preferred embodiment, the DirAC synthesizer 220, 240 comprises a DirAC
renderer
222 as illustrated in Fig. 5b and the subsequently connected spectral-time
converter 240
that outputs the manipulated time domain signal. Particularly, the manipulator
500 is con-
.. figured to perform a position-dependent weighting operation prior to DirAC
rendering.
Particularly, when the DirAC synthesizer is configured to output a plurality
of objects of a
first order Ambisonios signals or a high order Ambisonics signal or a multi-
channel signal,
the DirAC synthesizer is configured to use a separate spectral-time converter
for each
object or each component of the first or the high order Ambisonics signals or
for each
channel of the multichannel signal as illustrated in Fig. 5d at blocks 506,
508. As outlined
in block 510 then the output of the corresponding separate conversions are
added togeth-
er provided that all the signals are in a common format, i.e., in compatible
format.
Therefore, in case of the input interface 100 of Fig. 5a, receiving more than
one, i.e., two
or three representations, each representation could be manipulated separately
as illus-
trated in block 502 in the parameter domain as already discussed with respect
to Fig. 2b
or 2c, and, then, a synthesis could be performed as outlined in block 504 for
each manipu-
lated description, and the synthesis could then be added in the time domain as
discussed
with respect to block 510 in Fig. 5d. Alternatively, the result of the
individual DirAC syn-
thesis procedures in the spectral domain could already be added in the
spectral domain
and then a single time domain conversion could be used as well. Particularly,
the manipu-
lator 500 may be implemented as the manipulator discussed with respect to Fig.
2d or
discussed with respect to any other aspect before.
Hence, the fifth aspect of the present invention provides a significant
feature with respect
to the fact, when individual DirAC descriptions of very different sound
signals are input,
and when a certain manipulation of the individual descriptions is performed as
discussed
with respect to block 500 of Fig. 5a, where an input into the manipulator 500
may be a
.. DirAC description of any format, including only a single format, while the
second aspect
was concentrating on the reception of at least two different DirAC
descriptions or where
the fourth aspect, for example, was related to the reception of a DirAC
description on the
one hand and an object signal description on the other hand.
.. Subsequently, reference is made to Fig. 6, Fig, 6 illustrates another
implementation for
performing a synthesis different from the DirAC synthesizer. When, for
example, a sound
field analyzer generates, for each source signal, a separate mono signal S and
an original
direction of arrival and when, depending on the translation information, a new
direction of
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
34
arrival is calculated, then the Ambisonics signal generator 430 of Fig. 6, for
example,
would be used to generate a sound field description for the sound source
signal, i.e. the
mono signal S but for the new direction of arrival (DoA) data consisting of a
horizontal
angle 0 or an elevation angle 6 and an azimuth angle cp. Then, a procedure
performed by
.. the sound field calculator 420 of Fig. 6 would be to generate, for example,
a first-order
Ambisonics sound field representation for each sound source with the new
direction of
arrival and, then, a further modification per sound source could be performed
using a scal-
ing factor depending on the distance of the sound field to the new reference
location and,
then, all the sound fields from the individual sources could superposed to
each other to
finally obtain the modified sound field, once again, in, for example, an
Ambisonics repre-
sentation related to a certain new reference location_
When one interprets that each time/frequency bin processed by the DirAC
analyzer 422
represents a certain (bandwidth limited) sound source, then the Ambisonics
signal gen-
.. erator 430 could be used, instead of the DirAC synthesizer 425, to
generate, for each
time/frequency bin, a full Ambisonics representation using the downmix signal
or pressure
signal or omnidirectional component for this time/frequency bin as the "mono
signal S" of
Fig. 6. Then, an individual frequency-time conversion in frequency-time
converter 426 for
each of the W, X, Y, Z component would then result in a sound field
description different
from what is illustrated in Fig. 6.
Subsequently, further explanations regarding a DirAC analysis and a DirAC
synthesis are
given as known in the art_ Fig. 7a illustrates a DirAC analyzer as originally
disclosed, for
example, in the reference "Directional Audio Coding" from IWFDASki of 2009.
The DirAC
analyzer comprises a bank of band filters 1310, an energy analyzer 1320, an
intensity
analyzer 1330, a temporal averaging block 1340 and a diffuseness calculator
1350 and
the direction calculator 1360. In DirAC, both analysis and synthesis arc
performed in the
frequency domain. There are several methods for dividing the sound into
frequency
bands, within distinct properties each. The most commonly used frequency
transforms
include short time Fourier transform (STFT), and Quadrature mirror filter bank
(QMF). In
addition to these, there is a full liberty to design a filter bank with
arbitrary filters that are
optimized to any specific purposes. The target of directional analysis is to
estimate at
each frequency band the direction of arrival of sound, together with an
estimate if the
sound is arriving from one or multiple directions at the same time_ In
principle, this can be
performed with a number of techniques, however, the energetic analysis of
sound field
has been found to be suitable, which is illustrated in Fig. 7a. The energetic
analysis can
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
be performed, when the pressure signal and velocity signals in one, two or
three dimen-
sions are captured from a single position. In first-order B-format signals,
the omnidirec-
tional signal is called W-signal, which has been scaled down by the square
root of two.
The sound pressure can be estimated as S = A,T2*W, expressed in the STFT
domain,
5
The X-, Y- and Z channels have the directional pattern of a dipole directed
along the Car-
tesian axis, which form together a vector U = [X, Y, Z]. The vector estimates
the sound
field velocity vector, and is also expressed in STFT domain. The energy E of
the sound
field is computed. The capturing of B-format signals can be obtained with
either coincident
10 positioning of directional microphones, or with a closely-spaced set of
omnidirectional mi-
crophones. In some applications, the microphone signals ma-y be formed in a
computa-
tional domain, i.e., simulated. The direction of sound is defined to be the
opposite direc-
tion of the intensity vector I. The direction is denoted as corresponding
angular azimuth
and elevation values in the transmitted metadata_ The diffuseness of sound
field is also
15 computed using an expectation operator of the intensity vector and the
energy. The out-
come of this equation is a real-valued number between zero and one,
characterizing if the
sound energy is arriving from a single direction (diffuseness is zero), or
from all directions
(diffuseness is one). This procedure is appropriate in the case when the full
3D or less
dimensional velocity information is available.
Fig. 7b illustrates a DirAC synthesis, once again having a bank of band
filters 1370, a vir-
tual microphone block 1400, a directidiffUSe Synthesizer block 1450, and a
certain loud-
speaker setup or a virtual intended loudspeaker setup 1460. Additionally, a
diffuseness-
gain transformer 1380, a vector based amplitude panning (VBAP) gain table
block 1390, a
microphone compensation block 1420, a loudspeaker gain averaging block 1430
and a
distributer 1440 for other channels is used. In this DirAC synthesis with
loudspeakers, the
high quality version of DirAC synthesis shown in Fig. 7b receives all B-format
signals, for
which a virtual microphone signal is computed for each loudspeaker direction
of the loud-
speaker setup 1460, The utilized directional pattern is typically a dipole.
The virtual micro-
phone signals are then modified in non-linear fashion, depending on the
metadata. The
low bitrate version of DirAC is not shown in Fig. 7b, however, in this
situation, only one
channel of audio is transmitted as illustrated in Fig. 6. The difference in
processing is that
all virtual microphone signals would be replaced by the single channel of
audio received.
The virtual microphone signals are divided into two streams the diffuse and
the non-
diffuse streams, which are processed separately.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
36
The non-diffuse sound is reproduced as point sources by using vector base
amplitude
panning (VBAP). In panning, a monophonic sound signal is applied to a subset
of loud-
speakers after multiplication with loudspeaker-specific gain factors. The gain
factors are
computed using the information of a loudspeaker setup, and specified panning
direction.
.. In the low-bit-rate version, the input signal is simply panned to the
directions implied by
the metadata. In the high-quality version, each virtual microphone signal is
multiplied with
the corresponding gain factor, which produces the same effect with panning,
however it is
less prone to any non-linear artifacts.
In many cases, the directional metadata is subject to abrupt temporal changes,
To avoid
artifacts, the gain factors for loudspeakers computed with VBAP are smoothed
by tem-
poral integration with frequency-dependent time constants equaling to about.
50 cycle pe-
riods at each band. This effectively removes the artifacts, however, the
changes in direc-
tion are not perceived to be slower than without averaging in most of the
cases. The aim
of the synthesis of the diffuse sound is to create perception of sound that
surrounds the
listener. In the low-bit-rate version, the dittuse stream is reproduced by
decorrelating the
input signal and reproducing it from every loudspeaker, In the high-quality
version, the
virtual microphone signals of diffuse stream are already incoherent in some
degree, and
they need to be decorrelated only mildly. This approach provides better
spatial quality for
surround reverberation and ambient sound than the low bit-rate version. For
the DirAC
synthesis with headphones, DirAC is formulated with a certain amount of
virtual loud-
speakers around the listener for the non-diffuse stream and a certain number
of loud-
speakers for the diffuse steam. The virtual loudspeakers are implemented as
convolution
of input signals with a measured head-related transfer functions (HRTFs).
Subsequently, a further general relation with respect to the different aspects
and, particu-
larly, with respect to further implementations of the first aspect as
discussed with respect
to Fig. la is given. Generally, the present invention refers to the
combination of different
scenes in different formats using a common format, where the common format
may, for
example, be the B-format domain, the pressure/velocity domain or the metadata
domain
as discussed, for example, in items 120, 140 of Fig. la.
When the combination is not done directly in the DirAC common format, then a
DirAC
analysis 802 is performed in one alternative before the transmission in the
encoder as
discussed before with respect to itern 180 of Fig. la.
Date Recue/Date Received 2023-11-09
WO 2019/068638
PCT/EP2018/076641
37
Then, subsequent to the DirAC analysis, the result is encoded as discussed
before with
respect to the encoder 170 and the motadata encoder 100 and the encoded result
is
transmitted via the encoded output signal generated by the output interface
200. However,
in a further alternative, the result could be directly rendered by a Fig, la
device when the
output of block 160 of Fig. 1 a and the output of block 180 of Fig. la is
forwarded to a Di-
rAC renderer. Thus, the Fig. la device would not be a specific encoder device
but would
be an analyzer and a corresponding renderer.
A further alternative is illustrated in the right branch of Fig. 8, where a
transmission from
.. the encoder to the decoder is performed and, as illustrated in block 804,
the DirAC analy-
sis and the DirAC synthesis are performed subsequent to the transmission,
i.e., at a de-
coder-side. This procedure would be the case, when the alternative of Fig. la
is used, Le.,
that the encoded output signal is a B-format signal without spatial metadata.
Subsequent
to block 808, the result could be rendered for replay or, alternatively, the
result could even
be encoded and again transmitted. Thus, it becomes clear that the inventive
procedures
as defined and described with respect to the different aspects are highly
flexible and can
be very well adapted to specific use cases.
1 Aspect of Invention: Universal DirAG- based spatial audio coctincVrendering
A Dirac-based spatial audio coder that can encode multi-channel signals,
Ambisonics:
formats and audio objects separately or simultaneously.
peneflt.ond Advantp.s..ov.e.r. State of the Art
= Universal DirAC-based spatial audio coding scheme for the most relevant
immer-
sive audio input formats
= Universal audio rendering of different input formats on different output
formats
29' Aspect of Invention: Combining two or more DirAC descriptions on a decoder
The second aspect of the invention is related to the combination arid
rendering two or
more DirAC descriptions in the spectral domain.
Benefits and Advantages over State gf the Art
= Efficient and precise DirAC stream combination
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
38
= Allows the usage of DirAC universally represent any scene and to
efficiently com-
bine different streams in the parameter domain or the spectral domain
= Efficient and intuitive scene manipulation of individual DirAC scenes or
the com-
bined scene in the spectral domain and subsequent conversion into the time do-
main of the manipulated combined scene.
3rd Aspect of Invention: Conversion of audio objects into the DirAC domain
The third aspect of the invention is related to the conversion of object
metadata and op-
tionally object waveform signals directly into the DirAC domain and in an
embodiment the
combination of several objects into an object representation.
E3enefits and Advantages over State of the Art
= Efficient and precise DirAC metadata estimation by simple metadata
transcoder of
the audio objects metadata
= Allows DirAC to code complex audio scenes involving one or more audio
objects
^ Efficient method for coding audio objects through DirAC in a single
parametric rep-
resentation of the complete audio scene.
4:. Aspect of Invention: Combination of Object rnetadata and regular DirAc
metadata
The third aspect of the invention addresses the amendment of the DirAC
metadata wth
the directions and, optimally, the distance or diffuseness of the individual
objects compos-
ing the combined audio scene represented by the DirAC parameters. This extra
infor-
mation is easily coded, since it consist mainly of a single broadband
direction per time unit
and can be refreshed less frequently than the other DirAC parameters since
objects can
be assumed to be either static or moving at a slaw pace.
Benefits and Advantages over State of the. Art
= Allows DirAC to code a complex audio scene involving one or more audio
objects
= Efficient and precise DirAC metadata estimation by simple metadata
transcoder of
the audio objects metadata.
= More efficient method for coding audio objects through DirAC by combining
effi-
ciently their metadata in DirAC domain
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
39
e, Efficient method for coding audio objects and through DirAC by combining
effi-
ciently their audio representations in a single parametric representation of
the au-
dio scene.
5.Aspect oftovention: Manipulation of Objects MC scenes and FOA/HOA C in DirAC
.syrithesis
The fourth aspect is related to the decoder side and exploits the known
positions of audio
objects. The positions can be given by the user though an interactive
interface and can
also be included as extra side-information within the bitstrearn.
The aim is to be able to manipulate an output audio scene comprising a number
of objects
by individually changing the objects' attributes such as levels, equalization
and/or spatial
positions. It can also be envisioned to filter completely the object or
restitute individual
objects from the combined stream.
The manipulation of the output audio scene can be achieved by jointly
processing the spa-
tial parameters of the DirAC metadata, the objects' metadata, interactive user
input if pre-
sent and the audio signals carried in the transport channels.
Benefits and Advantages over State of the Art
* Allows DirAC to output at the decoder side audio objects as presented at
the input
of the encoder.
gi Allows DirAC reproduction to manipulate individual audio object by applying
gains,
rotation , or,..
= Capability requires minimal additional computational effort since it only
requires a
position-dependent weighting operation prior to the rendering & synthesis fir-
terbank at the end of the DirAC synthesis (additional object outputs will just
require
one additional synthesis filterbank per object output).
Refefences pat are .alipcorporated it their entirety by reference:
[1] V. Pulkki, M-V Laitinen, J Vilkarno, J Ahonen, T Lokki and T Pihlajarnaki,
"Directional
audio coding - perception-based reproduction of spatial sound", International
Workshop
on the Principles and Application on Spatial Hearing, Nov. 2009, Zao; Miyagi,
Japan.
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
[2] Ville Pulkki. "Virtual source positioning using vector base amplitude
panning". J. Audio
Eng. Soc., 415(6):456[466, June 1997,
[3] M. V. Laitinen and V. Pulkki, "Converting 5.1 audio recordings to B-format
for direc-
5 tional audio coding reproduction," 2011 IEEE International Conference on
Acoustics,
Speech and Signal Processing (ICASSP), Prague, 2011, pp. 61-64.
[4] G. Del GaIdo, F. Kuech, M. Kallinger and R. Schultz-Amling, "Efficient
merging of mul-
tiple audio streams for spatial sound reproduction in Directional Audio
Coding," 2009 IEEE
10 International Conference on Acoustics, Speech and Signal Processing,
Taipei, 2009, pp.
265-268.
[5] Jurgen HERRE, CORNELIA FALCH, DIRK rviAHNE, GIOVANNI DEL GALDO,
MARKUS KALLINGER, AND OLIVER THIERGART, "Interactive Teleconferencing Corn-
15 Pining Spatial Audio Object Coding and DirAC Technology", J. Audio Eng.
Soc., Vol. 59,
No. 12, 2011 December.
[6] R. Schultz-Amting, F. Kueoh, M. Katlinger, G. Del Galdo, J. Ahonen, V.
Pulkki, "Planar
Microphone Array Processing for the Analysis and Reproduction of Spatial Audio
using
20 Directional Audio Coding,'' Audio Engineering Society Convention 124,
Amsterdam, The
Netherlands, 2008.
[7] Daniel P. Jarrett and Oliver Thiergart and Emanuel A. P. Habets and
Patrick A. Naylor,
"Coherence-Based Diffuseness Estimation in the Spherical Harmonic Domain",
IEEE 27th
25 Convention of Electrical and Electronics Engineers in Israel (IEEE!),
2012.
[8] US Patent 9,015,054.
The present invention provides, in further embodiments, and particularly with
respect to
30 the first aspect and also with respect to the other aspects different
alternatives. -These
alternatives are the following:
Firstly, combining different formats in the B format domain and either doing
the DirAC
analysis in the encoder or transmitting the combined channels to a decoder and
doing the
35 DirAC analysis and synthesis there.
Secondly, combining different formats in the pressure/velocity domain and
doing the Di-
rAC analysis in the encoder. Alternatively, the pressure/velocity data are
transmitted to the
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
41
decoder and the DirAC analysis is done in the decoder and the synthesis is
also done in
the decoder.
Thirdly, combining different formats in the metadata domain and transmitting a
single Di-
rA.0 stream or transmitting several DirAC streams to a decoder before
combining them
and doing the combination in the decoder.
Furthermore, embodiments or aspects of the present invention are related to
the following
aspects:
Firstly, combining of different audio formats in accordance with the above
three alterna-
tives.
Secondly, a reception, combination and rendering of two DirAC descriptions
already in the
same format is performed.
Thirdly, a specific object to DirAC converter with a "direct conversion" of
object data to
DirAC data is implemented.
Fourthly, object metadata in addition to normal DirAC metadata and a
combination of both
metadata; both data are existing in the bitstream side-by-side, but audio
objects are also
described by DirAC metadata-style.
Fifthly, objects and the DirAC stream are separately transmitted to a decoder
and objects
are selectively manipulated within the decoder before converting the output
audio (loud-
speaker) signals into the time-durnain.
It is to be mentioned here that all alternatives or aspects as discussed
before and all as-
pects as defined by independent claims in the following claims can be used
individually,
i.e., without any other alternative or object than the contemplated
alternative, object or
independent claim. However, in other embodiments, two or more of the
alternatives or the
aspects or the independent claims can be combined with each other and, in
other embod-
iments, all aspects, or alternatives and all independent claims can be
combined to each
other.
An inventively encoded audio signal can be stored on a digital storage medium
or a non-
transitory storage medium or can he transmitted on a transmission medium such
as a
wireless transmission medium or a wired transmission medium such as the
Internet,
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
42
Although some aspects have been described in the context of an apparatus, it
is clear that
these aspects also represent a description of the corresponding method, where
a block or
device corresponds to a method step or a feature of a method step.
Analogously, aspects
described in the context of a method step also represent a description of a
corresponding
block or item or feature of a corresponding apparatus.
Depending on certain implementation requirements, embodiments of the invention
can be
implemented in hardware or in software, The implementation can be performed
using a
digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM,
an
EPROM, an EEPROM or a FLASH memory, having electronically readable control
signals
stored thereon, which cooperate (or are capable of cooperating) with a
programmable
computer system such that the respective method is performed.
.. Some embodiments according to the invention comprise a data carrier having
electroni-
cally readable control signals, which are capable of cooperating with a
programmable
computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a
computer pro-
gram product with a program code, the program code being operative for
performing one
of the methods when the computer program product runs on a computer. The
program
Code may for example be stored on a machine readable carrier,
Other embodiments comprise the computer program for performing one of the
methods
.. described herein, stored on a machine readable carrier or a non-transitory
storage medi-
um.
In other words, an embodiment of the inventive method is, therefore, a
computer program
having a program code for performing one of the methods described herein, when
the
computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier
(or a digital
storage medium, or a computer-readable medium) comprising, recorded thereon,
the
computer program for performing one of the methods described herein_
A further embodiment of the inventive method is, therefore, a data stream or a
sequence
of signals representing the computer program for performing one of the methods
de-
Date Recue/Date Received 2023-11-09
WO 2019/068638 PCT/EP2018/076641
43
scribed herein. The data stream or the sequence of signals may for example be
config-
ured to be transferred via a data communication connection, for example via
the Internet.
A further embodiment comprises a processing means, for example a computer, or
a pro-
gramrnable logic device, configured to or adapted to perform one of the
methods de-
scribed herein.
A further embodiment comprises a computer having installed thereon the
computer pro-
gram for performing one of the methods described herein,
In some embodiments, a programmable logic device (for example a field
programmable
gate array) may be used to perform some or all of the functionalities of the
methods de-
scribed herein. In some embodiments, a field programmable gate array may
cooperate
with a microprocessor in order to perform one of the methods described herein.
Generally,
the methods are preferably performed by any hardware apparatus.
The above described embodiments are merely illustrative for the principles of
the present
invention. It is understood that modifications and variations of the
arrangements and the
details described herein will be apparent to others skilled in the art. It is
the intent, there-
fore, to be limited only by the scope of the impending patent claims and not
by the specific
details presented by way of description and explanation of the embodiments
herein.
Date Recue/Date Received 2023-11-09
