Note: Descriptions are shown in the official language in which they were submitted.
~WO 96/36039 ~ ~ p p ,~ PCT/US96/06668
to Efficient Synthesis of Musical. Tones having Nonlinear
Excitations
Field of the Invention
This invention relates to methods for digital synthesis of
tones, and particularly to computationally efficient digital
waveguide techniques for the synthesis of tones that are
simulations of musical tones produced by musical instruments,
such as pianos, whose waveguide elements are .nonlinearly
excited.
Background of the Invention
A common method for the digital synthesis of musical tones is
waveform or spectrum matching, which includes techniques such
as sampling, wavetable, wave-shaping, FM synthesis, and
additive/subtractive synthesis. This approach generates tones
by processing samples taken from a.fixed wavetable containing
the waveforms produced by a particular instrument. The pitch
of the synthesized note is determined from the frequency of
3o the sample in the wavetable. Although these methods reproduce
certain tones well, expensive computational resources are
often required to sufficiently process the samples to produce
a versatile selection of rich and natural sounds. Moreover,
the complex processing is controlled by a large number of
parameters that are not intuitively related to the
characteristics of particular musical instruments or their
tones.
1
I
WO 96136039 PCTlU896I06668
~-220~;.
An alternative method for the synthesis o~ musical tones is
digital waveguide filtering. Strings, woodwind bores, horns,
and the human vocal tract are examples of acoustic waveguides.
Rather than processing tone samples from a fixed wavetable,
waveguide filtering simulates the physical vibration of a
musical instrument's acoustic waveguide with a "filtered delay
~- . - ~ - ~ -loop" consisting of a delay line and one or more filters
arranged in a loop. Consequently, the pitch of the
1o synthesized note is determined by the total loop delay, which
corresponds to the length of-the instrument's waveguide, e.g.,
the length of a string, or distance to the first open tone
whole in a woodwind instrument. The delay line loop is
excited with a waveform corresponding, for example, to the
plucking of a string. The waveguide filtering technique,
therefore, can be distinguished from the waveform or spectrum
matching techniques by the fact that the waveguide filter is
not normally excited by samples that are substantially related
to the pitch of the resulting note. The stored waveforms used
in waveguide synthesis, consequently, typically require less
memory. In addition, because this method models the physical
dynamics of an~ instrument's waveguide, its operational
parameters are easily related to the characteristics of
particular musical instruments.
Perhaps the most important advantage of this approach is that
simple computational waveguide filtering models can produce
some surprisingly rich sounds without requiring expensive
computational resources. For example, K. Karplus and A.
3o Strong describe a simple implementation of a plucked string in
U.S. Patent No. 4,649,783 issued March 17, 1987 and in
"Digital Synthesis of Plucked-String and Drum Timbres,"
Computer Music J., vol. 7, no. 2, pp. 43-55, 1983. A simple
block diagram of this system is shown in Fig. 1. A noise
burst from a noise generator 20 is used to initialize the
signal in a delay line 22, thereby simulating the pluck of the
string. A simple digital filter 24 in the delay line loop
2
WO 96/36039 , - - ~ ~ ~ PCT/US96/(J6668
i
causes high frequency components of the initial signal to
decay quickly, leaving lower frequency harmonics which are
determined by the length of the delay line. The use of the
random noise burst gives each note a unique timbre and adds
s realistic variation to the tones produced. Although the
invention of Karplus and Strong produces surprisingly rich
sounds with inexpensive computational resources, its
~- ~--simplicity neglects many subtle f~atu.~'es of, musicah-. i~ones, and _.
. -...
introduces several digital artifacts. Because Karplus and
1o Strong did not recognize their algorithm as a physical
modeling synthesis technique, it did not include features
related to physical strings that could be added with very
little cost.
15 Various limitations to the above approach of Karplus and
Strong were addressed by J.O. Smith in "Techniques for Digital
Filter Design and System Identification with Application to
the Violin", Ph.D. Dissertation, Elec. Eng. Dept., Stanford
University, June 1983, and D. Jaffe and J.O. Smith in
20 "Extensions of the Karplus-Strong Plucked-String Algorithm,"
Computer Music J., vol. 7, no. 2, pp. 56-69, 1983. Jaffe and
Smith used additional computational resources to add more
usefulness, realism, and flexibility to the basic approach of
Karplus-Strong. For example, the decay rates of high and low
2s harmonics were altered to produce more authentic tones, a
dynamics filter was added to give control over the strength of
the pluck, and effects due to the stiffness of strings were
implemented with an allpass filter.
3o In addition to the computational expense required to implement
subtleties of an instrument's waveguide dynamics, complex
filtering is also required to realistically model the
resonances in the instrument's body. Since the specific
characteristics of an instrument's body determine to a large
35 extent its particular sound, a realistic simulation of the
body resonator is very desirable in music synthesis systems.
Due to the complexity of the body resonator, however, modeling
3
CA 02200447 2005-10-31
these resonances using known techniques is very expensive.
Moreover, the complete modeling of resonances may include the
coupling between the waveguide and the body resonator, the body
resonator itself, the air absorption, and the room response.
A novel synthesis technique for dramatically reducing the
computational resources required, to model resonators is
described by J.O. Smith in U.S. Patent No. 5,500,486 and U.S.
Patent No. 5,587,548. Fig. 2 shows a sequence of three block
diagrams indicating how the conventional architecture for a
synthesis system may be restructured to yield a much simpler
system. The conventional 20 architecture, show at the top of
the figure, includes an excitation 26 which drives a string loop
28. The signal from the string loop then enters a resonator 30.
The first step in the simplification of this architecture is
made possible by the fact that the properties of the resonator
and the string are time-invariant and linear. Consequently, the
order in which they are performed can be reversed. The resulting
commuted system, shown in the middle of the figure, includes an
excitation 32 which drives a resonator 34. The signal from the
resonator then enters a string loop 36.
The next step in the simplification is to eliminate the
resonator by absorbing it into the excitation. Many common
excitations, such as a plucked string, are qualitatively
impulses. Consequently, the output of a resonator excited by an
impulse is simply the impulse response of the resonator. Since
the resonator and excitation, are both time-invariant, the
dynamics of the resonator can be eliminated entirely and
4
PCTYUS96l06668
~WO 96136039
the excitation-resonator pair can be replaced by a single
aggregate excitation 38 which consists of a pre-convolution of
the excitation with the impulse response of the resonator.
This signal excites a string 40 with a signal that implicitly
includes the effects of the resonator. Consequently, the
necessity for expensive computational resources to implement
the effects of the resonator is entirely eliminated.
In spite of the significant advantages provided by the
1o technique of commuting the resonator and convolving its
impulse response with the excitation, this technique 'is
limited to plucked and linearly-struck waveguides. In
particular, it does not apply to a struck piano string since
the hammer-string interaction in a piano requires a nonlinear
response for accurate modeling and realistic attacks.
Consequently, there is no obvious way the resonator can be
commuted and the synthesizer complexity reduced as before.
The same difficulties arise in other cases where the
excitation is nonlinear, such as with vigorously bowed
2o strings. Realistic synthesis of tones from these instruments,
therefore, presently require expensive computational resources
in order to implement the effects of the resonator.
Objects and l~dvantaQes of the Invent3.on
Accordingly, it is a primary object of the present invention
to provide a computationally efficient method for the
synthesis of tones produced by musical instruments whose
waveguide elements are nonlinearly excited. It is a further
object of the invention to provide a method for reducing the
3o computational power required to implement a resonator in a
waveguide filtering synthesis system where the excitation of
the' waveguide is nonlinear. It is another object of the
present invention to provide a computationally efficient piano
synthesizer.
By reducing the computational resources required to implement
the effects of a resonator in nonlinearly excited instruments,
5
~O 96136039 _ 2 2 ~ ~ PCTIIJS96/06668
the cost of producing synthesizers for such instruments is
reduced. Moreover, since computational 'resources are not
consumed by simulating the resonator, they can be used to
implement additional features that will further improve the
quality of synthesis.
summary of the Invention
' ' ' "' ' These ' obi ects and advantages are . atta.i.ned. by a surprising.
. . . . ..
synthesizer design that permits the commutation of the
to resonator through an effectively nonlinear filter. The device
includes an excitation means for producing an excitation
pulse, an excitation filtering means for producing a filtered
excitation pulse, and a waveguide simulating means for
producing the tone. The properties of the excitation means
are determined by the characteristics of the resonator. In
one embodiment the excitation means includes an excitation
table and a pointer for reading values out of the table to
produce the excitation pulse. In another embodiment the
excitation means generates the excitation pulse by filtering a
2o repeated segment of the resonator impulse response. In
another embodiment the excitation pulse is completely
synthesized by~filtering white noise.
The response of the excitation filtering means is dependent
upon the amplitude ~of the tone and is therefore effectively
nonlinear. In a preferred embodiment, the response becomes
shorter as the amplitude of the tone becomes larger. A
plurality of such filters may be combined with delay lines to
model the reflection excitation pulses. The waveguide
simulating means comprises a delay line means and a waveguide
filtering means whose response is dependent on the
characteristics of the vibrating element. Additional
embodiments of the synthesizer include additional filters for
simulating high-Q portions of the resonator, and for producing
effects such as reverberation, equalization, echo, and
flanging.
6
' ~'O 96!36039 ' - ~ PCT/U896/06668
Descriptioa of the Figures
Fig..i is a block diagram of a plucked-string synthesizer
according to the teaching of Karplus and Strong.
Fig. 2 is an illustration of the technique of J.O. Smith for
s commuting a resonator through string filters and
convolving it with an excitation.
Fig. 3 is an illustration of the modeling of a collision pulse
.. . .. . , .... by .a filtered impulse, according. to the. invention. . .. ..
Fig. 4 shows the graph of a collision pulse including an
to initial pulse and two reflected pulses, according to the
invention.
Fig. 5 is a block diagram of a circuit for creating the
collision pulse shown in Fig. 4, in accordance with the
teachings of the invention.
ss Fig. 6 is a block diagram of a synthesizer of the invention
before the resonator is commuted.
Fig. 7 is a block diagram of a synthesizer of the invention
after the resonator is commuted through the filters and
convolved with the excitation.
2o Fig. 8 is a block diagram of a synthesizer of the invention
reducing the number of filters used to model the
collision pulse.
Fig. 9 is a block diagram of a synthesizer of the invention
reducing the complexity of the filters used to model the
25 reflected collision pulses.
Fig. 10 is a block diagram of a synthesizer of the invention
using feedback to model the reflected collision pulses.
Fig. 11 is a block diagram of a synthesizer of the invention
using an equalizer bank to model the reflected collision
3o pulses.
Fig. 12 is a block diagram illustrating the decomposition of
the excitation into dry and wet parts, according to the
invention.
Fig. 13 is a block diagram showing how the wet portion of the
35 soundboard impulse response can be synthesized, according
to the invention.
7
' WO '96136039 , ~ pcTiUS96iossss
Fig. 14 is a block diagram of an entire synthesizer of the
invention including additional filters for supplementary
ef f ects .
Fig. 1~ is a block diagram showing three string loops coupled
together to model the three strings of a single piano
note.
.. .. . . . .. ... ' p
D~tailed .Descri tioai... . .. . .... . . . . : . . . _ ... _ _ . ...... . ..
. . . . ... .. ...
In a preferred embodiment, the method for efficiently
to synthesizing tones from a nonlinearly excited waveguide is
applied to the case of the piano. The excitation of a piano
string by a piano hammer is nonlinear because the felt tip of
the piano hammer acts like a spring whose spring constant
rapidly increases as the felt is compressed against the
s5 string. In order for a model of the hammer-string interaction
to be authentic, this nonlinear effect can not be ignored. At
the same time, in order to take advantage of the computational
savings of commutation, a linear and time-invariant model of
the hammer-string interaction must be found.
Because the wave impedance of the string is resistive for an
infinitely long string, the hammer will not bounce away from
the string until reflected pulses push it away or unless it
falls away due to gravity. Consequently, the initial
collision pulse can be well modeled by a filtered impulse, as
shown in Fig. 3, where the impulse response of the filter
corresponds to the compression force signal of a single
. collision pulse. A fully physical nonlinear computational
model of the hammer-string interaction can be used to
3o determine the form of the pulse. Then a linear filter is
designed whose response closely approximates this calculated
pulse. The form of the force signal is qualitatively similar
to the difference of two exponential decays, i.e.,
h(t) - A [ exp(-t/~l) - exp(-t/~2) ~ , where ~l > ~2. A filter
of the form
H ( z ) - A ( Pi - P2 ) / [ ( 1 - p1 z-1 ) ( 1 - p2 z-1 ) 7
s
'4'O 96!36039 , . ~ PCT/US96/06668
will produce such an impulse response. If desired, the two
additional poles can be added to the filter to give a smoother
initial rise and a better shock spectrum fit to the calculated
compression force signal.
When a piano key is pressed hard and fast, the hammer strikes
'- " w -wthe string wi~thw a highw velocity: Because of the nonlinear w ~ w
response of the felt tip, the force pulse is higher and
1o narrower. Consequently, the impulse response of the filter
needs to be adjusted in accordance with the hammer velocity~so
that higher strike velocities will correspond to filters with
broader bands, i.e., shorter impulse responses. For example,
a simple filter with this property can be designed with a
is transfer function of the form H(z) - C / ( 1 - p z-1 )4, where
p is a monotonic function of the hammer velocity, and C is a
constant.
By using a linear filter whose response depends upon the
2o hammer velocity, an effectively nonlinear filter is created.
Such a filter, however, is no longer absolutely time-
invariant. Nevertheless, since the hammer velocity for each
note is a constant, the filter is time-invariant with respect
to the synthesis of each note. Thus the resonator can be
2s commuted through the filter and convolved with the excitation.
Because the hammer does not typically bounce off the string
immediately afr_Pr t-ha ; n; t-; ~ ~ ...~, , , ._, _ ~,-
--j ----~- ~~~~- .~ii.~ar.a.u.w.v111.~71011 purse, Lne additional
interactions between the. hammer and reflected pulses usually
so must be taken into account. In many cases, the hammer is in
contact with the string for a time interval that is long
enough for it to interact with several pulses ref lected off
the near end of the string (the agraffe) . For most piano
strings, however, the reflected pulses from the far end (the
35 bridge) do not return before the hammer leaves the string.
Since the reflected pulses are merely slightly filtered
versions of the initial collision pulse, they can also be
9
WO 96J36039 ' , PCT/US96J06668
modeled as filtered impulses. Fig. 4 shows the graph of the
interaction including an initial collision pulse and two
reflected pulses. Fig. 5 is a block diagram showing one way
this hammer-string interaction may be implemented. Three
impulses, staggered in time, enter three filters. The signals
from the filters are then superimposed and fed into the
string. The number of impulses will generally be fixed for a
.. _ given. string., It is. also important to..notethat, , since.. we., are
assuming that the string is initially at rest, all interaction
to impulses are predetermined by the initial collision velocity
and the string length.
The synthesizer, before commuting the sound board and
enclosure resonator, is shown in Fig. 6. A trigger signal
which contains the hammer velocity information enters the
impulse generator and triggers the creation of an impulse.
. The tapped delay line creates three copies of the impulse, two
of which are delayed by differing lengths of time. The three
impulses then enter three lowpass filters, LPFl, LPF2, and
LPF3, which produce three pulses. Note that the trigger
signal is also fed into the three filters in order to adjust
their response in accordance with the hammer velocity, thereby
producing an effective nonlinear response. The three pulses
are superimposed by an adder, and the output of the adder is
used to excite a string loop. The output of the string loop
then enters the complex sound board and enclosure resonator,
which then produces the final output.
Fig. 7 shows the synthesizer after the sound board and
3o enclosure resonator has been commuted and convolved with the
impulse generator. GOhen triggered, the impulse response of
the sound board and enclosure passes through the same tapped
delay line and interaction pulse filters as in Fig. 6. The
resulting signals are added and used to excite the string
loop. Since the trigger alters the response of the collision
pulse filters, the excitation is effectively nonlinear even
though the filters are linear with respect to each note
1o
0 96/36039 ~ ~ ~ PCTlIJS96/06<68
played. Moreover, because the effects of the resonator are
built-in to the excitation, the string excitation already
includes effects due to the resonator. With the resonator
commuted and convolved with the excitation generator, the
~5 expensive processing normally required to implement the
resonator is entirely eliminated. If desired, an optional
output scaling circuit can be included in order to scale the
,, string output in.accordance with. the. hammer velocity..- .. . _ .. _ . ..
,.._.
1o Fig. 8 shows a slightly different implementation that trades
some accuracy in the modeling of the collision pulse for
computational efficiency. Because the collision pulse filters
are nearly identical, the adder can be commuted and the three
filters can be consolidated into one. Rather than
15 implementing the impulse delays with tapped delay lines, this
embodiment uses three separate pointers to read the values
from the excitation table.. Otherwise, the operation of this
synthesizer is identical to that described above.
2o Fig. 9 shows an alternate embodiment that improves
computational efficiency without sacrificing the accuracy of
the collision pulse modeling. Since each reflected pulse is
smoother than the one preceding it, as long as the hammer
remains in contact with the string, the reflected pulse
25 filters can be simplified by using the result of one filter as
the input for the next. Since each filter in this embodiment
need only provide mild smoothing and attenuation, it is
computationally cheap to implement. A further simplification
can be made by convolving the impulse response of the first
3o filter at a particular hammer velocity with the excitation.
The first filter can then be replaced with a simpler filter
that merely modifies the excitation to account for the
difference between the preconvolved velocity and the desired
velocity.
In the embodiment shown in Fig . ~.0 , rather than us ing the
above "feed-forward " approach to modeling the multiple force
m
', WO 96!36039 ~ ~ PCT/ITS96/06668
pulses of the hammer-string interaction, a "feed-backward"
approach is implemented. In this implementation the initial
pulse is fed back through a delay and a recursion filter and
added to the signal at the input of the collision pulse
filter. A simplification of this implementation combines the
recursion filter with the collision pulse filter and
prefilters the signal entering the feedback loop with an
. . . inverse recursion, filter. . . ... .. - . . -
to In the embodiment shown in Fig. 11, the multiple collision
pulse filtering is performed by an equalizer bank. Using a
computational model of the multiple collision force pulse, the
ratio spectrum of the multiple pulse spectrum to the single
pulse. spectrum is modeled by an EQ bank of 2-pole/2-zero
filters. Combining this bank with a single collision pulse
filter then yields a multiple collision pulse filter.
In a versatile synthesizer, the resonator includes the
response of the piano~with the pedal down and the response
2o with the pedal up. When the pedal is down, the sound of the
strings couples into the whole set of strings attached to the
sound board, creating a rich reverberant color change to the
piano sound. Whereas the pedal up response lasts less than
half a second, the rich pedal down impulse response can last
from 10 to 20 seconds and includes the many modes from
hundreds of strings. Because such a long impulse response
requires so much memory, it is desirable to find ways to
reduce the length of the pedal down impulse response.
3o One way to reduce the length of the pedal down impulse
response is to decompose the response into two parts, as shown
in Fig. 12. The dry part is the impulse response of the
soundboard and enclosure with the pedal up. The wet part is
the impulse response of just the open strings resonating. The
sum of the two is approximately equivalent to the impulse
response of the piano with the pedal down. Although this
decomposition in itself does not reduce the required memory,
12
1 WO 96!36039 ' ~ PCTlUS96/06668
,.
once the dry and wet parts have. been separated, the wet
impulse response can now be shortened by the implementation
shown in Fig. 13. It is possible to normalize its amplitude,
c~.i'p' out a representative . section pf .its quasi-steady state,
and use a loop to play this section repeatedly. A slow
exponential decay amplitude envelope is applied to model the
decay rate of the original impulse response, and a slowly
time-varying lowpass filter is applied to adjust the decay
rates of high and low frequency components. In short, the wet
part can be synthesized using any of the well known methods of
wavetable synthesis or sampling synthesis.
The following technique provides another method for reducing
even further the memory required to store the sbundboard
impulse response_ In a linear approximation, the soundboard
impulse response is a superposition of many exponentially
decaying sinusoids. Since an ideal piano soundboard does not
preferentially couple to any specific notes, its spectral
response is very flat (although high frequency modes decay a
little faster than low frequency modes). The impulse response
of such a system can be modeled as exponentially decaying
white noise with a time-varying lowpass filter to attenuate
high-frequency modes faster than low-frequency modes. The
bandwidth of this filter shrinks as time increases.
This above model can be refined by introducing a simple
lowpass filter to more accurately shape the noise spectrum
before it is modified dynamically during the playing of a
note. In addition, several bandpass filters can be introduced
to provide more detailed control over the frequency dependence
of the decay rates of the soundboard impulse response. An
advantage of this technique is that it provides complete
control over the quality of the soundboard. Moreover, using
this technique the impulse response of the soundboard can be
synthesized without expensive computational resources or large
amounts of memory. In general, this technique can be used to
synthesize any number of reverberant systems that have
13
. W096J36039 1 - PCTlUS96106668
substantially smooth responses over the frequency spectrum.
The piano soundboard and the soundboard with open strings are
both systems. of this kind. High quality artificial
reverberation devices ideally have this property as well.
In general, when the resonator becomes very complex and has a
very long impulse response, it is possible to reduce the
.. ... ..length o.f the . stored . excitations, required. by . factoring. .the
. . .. ..
resonator into two parts and only commuting one of them. Thus
l0 computational and memory resources can be interchanged to
suit the particular application. For example, a.t is often
profitable to implement the longest ringing resonances of the
soundboard and piano enclosure using actual digital filters.
This shortens the length of the excitation and saves memory.
s5 Note that the resonator may include the resonances of the room
as well as those of the instrument.
In addition to the high-Q resonator filters, other filters may
also be included in the synthesizer. For example, the
2o synthesizer may include reverberation filters, equalization
filters to implement piano color variations, and comb filters
for flanging, chorus, and simulated hammer-strike echoes on
the string. Since these filters are linear and time-
invariant, they may be ordered arbitrarily. A general
25 synthesis system of this type is shown in Fig. 14. Multiple
outputs are provided for enhanced multi-channel sound.
For purposes of simplicity, the embodiments above are
described for only a single string. Nevertheless, the
30 techniques and methods are.generally applicable to any string
and can be used to model multiple strings simultaneously.
Indeed, the synthesis of realistic piano tones requires the
modeling of up to three strings per note and up to three modes
of vibration per string corresponding to vertical and
35 horizontal planes of transverse vibration, together with the
longitudinal mode of vibration in the string. Coupling
between these vibrational modes must also be included~in the
14
Y.,WO 96/36039 ~ - ~ PCT/LJS96/06668
model. The complete modeling of a piano note, therefore,
would require a model with as many as nine filtered delay
loops coupled together.
Fig. 15 shows an implementation of the transverse vibrations
of three coupled strings corresponding to a single note. The
coupling filter models the loss at the yielding bridge
. . .. . . ....termination,. and ..contr:ols . the . coupling , between
the....thre.e.. _ . . ....
strings. Each string loop contains two delay elements for
to modeling the round-trip delay from the hammer strike point to
the agraffe and the round-trip delay from the hammer strike
point to the bridge. For a typical piano string the ratio of
these delays is about 1:8. The three string loops are excited
by three excitation signals, each of which is produced as
described earlier. To model the spectral combing effect of
the relative strike position of the hammer on the string,
these excitation signals enter their respective string loops
at two different points, in positive and negative form. To
model una corda pedal effects, one or more of these excitation
2o signals are set to zero at key strike time, causing the
coupled string system to quickly progress into its second
stage~decay rate.
Sustain signals for each.string loop in Fig. 15 are set to 1.0
during the sustain portion of the note and are camped to an
attenuation factor, e.g., 0.95, when the key is released. The
delay lengths in this coupled string model are fine-tuned with
tuning filters such that the effective pitch of the three
strings vary slightly from being exactly equal. This slight
dissonance between the strings results in the two-stage decay
!I that is a very important quality of piano notes. To model the
effect of the natural inharmonicity of the piano string
partials, the phase response of the loops are modified by
stiffness filters, typically having allpass filter structures.
To permit the playing of several notes at once, a collection
of strings as just described are implemented in parallel. The
~1'O 96!36039 ' , ~ PCT/iTS96/06668
sound of the complete piano is then obtained from the addition
of the sounds synthesized for each note. In a complete piano
synthesizer such as this, filtering of the tones after the
strings .
The above embodiments are only specific implementations of the
invention. Anyone skilled in the art of electronic music
. ... .. . . synthesis . can ..easi.ly design. .many obvious . variations -
..on and _.-.. .-..
implementations of the above synthesis systems based on the
1o teachings of the invention. Accordingly, the scope of the
invention should be determined by the following claims and
their legal equivalents.
16
