Audio source separation based on independent component analysis.

Abstract

This paper introduces the blind source separation (BSS) of convolutive mixtures of acoustic signals, especially speech. A statistical and computational technique, called independent component analysis (ICA), is examined. By achieving nonlinear decorrelation, nonstationary decorrelation, or time-delayed decorrelation, we can find source signals only from observed mixed signals. Particular attention is paid to the physical interpretation of BSS from the acoustical signal processing point of view. Frequency-domain BSS is shown to be equivalent to two sets of frequency domain adaptive microphone arrays, i.e., adaptive beamformers (ABFs). Although BSS can reduce reverberant sounds to some extent in the same way as ABF, it mainly removes the sound from the jammer direction. This is why BSS has difficulties with long reverberation in the real world. If sources are not "independent," the dependence results in bias noise when obtaining the correct unmixing filter coefficients. Therefore, the performance of BSS is limited by that of ABF. Although BSS is upper bounded by ABF, BSS has a strong advantage over ABF. BSS can be regarded as an intelligent version of ABF in the sense that it can adapt without any information on the array manifold or the target direction, and sources can be simultaneously active in BSS.

Full text

Page 1

AUDIO SOURCE SEPARATION
BASED ON INDEPENDENT COMPONENT ANALYSIS
Shoji Makino†
Shoko Araki†
Ryo Mukai†
Hiroshi Sawada†
†NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
{maki, shoko, ryo, sawada}@cslab.kecl.ntt.co.jp
ABSTRACT
This paper introduces the blind source separation (BSS) of
convolutive mixtures of acoustic signals, especially speech.
A statistical and computational technique, called indepen-
dent component analysis (ICA), is examined. By achiev-
ing nonlinear decorrelation, nonstationary decorrelation, or
time-delayed decorrelation, we can find source signals only
from observed mixed signals. Particular attention is paid to
the physical interpretation of BSS from the acoustical signal
processing point of view. Frequency-domain BSS is shown
to be equivalent to two sets of frequency domain adaptive
microphone arrays, i.e., adaptive beamformers (ABFs). Al-
though BSS can reduce reverberant sounds to some extent
in the same way as ABF, it mainly removes the sounds
from the jammer direction. This is why BSS has difficulties
with long reverberation in the real world. If sources are not
“independent,” the dependence results in bias noise when
obtaining the correct unmixing filter coefficients. There-
fore, the performance of BSS is limited by that of ABF.
Although BSS is upper bounded by ABF, BSS has a strong
advantage over ABF. BSS can be regarded as an intelligent
version of ABF in the sense that it can adapt without any
information on the array manifold or the target direction,
and sources can be simultaneously active in BSS.
1. INTRODUCTION
Speech recognition is a fundamental technology for com-
munication with computers, but with existing computers,
the recognition rate drops rapidly when more than one per-
son is speaking or when there is background noise.
the other hand, humans can engage in comprehensible con-
versations at a noisy cocktail party. This is the well known
cocktail-party effect, where the individual speech waveforms
are found from the mixtures. The aim of audio source sepa-
ration is to provide computers with this cocktail party abil-
ity, thus making it possible for computers to understand
what a person is saying at a noisy cocktail party.
Blind source separation (BSS) is an emerging technique,
which enables the extraction of target speech from ob-
served mixed speeches without the need for source position-
ing, spectral construction, or a mixing system. To achieve
this, attention has focused on a method based on indepen-
dent component analysis (ICA). ICA extracts independent
sounds from among mixed sounds. This paper considers
ICA in a wide sense, namely nonlinear decorrelation to-
gether with nonstationary decorrelation and time-delayed
decorrelation. These three methods are discussed in a uni-
fied manner [1, 2]. There are a number of applications for
On
Figure 1: BSS system configuration.
the BSS of mixed speech signals in the real world [3], but
the separation performance is still not good enough [4, 5].
Since ICA is a purely statistical process, the separation
mechanism has not been clearly understood in the sense of
acoustic signal processing, and it has been difficult to know
which components were separated, and to what degree. Re-
cently, the ICA method has been investigated in detail, and
its mechanisms have been gradually uncovered by using the-
oretical analysis from the perspective of acoustic signal pro-
cessing [6] as well as experimental analysis based on impulse
response [7]. The mechanism of BSS based on ICA has been
shown to be equivalent to that of an adaptive microphone
array system, i.e., N sets of adaptive beamformers (ABFs)
with an adaptive null directivity aimed in the direction of
unnecessary sounds.
From the equivalence between BSS and ABF, it becomes
clear that the physical behavior of BSS reduces the jammer
signal by making a spatial null towards the jammer. BSS
can further be regarded as an intelligent version of ABF in
the sense that it can adapt without any information on the
source positions or period of source existence/absence.
2. WHAT IS BSS?
Blind source separation (BSS) is an approach for estimat-
ing source signals si(n) using only the information of mixed
signals xj(n) observed at each input channel. Typical ex-
amples of such source signals include mixtures of simul-
taneous speech signals that have been picked up by several
microphones, brain waves recorded by multiple sensors, and
interfering radio signals arriving at a mobile station.
V - 668 0-7803-8251-X/04/$17.00 ©2004 IEEEISCAS 2004
«¬

Page 2

Figure 2: Task of blind source separation of speech signals.
2.1. Mixed Signal Model for Speech Signals in a
Room
In the case of audio source separation, several sensor micro-
phones are placed in different positions so that each records
a mixture of the original source signals at a slightly different
time and level. In the real world where the source signals
are speech and the mixing system is a room, the signals that
are picked up by the microphones are affected by reverber-
ation. Therefore, the N signals recorded by M microphones
are modeled as
xj(n) =
N
?
i=1
P
?
p=1
hji(p)si(n − p + 1) (j = 1,···,M), (1)
where si is the source signal from a source i, xj is the signal
received by a microphone j, and hji is the P-taps impulse
response from source i to microphone j.
This paper focuses on speech signals as sources that are
nongaussian, nonstationary, nonwhite, and that have a zero
mean.
2.2. Unmixed Signal Model
To obtain unmixed signals, unmixing filters wij(k) of Q-
taps are estimated, and the unmixed signals are obtained
as
?
The unmixing filters are estimated so that the unmixed
signals become mutually independent. This paper considers
a two-input, two-output convolutive BSS problem, i.e., N =
M = 2 (Fig. 1) without a loss of generality.
yi(n) =
M
j=1
Q
?
q=1
wij(q)xj(n − q + 1) (i = 1,···,N).(2)
2.3. Task of Blind Source Separation of Speech Sig-
nals
It is assumed that the source signals s1 and s2 are mutu-
ally independent. This assumption usually holds for sounds
in the real world. There are two microphones which pick
up the mixed speech. Only the observed signals x1 and x2
are available and they are dependent. The goal is to adapt
the unmixing systems wij, and extract y1 and y2 so that
they are mutually independent. With this operation, we
can obtain s1 and s2 in the output y1 and y2. No informa-
tion is needed on the source positions or period of source
existence/absence. Nor is any information required on the
mixing systems hji. Thus, this task is called blind source
separation (Fig. 2).
Note that the unmixing systems wij can at best be ob-
tained up to a scaling and a permutation, and thus cannot
itself solve the dereverberation/deconvolution problem [8].
A robust and precise method for solving the permutation
problem of frequency-domain BSS was proposed in [9], and
a minimal distortion principle for solving the scaling prob-
lem was proposed in [10].
3. WHAT IS ICA?
Independent component analysis (ICA) is a statistical
method that was originally introduced in the context of
neural network modeling [11]. Recently, this method has
been used for the BSS of sounds, fMRI and EEG signals
of biomedical applications, wireless communication signals,
images, and other applications. ICA thus became an ex-
citing new topic in the fields of signal processing, artificial
neural networks, advanced statistics, information theory,
and various application fields.
Very general statistical properties are used in ICA the-
ory, namely information on statistical independence. In a
source separation problem, the source signals are the “inde-
pendent components” of the data set. In brief, BSS poses
the problem of finding a linear representation in which the
components are mutually independent. ICA consists of es-
timating both the unmixing matrix W(ω) and sources si,
when we only have the observed signals xj.
The unmixing matrix W(ω) is determined so that one
output contains as much information on the data as pos-
sible. The value of any one of the components gives no
information on the values of the other components. If the
unmixed signals are mutually independent, then they are
equal to the source signals.
4. HOW SPEECH SIGNALS CAN BE
SEPARATED?
This paper attempts a simple and comprehensive (rather
than accurate) exploration from the acoustical signal pro-
cessing perspective in the frequency domain. With the ICA-
based BSS framework, how can we separate speech signals?
The simple answer is to diagonalize RY in each fre-
quency bin, where RY is a (2×2) matrix:
?
The function Φ(·) is a nonlinear function. The operation ?·?
is the averaging operation used to obtain statistical infor-
mation. We want to minimize the off-diagonal components,
while at the same time, constraining the diagonal compo-
nents to proper constants.
The components of the matrix RY correspond to the
mutual information between Yi and Yj. At the convergence
point, the off-diagonal components, which are the mutual
information between Y1 and Y2, become zero:
RY =
?Φ(Y1)Y1?
?Φ(Y2)Y1?
?Φ(Y1)Y2?
?Φ(Y2)Y2?
?
. (3)
?Φ(Y1)Y2? = 0,
?Φ(Y2)Y1? = 0.(4)
While at the same time, the diagonal components, which
only control the amplitude scaling of the output Y1 and Y2,
V - 669
¬¬

Page 3

are constrained to proper constants:
?Φ(Y1)Y1? = c1,
?Φ(Y2)Y2? = c2. (5)
To achieve this convergence, we use the recursive learn-
ing rule [12, 13].
Wi+1 = Wi+ η∆Wi, (6)
∆Wi =
?
c1− ?Φ(Y1)Y1?
?Φ(Y2)Y1?
?Φ(Y1)Y2?
c2− ?Φ(Y2)Y2?
?
Wi. (7)
When RY is diagonalized, ∆W converges to zero.
4.1. Second Order Statistics (SOS) Approach
If Φ(Y1) = Y1, we have the simple decorrelation:
?Φ(Y1)Y2? = ?Y1Y2? = 0. (8)
This is not sufficient to achieve independence, therefore, we
cannot solve the problem. This can be understood in a com-
prehensive way in that we have four unknown parameters
Wij in each frequency bin, but only three equations in (4)
and (5) since Y1Y2 = Y2Y1 when Φ(Yi) = Yi, that is, the
simultaneous equations become underdetermined. Accord-
ingly the simultaneous equations cannot be solved.
However, when the sources are nonstationary, the sec-
ond order statistics is different in each time block. As a
result, more equations are available and the simultaneous
equations can be solved. This is the nonstationary decorre-
lation approach [14].
Similarly, when the sources are nonwhite, we have a
delayed correlation for a multiple time delay:
?Φ(Y1)Y2? = ?Y1(m)Y2(m + τi)? = 0, (9)
The second order statistics is different in each time de-
lay, thus more equations are available and the simultaneous
equations can be solved. This is the time-delayed decorre-
lation (TDD) approach [15].
These are the approaches of second order statistics
(SOS).
4.2. Higher Order Statistics (HOS) Approach
On the other hand if, for example, Φ(Y1) = tanh(Y1), we
have:
?Φ(Y1)Y2? = ?tanh(Y1)Y2? = 0.
With a Tailor expansion of tanh(·), (10) can be expressed
as
?(Y1−Y3
315
(10)
1
+2Y5
1
−17Y7
1
315...) Y2? = 0,(11)
thus we have higher order or nonlinear decorrelation, then
we can solve the problem. Or more simply, we could say
that we have four equations in (4) and (5) for four unknown
parameters Wij in each frequency bin. Accordingly the si-
multaneous equations can be solved.
This is the approach of higher order statistics (HOS)
[16].
H12
H22
W11
W12
S1
S2
X1
X2
Y1
0
(a) ABF for target S1 and jammer S2.
S1
S2
X1
X2
Y2
0
H11
H21
W21
W22
(b) ABF for target S2 and jammer S1.
Figure 3: Two sets of ABF-system configurations.
5. SEPARATION MECHANISM OF BSS
BSS is a statistical, or mathematical method, so the physi-
cal behavior of BSS is not obvious. We are simply attempt-
ing to make the two output signals Y1 and Y2 independent.
Then, what is the physical interpretation of BSS?
We can understand the behavior of BSS as two sets of
ABFs [6]. An ABF can create only one null towards the
jammer when two microphones are used. BSS and ABFs
form an adaptive spatial null in the jammer direction, and
extract the target.
5.1. Frequency-Domain
(ABF)
Here, we consider the frequency-domain adaptive beam-
former (ABF), that can adaptively remove a jammer signal.
Since the aim is to separate two signals S1 and S2 with two
microphones, two sets of ABFs are used (see Fig. 3). That
is, an ABF that forms a null directivity pattern towards
source S2 by using filter coefficients W11 and W12, and an
ABF that forms a null directivity pattern towards source
S1 by using filter coefficients W21 and W22. Note that the
direction of the target or the impulse responses from the
target to the microphones should be known, and that the
ABF can adapt only when a jammer is active but a target
is silent.
The separation performance of BSS is compared with
that of ABF. Figure 4 shows the directivity patterns ob-
tained by BSS and ABF. In Fig. 4, (a) and (b) show di-
rectivity patterns by W obtained by BSS, and (c) and (d)
show directivity patterns by W obtained by ABF. When
TR = 0, a sharp spatial null is obtained by both BSS and
ABF [see Figs. 4(a) and (c)]. When TR= 300 ms, the direc-
tivity pattern becomes duller for both BSS and ABF [see
Figs. 4(b) and (d)].
AdaptiveBeamformer
6. DISCUSSIONS
BSS was interpreted from the physical standpoint showing
the equivalence between frequency-domain BSS and two
sets of microphone array systems, i.e., two sets of adap-
tive beamformers (ABFs) [6]. Convolutive BSS can be un-
derstood as multiple ABFs that generate statistically inde-
V - 670
¬¬

Page 4

-90
-80
-60
-40
-20
angle (deg.)
0
20
40
60
80
90
-40
-20
0
10
gain [dB]
0
1
2
3
4
frequency (kHz)
(a)
-60
BSS TR = 0 ms
-90
-80
-60
-40
-20
angle (deg.)
0
20
40
60
80
90
-40
-20
0
10
gain [dB]
0
1
2
3
4
frequency (kHz)
(b)
BSS TR = 300 ms
-90
-80
-60
-40
-20
angle (deg.)
0
20
40
60
80
90
-40
-20
0
10
gain [dB]
0
1
2
3
4
frequency (kHz)
(c)
-60
ABF TR = 0 ms
-90
-80
-60
-40
-20
angle (deg.)
0
20
40
60
80
90
-40
-20
0
10
gain [dB]
0
1
2
3
4
frequency (kHz)
(d)
ABF TR = 300 ms
Figure 4:
(TR=0 ms), (b) obtained by BSS (TR=300 ms), (c) ob-
tained by ABF (TR=0 ms), and (d) obtained by ABF
(TR=300 ms).
Directivity patterns (a) obtained by BSS
pendent output, or more simply, an output with minimal
crosstalk.
Because ABF and BSS mainly deal with sound from the
jammer direction by making a null towards the jammer, the
separation performance is fundamentally limited [5]. This
understanding clearly explains the poor performance of BSS
in the real world with long reverberation. If the sources are
not “independent,” their dependency results in bias noise
to obtain the correct unmixing filter coefficients. Therefore,
the BSS performance is upper bounded by that of the ABF.
However, in contrast to the ABF, no assumptions re-
garding array geometry or source location need to be made
in BSS. BSS can adapt without any information on the
source positions or period of source existence/absence. This
is because, instead of adopting power minimization crite-
rion that adapt the jammer signal out of the target signal
in ABF, a cross-power minimization criterion is adopted
that decorrelates the jammer signal from the target signal
in BSS. It was shown that the least squares criterion of ABF
is equivalent to the decorrelation criterion of the output in
BSS. The error minimization was shown to be completely
equivalent to a zero search in the cross-correlation.
Although the performance of the BSS is limited by that
of the ABF, BSS has a major advantage over ABF. A strict
one-channel power criterion has a serious crosstalk or leak-
age problem in ABF, whereas sources can be simultaneously
active in BSS. Also, ABF needs to know the array manifold
and the target direction. Thus, BSS can be regarded as an
intelligent version of ABF.
7. CONCLUSIONS
The blind source separation (BSS) of convolved mixtures of
acoustic signals, especially speech, was examined. Source
signals can be extracted only from observed mixed signals,
by achieving nonlinear, nonstationary, or time-delayed de-
correlation. The statistical technique of independent com-
ponent analysis (ICA) was studied from the acoustic signal
processing point of view.
ACKNOWLEDGMENTS
We thank Dr. Hiroshi Saruwatari for detailed discussion.
REFERENCES
[1] S. Makino, “Blind Source Separation of Convolutive
Mixtures of Speech,” in Adaptive Signal Processing:
Applications to Real-World Problems, J. Benesty and
Y. Huang, Eds., Springer, Berlin, Jan. 2003.
[2] J. F. Cardoso, “The three easy routes to independent
component analysis; contrasts and geometry,” in Proc.
ICA, Dec. 2001, pp. 1–6.
[3] T. W. Lee, A. J. Bell, and R. Orglmeister, “Blind
source separation of real world signals,” Neural Net-
works, vol. 4, pp. 2129–2134, 1997.
[4] M. Z. Ikram and D. R. Morgan, “Exploring permuta-
tion inconsistency in blind separation of speech signals
in a reverberant environment,” in Proc. ICASSP2000,
June 2000, pp. 1041–1044.
[5] S. Araki, R. Mukai, S. Makino, T. Nishikawa, and H.
Saruwatari, “The fundamental limitation of frequency
domain blind source separation for convolutive mix-
tures of speech,” IEEE Trans. Speech Audio Process-
ing, vol. 11, no. 2, pp. 109–116, Mar. 2003.
[6] S. Araki, S. Makino, Y. Hinamoto, R. Mukai, T.
Nishikawa, and H. Saruwatari, “Equivalence between
frequency domain blind source separation and fre-
quency domain adaptive beamforming for convolutive
mixtures,” EURASIP Journal on Applied Signal Pro-
cessing, accepted.
[7] R. Mukai, S. Araki, H. Sawada, and S. Makino, “Sepa-
ration and dereverberation performance evaluation of
frequency domain blind source separation,” Journal on
Acoust. Sci. & Tech., accepted.
[8] E. Weinstein, M. Feder, and A. V. Oppenheim, “Multi-
channel signal separation by decorrelation,”
Trans. Speech Audio Processing, vol. 1, no. 4, pp. 405–
413, Oct. 1993.
[9] H. Sawada, R. Mukai, S. Araki, and S. Makino,
“A robust approach to the permutation problem of
frequency-domain blind source separation,” in Proc.
ICASSP, Apr. 2003, pp. 381–384.
[10] K. Matsuoka and S. Nakashima, “ Minimal distortion
principle for blind source separation,” in Proc. ICA,
Dec. 2001, pp. 722–727.
[11] J. Herault and C. Jutten,
signal processing by neural network models,” in Neural
networks for computing: AIP conference proceedings
151, New York J. S. Denker, ed., American Institute
of Physics, Ed., 1986.
[12] S. Amari, A. Cichocki, and H. Yang, “A new learning
algorithm for blind source separation,” in Advances in
Neural Information Processing Systems 8, pp. 757–763,
MIT Press, 1996.
[13] A. Cichocki, R. Unbehauen, and E. Rummert, “Ro-
bust learning algorithm for blind separation of sig-
nals,” Electronics Letters, vol. 30, no. 17, pp. 1386–
1387, 1994.
[14] K. Matsuoka, M. Ohya, and M. Kawamoto, “A neu-
ral net for blind separation of nonstationary signals,”
Neural Networks, vol. 8, no. 3, pp. 411–419, 1995.
[15] L. Molgedey and H. G. Schuster,
mixure of independent signals using time delayed cor-
relations,” Physical Review Letters, vol. 72, no. 23, pp.
3634–3636, 1994.
[16] A. Hyvarinen, H. Karhunen, and E. Oja, Independent
component analysis, John Wiley & Sons, 2001.
IEEE
“Space or time adaptive
“Separation of a
V - 671
¬«