Gaussian moments for noisy complexity pursuit

Abstract

Complexity pursuit is an extension of projection pursuit to time series data and the method is closely related to blind separation of time-dependent source signals and independent component analysis (ICA). In this paper, we consider the estimation of the data model of ICA when Gaussian noise is present and the independent components are time dependent. We derive a simple algorithm combining Gaussian moments and complexity pursuit for noisy ICA. Validity and performance of the described approaches are demonstrated by computer simulations.

Full text

Page 1

Neurocomputing 69 (2006) 917–921
Letters
Gaussian moments for noisy complexity pursuit
Zhenwei Shi?, Changshui Zhang
State Key Laboratory of Intelligent Technology and Systems, Department of Automation, Tsinghua University, Beijing 100084, PR China
Received 12 August 2005; received in revised form 27 September 2005; accepted 31 October 2005
Available online 4 January 2006
Communicated by R.W. Newcomb
Abstract
Complexity pursuit is an extension of projection pursuit to time series data and the method is closely related to blind separation of
time-dependent source signals and independent component analysis (ICA). In this paper, we consider the estimation of the data model of
ICA when Gaussian noise is present and the independent components are time dependent. We derive a simple algorithm combining
Gaussian moments and complexity pursuit for noisy ICA. Validity and performance of the described approaches are demonstrated by
computer simulations.
r 2005 Elsevier B.V. All rights reserved.
Keywords: Independent component analysis; Blind source separation; Complexity pursuit; Projection pursuit; Time series
1. Introduction
Recently, blind source separation
component analysis (ICA) has received attention because
of its potential signal processing applications. The model of
ICA consists of mixing independent random variables,
usually linearly [1–3,6,10,12,13]. In many applications,
however, what is mixed is not random variables but time
signals, or time series. Complexity pursuit, whose goal is to
find projections of time series that have interesting
structure, is introduced by Hyva ¨ rinen [9]. It is an extension
of projection pursuit [5] to time series data having
interesting structure, defined using criteria related to
Kolmogoroff complexity [11] or coding length. Time series
which have the lowest coding complexity are considered
the most interesting. This idea is probably connected to
information-processing principles used in the brain. Shi
et al. [14] derived a simple approximation of Kolmogoroff
complexity that takes into account both the non-Gaus-
sianity and the autocorrelations of the time series. And
by independent
they developed a fixed-point algorithm for complexity
pursuit. The method is closely related to blind separation
of time-dependent source signals and ICA.
In this paper, we consider the estimation of the data
model of ICA when Gaussian noise is present and the
independent components (ICs) are time dependent. Hyva ¨ r-
inen [7,8] introduced a modification of the FastICA
algorithm [6] using the Gaussian moments for the problem
of estimating the data model of ICA in the presence of
Gaussian noise when the ICs are random variables.
Motivated by these methods, we combine Gaussian
moments to complexity pursuit and derive a simple
algorithm for noisy ICA when the ICs are time dependent.
2. Complexity pursuit and Gaussian moments
In this section, we consider the estimation of the ICA
model when the ICs are time signals. The model is
expressed by the following noisy linear model:
xðtÞ ¼ AsðtÞ þ nðtÞ,
where xðtÞ ¼ ðx1ðtÞ;...;xMðtÞÞT2 RMis a sensor signal,
A 2 RM?M
ðs1ðtÞ;...;sMðtÞÞT2 RMis the source signal and nðtÞ is the
noise which is modeled as Gaussian with zero mean and
(1)
isan unknownmixing matrix,sðtÞ ¼
ARTICLE IN PRESS
www.elsevier.com/locate/neucom
0925-2312/$-see front matter r 2005 Elsevier B.V. All rights reserved.
doi:10.1016/j.neucom.2005.10.002
?Corresponding author. Tel.: +861062796872;
fax: +861062786911.
E-mail addresses: szw1977@yahoo.com.cn (Z. Shi),
zcs@mail.tsinghua.edu.cn (C. Zhang).

Page 2

covariance matrix R (i.e., R ¼ EfnðtÞnðtÞTg), where t ¼
1;...;T and t is the time index. The source signals are
assumed independent such that pðsðtÞÞ ¼QM
we assume that the noise covariance matrix R is known and
the source signals have zero mean and unit variance.
i¼1pðsiðtÞÞ where
p denotes the probability density function. For simplicity,
2.1. Quasi-whitening
A useful preprocessing strategy in ICA is to first whiten
the observed variables. However, the effect of noise must be
considered in the preliminary whitening of the data. If the
noise covariance matrix is known, the ordinary whitening
should be replaced by the ‘quasi-whitening’ operation
~ xðtÞ ¼ ðC ? RÞ?1=2xðtÞ,
covariance matrix of the observed noisy data. The quasi-
whitened data ~ x follows a noisy ICA model as well, i.e.,
~ xðtÞ ¼ BsðtÞ þ ~ nðtÞ, with an orthogonal mixing matrix B
(from EfxðtÞxðtÞTg ¼ C ¼ AATþ R and B ¼ ðC ? RÞ?1=2A),
and the following noise covariance matrix [7,8]:
whereC ¼ EfxðtÞxðtÞTg
isthe
~R ¼ Ef~ nðtÞ~ nðtÞTg ¼ ðC ? RÞ?1=2RðC ? RÞ?1=2.(2)
2.2. Predictive coding and approximation of complexity
Kolmogoroff complexity is based on the interpretation
of coding length as structure [9]. Most natural signals
have redundancy; parts of the signal can be efficiently
predicted from other parts. Such a signal can be coded, or
compressed, so that the code length is shorter than the
original code length. We could thus measure the amount of
structure of the signal by the amount of compression that is
possible in coding the signal. For signals of fixed length,
the structure could be measured by the length of the
shortest possible code for the signal. This is a nonrigorous
definition of Kolmogoroff complexity [9] (for a more
rigorous definition of the concept, see the paper [11]).
Denote the noise-free data by yðtÞ ¼ BsðtÞ, the basic idea in
the complexity pursuit is to find projections wTyðtÞ such that
the Kolmogoroff complexity of the projection is minimized.
First, we derive an approximation of the Kolmogoroff
complexity of a scalar signal zðtÞðt ¼ 1;...;TÞ [14]. We
consider predictive coding of a scalar signal. The value zðtÞ
is predicted from the preceding values by some function to
be specified:
^ zðtÞ ¼ fðzðt ? 1Þ;...;zð1ÞÞ.
To code the actual value zðtÞ, the residual,
dzðtÞ ¼ zðtÞ ? ^ zðtÞ,
is coded by a scalar quantization method. According to the
basic principles of information theory [4], the length of this
code is asymptotically approximated by the sum of the
entropies H of the residuals. The coding complexity can be
approximated by [9,14]:
(3)
(4)
^KðzÞ ¼ THðdzÞ,(5)
where dz denotes a random variable with the marginal
distribution of the residual.
To use the approximation in Eq. (5) in practice, we need
to fix the structure of the predictor f and find an
approximation of the entropy of dz. We use a computa-
tionally simple predictor structure, given by a linear
autoregressive model:
X
To approximate the entropy of dz, we assume that we
know a good approximation of the (negative) logarithm of
the probability density of the residual, denoted by G. Then
we obtain the approximation [14]
^ zðtÞ ¼
t40
atzðt ? tÞ.(6)
HðdzÞ ? EfGðdzÞg.
In ICA, if the signals to be reconstructed satisfy certain
properties, an exact form of the contrast function is not
required in order to achieve the desired estimation results
[9,14]. We may therefore optimistically assume that the
exact form of the function G is not very important here
either, as long as it is qualitatively similar enough to the
(negative) logarithm of the probability density of the
residual [9,14].
To find the ‘most interesting’ directions w, use the above
approximation of complexity for zðtÞ ¼ wTyðtÞ. Thus, we
can express the approximation of complexity as a contrast
function of w only [14]:

(7)
min
kwk2¼1
^KðwTyðtÞÞ ¼ E G wT
yðtÞ ?
X
t40
atyðt ? tÞ
! !()
.
(8)
Using the contrast function, we can derive the original
gradient descent complexity pursuit algorithm.
2.3. Gaussian moments
The approach above could be used for noisy complexity
pursuit as well (i.e., ICs are time signals in the noisy
model), if only we had measures of the Kolmogoroff
complexity which are immune to Gaussian noise, or at
least, whose values for the original data can be easily
estimated from noisy observations. The main point is to
show that for certain choices of G, it is simple to estimate
the values of^K from noisy observation.
Denote by
? ?
the Gaussian density function of variance c2, and by
jðlÞ
by jð?lÞ
c
ðxÞ the lth integral function of jcðxÞ, obtained by
jð?lÞ
cc
ðxÞdx, where we define j0
Then we have the following theorem [7,8]:
jcðxÞ ¼1
cj
x
c
¼
1ffiffiffiffiffiffi
2p
p
cexp ?x2
2c2
??
, (9)
cðxÞ the lth (l40) derivative of jcðxÞ. Denote further
ðxÞ ¼Rx
Theorem 1. Let v be any non-Gaussian random variable, and
denote by n an independent Gaussian noise variable of
0jð?lþ1Þ
cðxÞ ¼ jcðxÞ.
ARTICLE IN PRESS
Z. Shi, C. Zhang / Neurocomputing 69 (2006) 917–921
918

Page 3

variance s2. Define the Gaussian functionj as in (9). Then
for any constant c4s2we have
EfjcðvÞg ¼ Efjdðv þ nÞg
with d ¼
replaced by jðkÞfor any integer index k.
(10)
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
c2? s2
p
. Moreover, (10) still holds when j is
This theorem means that we can estimate the ICs from
noisy observations by minimizing a general contrast
function of form (8), where the estimation of the statistics
^KðwTyðtÞÞ of the noise-free data is made possible by using
GðuÞ ¼ jðkÞ
is called the Gaussian moments [7,8]. Thus from (8)
and(10), we can estimate the noisy ICA model by
minimizing, for quasi-whitened data ~ x, the following
contrast function:

cðuÞ. The statistics of the form EfjðkÞ
cðwTyðtÞÞg
min
kwk2¼1
^KðwT~ xðtÞÞ ¼ E jðkÞ
dðwÞwT
~ xðtÞ ?
X
t40
at~ xðt ? tÞ
!!
(11)
()
,
where dðwÞ ¼
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
c2? ð1 þP
t40a2
tÞwT ~Rw
q
.
3. A simple algorithm for noisy complexity pursuit
To perform the optimization in (11), we can use a simple
gradient descent. The gradient of^K in Eq. (11) with respect
to w can be obtained as (see Appendix A):

X
?~RwE g0
rw^KðwT~ xðtÞÞ ¼ E
~ xðtÞ ?
X
t40
at~ xðt ? tÞ
!!)

!
g wT
~ xðtÞ
!
(
?
t40
at~ xðt ? tÞ

?
1 þ
X
t40
a2
t

X
wT
~ xðtÞ ?
t40
at~ xðt ? tÞ
!!
ð12Þ
()
.
The function g is here the derivative of G, and can be
chosen, for example: gðuÞ ¼ tanhðuÞ, which is an approx-
imation of jð?1Þ(one can choose other functions such as
given in [7–9,14]). Thus, after the operation ‘quasi-whiten’,
the algorithm is then as follows. At every step, first estimate
the autoregressive constants atin Eq. (6) for the time series
given by wT~ xðtÞ;t ¼ 1;...;T. Then do the gradient descent
and normalization:

X
?~RwE g0
t40
w w ? m E
~ xðtÞ ?
!!)

X
t40
at~ xðt ? tÞ

X
!
g wT
~ xðtÞ

!
!
((
?
t40
at~ xðt ? tÞ

?
1 þ
X
t40
a2
t
wT
~ xðtÞ ?
at~ xðt ? tÞ
!())
,
ð13Þ
w w=kwk, (14)
where m is a learning rate. To estimate several projections,
one can simply use a deflation scheme (Gram–Schmidt
orthogonalization scheme) [6].
A simple special case of the method is obtained when the
autoregressive model has just one predicting term [9]
^ zðtÞ ¼ a1zðt ? 1Þ.
The lag need not be equal to 1, but this is the basic case.
The parameter a1in the algorithm can then be estimated
simply by a least-squares method as
(15)
^ a1¼ wTEf~ xðtÞ~ xðt ? 1ÞTgw.(16)
4. Simulations
We created four signals using an AR(1) model. Signals 1
and 2 were created with super-Gaussian innovations and
signals 3 and 4 with Gaussian innovations; all innovations
had unit variance. Signals 1 and 3 had identical auto-
regressive coefficients (0.25) and therefore identical auto-
covariances; signals 2 and 4 had identical coefficients (0.5)
as well. The observations x were generated by a 4 ? 4
random mixing matrix with a white Gaussian noise
(covariance 0:01I). The sample size T was varied from
5000 to 65000, and the performance index defined as [1]

E ¼
X
M
i¼1
X
M
j¼1
jpijj
maxkjpikj? 1
!
þ
X
M
j¼1
X
M
i¼1
jpijj
maxkjpkjj? 1
!
,
(17)
where pijis the ijth element of M ? M matrix P ¼ WA (W
is the separating matrix in ICA). Ordinary ICA methods
and methods based on autocovariances would fail with
these data [9]. For the goal of comparison, we test three
algorithms (FastICA for noisy ICA (FastnoisyICA) [7,8];
complexity pursuit (CP) [9]; complexity pursuit for noisy
ICA in this paper (NoisyCP)) in the simulations. The step
size m was taken equal to 1 and the nonlinearity was chosen
as gðuÞ ¼ tanhðuÞ. At every trial, the three algorithms were
run with 80 iterations, which seemed to be always enough
for convergence. Fig. 1 plots the performance indexes of
the three algorithms averaged over 100 independent runs.
Obviously, the proposed NoisyCP algorithm performs
most efficiently.
In order to measure further the accuracy of separation,
the original signals and the matching estimated sources
were given for a true comparison. We performed the
maximum a posteriori approach [13] for obtaining the
estimated sources. Then we normalized the original sources
with ksjk2¼ 1 and the estimated sources with k~ sjk2¼ 1
(j ¼ 1;2;3;4). The errors were computed as k~ sj? sjk2
(j ¼ 1;2;3;4). Table 1 shows the average errors of the
four estimated sources computed by the three algorithms.
We can see that the FastnoisyICA algorithm based on
non-Gaussianity separated only the first three signals and
failed to separate the two Gaussian signals from each
ARTICLE IN PRESS
Z. Shi, C. Zhang / Neurocomputing 69 (2006) 917–921
919

Page 4

other. The CP algorithm did not work well because of the
presence of Gaussian noise.
5. Conclusions
In this letter, we have presented an approach to
estimation of the noisy ICA when the independent
components are time dependent. We combine Gaussian
moments to complexity pursuit and derive a simple
algorithm. Our method is closely connected to other well-
known algorithms. First, assume that the time signals are
not corrupted by additive Gaussian noise (covariance
matrix R is zero), our method is in fact the complexity
pursuit algorithm, so closely connected to classical ICA
algorithms and some other algorithms using the time-
dependency information [9]. Second, assume that the
signals have no time dependencies, our method is in fact
the gradient algorithm for noisy ICA using the Gaussian
moments.
Acknowledgements
The authors wish to gratefully thank all the anonymous
reviewers who provided insightful and helpful comments.
The work was supported by NSFC(60475001, 10261005,
10571018) and by the Chinese Postdoctoral Science
Foundation (2005038075).
Appendix A. Derivation of the gradient
Denote
dðwÞ ¼
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
t40
c2?
1 þ
X
a2
t
!
wT ~Rw
v
u
u

t
,(A.1)
X ¼ wT
~ xðtÞ ?
X
t40
at~ xðt ? tÞ
!
,(A.2)
~d ¼ c2?
1 þ
X
t40
a2
t
!
wT~Rw,(A.3)
and note that
jðkÞ
dðwÞðXÞ ¼ jðkÞ
X
dðwÞ
??
dðwÞð?k?1Þ.
dðwÞwT~ xðtÞ ?P
dðwÞðXÞrwX ?1
?k þ 1
2
(A.4)
Then the gradient of jðkÞ
with respect to w can be obtained as
t40at~ xðt ? tÞ
????
rwjðkÞ
dðwÞðXÞ ¼ jðkþ1Þ
2d?2ðwÞXjðkþ1Þ
dðwÞðXÞrw~d
d?2ðwÞjðkÞ
dðwÞðXÞrw~d.
ðA:5Þ
To proceed, we use the lemma in [8] to imply
d?2ðwÞðk þ 1ÞjðkÞ
¼ ?jðkþ2Þ
This means the gradient in (A.5) can be expressed as
dðwÞðXÞ þ d?2ðwÞXjðkþ1Þ
dðwÞðXÞ.
dðwÞðXÞ
ðA:6Þ
rwjðkÞ
Choosing GðuÞ ¼ jðkÞ
that the exact form of the nonquadratic function used to
probe high-order statistics is not very important, the
function G needs only to be of a given shape [7–9], we have

X
?~RwE g0
dðwÞðXÞ ¼ jðkþ1Þ
dðwÞðXÞrwX þ1
cðuÞ and again in ICA, it is well known
2jðkþ2Þ
dðwÞðXÞrw~d.(A.7)
rw^KðwT~ xðtÞÞ ¼ E
~ xðtÞ ?
X
t40
at~ xðt ? tÞ
!!)

!
g wT
~ xðtÞ
!
(
?
t40
at~ xðt ? tÞ

?
1 þ
X
t40
a2
t

X
wT
~ xðtÞ ?
t40
at~ xðt ? tÞ
!!
ðA:8Þ
()
,
where the function g is the derivative of G and the function
g0is the derivative of g. Thus, we have the approximation
of the gradient in (12).
ARTICLE IN PRESS
1234567
x 104
0
1
2
3
4
5
6
7
8
9
10
Fig. 1. Convergence of the three algorithms for artificially generated noisy
data. Horizontal axis: the sample size. Vertical axis: the average
performance indexes over 100 independent runs. Dashed line: Fastnoi-
syICA; dot-dashed line: CP; solid line: NoisyCP.
Table 1
The average errors of the four estimated sources computed by the three
algorithms
AlgorithmsSource1Source2Source3Source4
FastnoisyICA
CP
NoisyCP
0.3273
0.5827
0.3813
0.4183
0.4668
0.4603
0.2456
0.3468
0.2259
1.4848
0.6028
0.4085
Z. Shi, C. Zhang / Neurocomputing 69 (2006) 917–921
920

Page 5

References
[1] S.-I. Amari, A. Cichocki, H. Yang, A new learning algorithm for
blind source separation, in: D. Touretzky, M. Mozer, M. Hasselmo
(Eds.), Advances in Neural Information Processing System, vol. 8,
MIT Press, Cambridge, MA, 1996, pp. 757–763.
[2] A. Bell, T. Sejnowski, An information-maximization approach to
blind separation and blind deconvolution, Neural Comput. 7 (6)
(1995) 1129–1159.
[3] P. Comon, Independent component analysis—a new concept?, Signal
Process. 36 (1994) 287–314.
[4] T.M. Cover, J.A. Thomas, Elements of Information Theory, Wiley,
New York, 1991.
[5] J.H. Friedman, J.W. Tukey, A projection pursuit algorithm for
exploratory data analysis, IEEE Trans. Comp. C-23 (9) (1974)
881–890.
[6] A. Hyva ¨ rinen, Fast and robust fixed-point algorithms for indepen-
dent component analysis, IEEE Trans. Neural Networks 10 (3) (1999)
626–634.
[7] A.Hyva ¨ rinen,Gaussianmoments
component analysis, IEEE Signal Process. Lett. 6 (6) (1999)
145–147.
[8] A. Hyva ¨ rinen, Fast ICA for noisy data using Gaussian moments, in:
Proceeding of the International Symposium on Circuits and Systems,
Orlando, FL, 1999, pp. 57–61.
[9] A.Hyva ¨ rinen, Complexity
components from time-series, Neural Comput. 13 (4) (2001)
883–898.
[10] C. Jutten, J. Herault, Blind separation of sources, part I: an adaptive
algorithm based on neuromimetic architecture, Signal Process. 24
(1991) 1–10.
[11] P. Pajunen, Blind source separation using algorithmic information
theory, Neurocomputing 22 (1998) 35–48.
fornoisy independent
pursuit: separating interesting
[12] Z. Shi, H. Tang, Y. Tang, A new fixed-point algorithm for
independent component analysis, Neurocomputing 56 (2004) 467–473.
[13] Z. Shi, H. Tang, W. Liu, Y. Tang, Blind source separation of more
sources than mixtures using generalized exponential mixture models,
Neurocomputing 61 (2004) 461–469.
[14] Z. Shi, H. Tang, Y. Tang, A fast fixed-point algorithm for complexity
pursuit, Neurocomputing 64 (2005) 529–536.
Zhenwei Shi received his B.S. and M.S. degrees in
Mathematics from the Inner Mongolia Univer-
sity, Huhhot, China, in 1999 and 2002, respec-
tively.Hereceived his
Mathematics from the Dalian University of
Technology, Dalian, China, in 2005. He is
currently a Postdoctoral Researcher in the
Department of Automation, Tsinghua Univer-
sity, Beijing, China. His research interests
include blind signal processing, pattern recogni-
tion, machine learning and neuroinformatics.
Ph.D. degreein
Changshui
in Mathematics from the Peking University,
Beijing, China, in 1986, and Ph.D. degree from
the Department of Automation, Tsinghua Uni-
versity, Beijing, China, in 1992. He is currently a
Professor of the Department of Automation,
Tsinghua University. He is an Associate Editor
of the journal Pattern Recognition. His interests
include artificial intelligence, image processing,
pattern recognition, machine learning and evolu-
tionary computation, etc.
Zhangreceived hisB.S. degree
ARTICLE IN PRESS
Z. Shi, C. Zhang / Neurocomputing 69 (2006) 917–921
921