Noise robust estimates of correlation dimension and K2 entropy.

Noise� robust estimates of correlation dimension and K2 entropy
Guido� Nolte,1,* Andreas� Ziehe,2,† and� Klaus-Robert Mu¨ller� 2,3,‡
1Department of Computer Science, University of New Mexico, Albuquerque, New Mexico 87131-1386
2� GMD� FIRST.IDA, Kekule´strasse 7, 12489 Berlin, Germany
3� University of Potsdam, Am Neuen Palais 10, 14469 Potsdam, Germany
�
Received 1 February 2001; published 15 June 2001
Using
Gaussian kernels to define the correlation sum we derive simple formulas that correct the noise bias
in estimates of the correlation dimension and K2� entropy of chaotic time series. The corrections are only based
on� the difference of correlation dimensions for adjacent embedding dimensions and hence preserve the full
functional

dependencies on both the scale parameter and embedding dimension. It is shown theoretically that
the� estimates, which are derived for additive white Gaussian noise, are also robust for moderately colored
noise. Simulations underline the usefulness of the proposed correction schemes. It is demonstrated that the
method� gives satisfactory results also for non-Gaussian and dynamical noise.
DOI: 10.1103/PhysRevE.64.016112 PACS number � s� : 02.50. � r, 05.45.Tp, 05.45.Ac
I.� INTRODUCTION
Analysis of nonlinear dynamics plays an important role in
science.� Especially low-dimensional chaos has been found in
various� natural and technical systems, e.g., epileptic spike
trains� � 1� ,� EEG signals during sleep � 2ff ,� or the cardiovascular
system� fi 3fl ffi .
One� of the important invariant measures to characterize a
time� series generated by nonlinear dynamics is the correla-
tion� dimension D 4–6! . Following in the lines of Grass-
berger" and Procaccia # 4$ the� fractal dimension of the attrac-
tor� can be estimated from a time series (x% i)
& by using the
power' law behavior of the correlation sum
C (*) ,� m+ , -
i . j/ 021436587
x9 i : x
9
j/ ;=<?>A@
D
,� B 1C
whereD the vectors
xi E8F x
%
i ,� x
%
i GIH ,� ..., x
%
i JIK (L mM N 1) O
T P 2Q
are� constructed from mapping the time series (x% i)
& into an
m+ -dimensional embedding space R forS choosing the value of T
see,� e.g., U 7V WYX and� the Heaviside function Z is used to count
the� number of points inside1 a� hypersphere of radius [ .
For Eq. \ 1] to� hold, several requirements have to be ful-
filled:^ _ a� `ba mustc be sufficiently small to avoid finite size
effects,d e b" fhg must be sufficiently large to ensure sufficient
statistics� and to avoid discretization errors, i cj k the� embed-
dingl dimension m+ mustc be chosen large enough to unfold the
attractor,� and m dl n the� influence of noise—being itself infinite
dimensional—mustl be negligible. Assuming that all these
conditionsj hold, the correlation dimension can be estimated
by"
Do p dq r*s ,� m+ t
dq ln C
dq ln� u ,
� v 3fl w
whichD should be independent of xzy ‘‘scaling’’{ and� indepen-
dentl of m+ | ‘‘saturation’’} in some ~ range and for sufficiently
large� m+ .
In practice, the estimate of the correlation dimension will
be" biased if the number of available data points is insuffi-
cientj € 8,9 ‚ . Even more severe, the signals obtained from real-
worldD systems are unavoidably contaminated with noise.
Thisƒ makes a reasonably accurate estimate of the correlation
dimensionl extremely difficult if not impossible to achieve.
Depending on the specific dynamics, already a noise level of
1–2 % can ruin scaling and saturation, which are necessary
conditionsj to reliably assign a dimension to a time series „ 7V … .
So† far, several methods have been developed to reduce
the� error in estimates of dimensions caused by additive noise
‡
see� ˆ 10,11‰ forS overviewsŁ . One general idea is to account
forS the theoretically expected deviation from the simple scal-
ing behavior of Eq. ‹ 1Œ . This was done for the Grassberger-
Procaccia type of correlation sum  12–16Ž and� for the corre-
lation� sum with Gaussian kernels  17,18 ,� which we will
discussl in detail in the next section. Typically, the corrected
estimated is found by a functional fit. Similar methods were
applied� to estimate the noise level ‘ 19,20’ . A different ap-
proach' is taken in “ 12” proposing' a non-linear scale transfor-
mationc to compensate for the noise bias. This method, how-
ever,d is an approximation that becomes inaccurate for scales
smaller� than the noise level.
For• practical applications, functional fitting of refined
scaling� laws to empirical correlation sums has two severe
drawbacks.l –
1— It is unclear in what range of ˜ this� fit should be
performed' since the given functional form is only valid for
‘‘sufficiently’’ small ™ and� the estimated correlation dimen-
sion� is statistically relevant only for ‘‘sufficiently’’ large š .
Since,† especially, the former cannot be assessed without any
knowledge› of the underlying system, the estimation of Do isœ
likely� to be based on wrong assumptions.

2ž In order to conclude that a time series originates from
a� Ÿ low-dimensional  deterministicl dynamical system of a cer-
*Email¡ address: nolte@cs.unm.edu
†¢ Email address: ziehe@first.gmd.de
࣠Email address: klaus@first.gmd.de
1Distances are measured by the Euclidian norm ¤¦¥¨§ , which is ap-
propriate for the purpose of this paper.
PHYSICAL REVIEW E, VOLUME 64, 016112
1063-651X/2001/64 © 1ª /016112« ¬ 10­ /$20.00« ©2001 The American Physical Society64® 016112-1

tain� dimension, one has to verify that dq (¯ ° ,� m+ )& does not de-
pend' on ± withinD a suitable range. Though the explicit evalu-
ation� of ‘‘scaling regimes’’ is to some extent subjective, we
regard figures of an estimated dimension as a function of ² ,�
not just as an intermediate step but as an important final
result.³ However, when estimating Do by" a functional fit, the
dependencel on ´ is lost by definition, and hence it is no
longer possible to check the scaling behavior.
In order to avoid those functional fits in finite scaling
ranges,³ we will use Gaussian kernels in the definition of the
correlationj sum as proposed in µ 17¶ . This will allow us to
derivel an explicit formula for the correlation dimension as a
function of m+ and� · . We will show that for every scale, the
necessary information needed to correct for the noise bias is
completelyj contained in the difference of two uncorrected
dimensionl estimates for adjacent embedding dimensions.
Though, to our opinion, the estimate of correlation dimen-
sion� from fitting in scale space has significant drawbacks, the
actual� result may still be accurate. Since fitting, as proposed
in ¸ 17¹ ,� merely exploits º dependencel and we will here
merely exploit m+ dependence,l the results are based on mutu-
ally� independent information. In practice, comparing both
results³ may give additional insight into the dynamical system
under» study eventually leading to a stronger confirmation of
the� estimated dimension.
Section† II is devoted to theoretical aspects of our ap-
proach.' In Sec. II A we present the central idea of this paper
by" showing how one can remove the bias caused by white
Gaussian� noise in an extremely simple manner. Generaliza-
tions� with respect to other invariants and nonwhite noise will
be" done in Sec. II B, II C, and II D. General remarks on the
use» of Gaussian kernels are given in Sec. II E. In Sec. III we
demonstratel the usefulness of our method for various simu-
lation� examples and we finally give a conclusion in Sec. IV.
II. THEORY
A.¼ Corrected dimensions for white Gaussian noise
In order to remove the bias caused by white Gaussian
noise in the estimate of the correlation dimension we work
withD Gaussian kernels ½ 17¾ and� define
Cg
¿ À4Á
,� m+ Â?Ã Ä
i Å j/ Æ tÇ minÈ
expd ÉËÊ
xi Ì xj/ Í
2Î
4Ï Ð 2
Î . Ñ 4Ï Ò
To avoid spurious effects arising from autocorrelation we
haveÓ excluded pairs that are too close in time by introducing
a� minimal delay tÔ minM . This was proposed in Õ 21Ö whereD the
recommendation³ is to choose tÔ minM to
� be larger than the auto-
correlationj time.
In contrast to the formulation with the Heaviside step
functionS × hardÓ kernelØ inœ Eq. Ù 1Ú ,� using a Gaussian function
Û
soft� kernelÜ hasÓ the effect that contributions from pairs with
Ý
x9 i Þ x
9
j/ ßáàAâ do
l
not vanish but are exponentially suppressed.
Theƒ power law scaling behavior, however, coincides in both
casesj as can be checked explicitly by applying the transfor-
mation law of Sec. II E 2.
In the noisy case the measured values are given by
yã i ä x
%
i åçæ i ,� è 5
é ê
whereD we assume that ë i is white Gaussian noise with stan-
dardl deviation ì .
Let us now calculate the expectation of the correlation
sum� in the presence of noise. Denoting an index pair (ií ,� jî ) b& y
ï and� x9 ðòñ x9 i ó x9 j/ ô analogously� for yã and� õ )
&
, Eq. ö 4Ï ÷ reads³
forS the noisy case
Cg
¿ ø*ù
,� m+ ú?ûAüþý expd ß��
x �������
2
4 � 2
Î ,� � 6
�
whereD ��� isœ now a difference between two vectors of inde-
pendent' Gaussian random numbers with standard deviation
�
,� and hence corresponds to independent Gaussian noise
withD standard deviation � 2 � ,� with probability density
p� �����fiff�fl
kffi � 1
mM 1
2 "! #
expd $&% k
ffi
2
4 ' 2
Î . ( 7V )
Accordingly,� the expectation of the correlation sum with re-
spect� to the noise * reads
+
Cg
¿ ,.-
,� m+ /10.2 Do 3 Cg
¿ 465
,� m+ 7 p� 8�9;:
<>=@?BA
kffi C 1
mM 1
2D E"F G
dq H kffi
I
expd J�K
x% L kffi MON kffi P
2Î
4 Q 2 R
S
kffi
2Î
4 T 2
,� U 8 V
whereD x% W kffi denotes
l
the kX th� component of x Y and� D Z
[]\
kffi d
q ^
kffi . Note, that we have omitted the irrelevant index _
on` a .
In order to perform the integration with respect to b the�
exponentd of Eq. c 8 d is rewritten as
e�f
x% g kffi hOi kffi j
2
4 k 2
Î l&m
kffi
2
4 n 2
Î oqpsr
2 t>u 2
4 v 2
Î w 2Î x k
ffi y z
2
{ 2Î |>} 2Î x
% ~
kffi
2

x% € kffi
2
4Ï ƒ‚ 2 „>… 2 †
. ‡ 9ˆ ‰
WithŁ
1
‹
dq Œ kffi expd sŽ
2 > 2
4 ‘ 2 ’ 2 “ k
ffi ” •
2
– 2 —>˜ 2 x
% ™
kffi
2Î
š ›
œ 
2 ž>Ÿ 2
 
10¡
weD find
¢
Cg
¿ £6¤
,� m+ ¥§¦©¨ ª« ¬ 2 ­>® 2
mM ¯
°
expd ± ²
x9 ³
´ 2
Î
4 µƒ¶ 2 ·>¸ 2 ¹ º
11»
GUIDO NOLTE, ANDREAS ZIEHE, AND KLAUS-ROBERT MU¨ LLER¼ PHYSICAL REVIEW E 64½ 016112¾
016112-2

¿ À
Á Â
2Î Ã>Ä 2Î
mM ÅÇÆ
2Î È>É 2Î Ê DË /2Ì
. Í 12Î
Thisƒ result was already found in Ï 17Ð withinD a slightly
differentl approach. It is proposed there to estimate Do fromS a
functional fit of Eq. Ñ 12Ò to� the measured correlation sum.
Inserting Eq. Ó 12Ô intoœ Eq. Õ 3fl Ö and� using Cg
¿
as� the corre-
lation� sum leads to the estimate see� also Eq. × 8 Ø inœ Ù 19Ú
dq Û.Ü ,� m+ ÝfiÞ D ßáà m+ â D ãåä
2
æ 2Î ç>è 2Î . é 13ê
The above-mentioned drawbacks of fitting the ë dependencel
canj now be overcome by adding a subtleì but important
point:' According to Eq. í 13î one` has
ï 2
ð 2Î ñóò 2Î ô d
q õ6ö
,� m+ ÷ 1 øfiù dq ú.û ,� m+ üfiý : þ ß � ,� m+ � � 14�
and� insertion into Eq. � 13� and� solving for Do leads� to the
simple� relation
D � dq �
�� ,� m+
dq ��� ,� m+ ��� m+ ����� ,� m+ �
1 �ff��fi�fl ,� m+ ffi . � 15
Therefore,ƒ it is no longer needed to determine the noise cor-
rected³ dimension dq ! fromS the functional behavior over a fi-
nite range of " . Instead, it is estimated independently for
eachd value of # by" merely using the results of the ‘‘stan-
dard’’l correlation dimension estimates for two adjacent em-
bedding" dimensions.
WeŁ note that the corrected dimension estimate of Eq. $ 15%
dependsl explicitly on m+ . For large m+ the� correction becomes
large and potentially inaccurate. However, from writing
Do & dq ')(�* ,� m+ +�, dq -�. ,� m+ /�021
m+ 3 dq 4�5 ,� m+ 68789;:�< ,� m+ =
1 >ff?;@�A ,� m+ B ,
� C 16D
weD see that the correction is proportional to m+ E dq (¯ F ,� m+ ).&
HenceG we can expect that the performance of the proposed
scheme� is essentially independent of dq (¯ H ,� m+ )& itself and can
be" also applied to systems with large correlation dimension
as� long as the embedding dimension does not exceed the
correlationj dimension by a large amount.
WeŁ want to emphasize that one should clearly distinguish
between" the ‘‘bare scale’’ I and� the ‘‘effective scale’’
J e fK f LNMPO 2Î QSR 2Î T 1/2 U 17V
of` the exponent in Eq. W 11X . While Y canj be set arbitrarily
small,� Z e f
K f is always larger than [ . Even with perfect bias
correction,j the correlation sum is blind to scales \ of` the
noise-free] signal^ below" the noise level. Note, however, that
noise already severely distorts the correlation sum for bare
scales� _ farS above noise level.
B. Robust estimates of the K2 entropy`
In order to correct an estimate of the Ka 2 entropy,d let us
first recall its definition b e.g.,d c 6,22
dfe . In the noise-free case
and� assuming proper scaling, one may write the m+ depen-l
dencel of the correlation sum for large m+ as�
Cg
¿ g�h
,� m+ i�j ck expd l�m mK+ 2Î npo D,� q 18r
whereD ck is a constant. K2Î canj hence be estimated as the limit
m+ sut of`
Ka 2 v�w ,� m+ x�y ln
� z Cg
¿ {�|
,� m+ }~p€ ln�  Cg
¿ ‚�ƒ
,� m+ „ 1 …† . ‡ 19ˆ
Theƒ Ka 2Î entropyd measures the exponential increase of the
uncertainty» about future values given the past up to finite
accuracy.� More precisely, it is a lower bound on the sum of
positive' Lyapunov exponents and is hence a measure of
‘‘how chaotic’’ a system is ‰ see,� e.g., Ł 2D 4–2 6‹fŒ . Linear sys-
tems,� for example, must have K2Î  0
Ž
while the correlation
dimensionl can be arbitrarily large.
In the noisy case Eq.  19 hasÓ to be modified. Including m+
dependencel of the noise-free correlation sum, the noisy cor-
relation sum Eq. ‘ 12’ canj be written as
“
Cg
¿ ”�•
,� m+ –˜—�™ ck expd šœ› mK+ 2Î  žŸ   2 ¡S¢ 2
mM £P¤
2Î ¥§¦ 2Î ¨ DË /2Ì
.©
20D ª
It follows that
ln� « Cg
¿ ¬�­
,� m+ ®˜¯ ° ln� ±³² Cg
¿ ´�µ
,� m+ ¶ 1 ·¹¸»ºp¼ Ka 2 ½
1
2D ln
� ¾
2Î ¿SÀ 2Î
Á 2 .
Â
21Ã
NotingÄ that with Eq. Å 14Æ
1 ÇffÈ;É�Ê ,� m+ Ë�Ì Í
2
Î 2Î ÏSÐ 2Î ,
� Ñ 22Ò
weD can calculate a bias-free estimator of the Ka 2Î entropyd from
the� limit m+ ÓuÔ of`
Ka 2Õ)Ö�× ,� m
+ Ø ln� Ù Cg
¿ Ú�Û
,� m+ Ü8ÝpÞ ln� ß Cg
¿ à�á
,� m+ â 1 ã8ä
å
1
2 ln æ 1 çffè;é�ê ,
� m+ ë8ì . í 23î
It is well known that the estimation of the K2 entropyd
requires³ in general a much larger embedding dimension to
show� a proper saturation than would be needed for the re-
spective� correlation dimension. However, in contrast to the
correlationj dimension, the correction here does not explicitlyï
increaseœ with m+ . Though this does not imply that implicit
dependenciesl are present, we may expect that the entropy
estimatesd are more robust than the dimension estimates for
m+ ð Do .
NOISE ROBUST ESTIMATES OF CORRELATION . . . PHYSICAL REVIEW E 64ñ 016112¾
016112-3

C. Estimating the noise level
Similarly† to the correction of the correlation dimension
and� K2 entropy,d the use of Gaussian kernels allows to obtain
estimatesd of the noise level itself. Solving Eq. ò 14ó for the
noise level ô leads to
õ÷öøõúù�û
,� m+ ü�ý þ
2 ß
1 �
�
1/2
. � 24�
This estimate is now a function of � and� m+ allowing� to check
forS scaling and saturation. Again, the estimate is based on
dependenciesl of the correlation dimension on m+ alone.� The
dependencel on � canj now serve as an independent consis-
tency� check, and comparisons with functional fits as done in
�
17–19� mayc mutually confirm the results obtained.
D. Colored noise
In the case of colored noise one finds approximately � see�
Appendix
dq
�� ,� m+
�� Do ��� m+ � Do ����� � 25D �
withD
ff�fiffifl
1
m+ �k ! 1
mM "
k
2
#
k
2 $&% 2Î ,
� ' 26(
whereD ) k
2Î is the k* th� eigenvalue of the ‘‘noise-autocorrelation
matrix’’
R+ i j ,.- i / j/ 0 1 27
D 2
and� where ií ,� jî 3 1, . . . ,m+ denotel the time points. For white
noise R i j 465 2 7 i j ,� the eigenvalues are all equal to 8 2 and� one
arrives� back at Eq. 9 13: .
As� explained in detail in the Appendix, colored noise
leads to ellipsoidal Gaussian kernels. The approximation in
Eqs. ; 25< and� = 26> consistsj of assuming that the correlation
sum� only depends on the volume of the ellipsoid and not on
its shape, which also depends on the scale ? . This depen-
dencel becomes smaller for larger @ and� hence Eqs. A 25D B and�
C
26D are� not only absolutely E because" of the smaller biasF but"
also� structurally more accurate for larger G .
Theƒ correction formula Eq. H 15I canj be applied as long as
JLKMJON
,� which is valid if POQ isœ sufficiently independent of m+ .
This is obvious for the ‘‘useless’’ case in the limit of RTS 0,Ž
since� then U�VffiW 1 and the dependence on Do dropsl out.
Less trivial, this is also the case for XZY6[ k ,� since then
\�]ffi^
_
k `
k
2
m+ a 2 b
tr� c R d
m+ e 2 f
28D g
and� tr(R)& is proportional to m+ if the diagonal elements are all
equald as for the case of stationary noise. In general, the m+
dependencel is a complicated function of the noise spectrum.
Roughlyh speaking, i�j correspondsj to averages in the fre-
quencyk domain: the larger the m+ ,� the better the resolution. If
the� resolution is sufficiently large to consider the power
spectrum� as locally constant, the m+ dependencel will disap-
pear.'
In conclusion, in the case of colored noise the corrected
dimensionl estimate of Eq. l 15m becomes" more accurate for
larger� n not] only because the noise bias is smaller but also
because" of two structural reasons: o a� p the� approximation in
Eqs. q 25r and� s 26t is not only absolutely but also relatively
morec accurate and u b" v the� dependence of wOx on` m+ decreases.l
In place of y (¯ z ,� m+ )& from Eq. { 14| one` can also use, e.g.,
}
˜ ~� ,� m+ €  dq ‚„ƒ ,� m+ … 1 †�‡ dq ˆ„‰ ,� m+ Ł 1 ‹ffiŒ /2, Ž 29D 
whichD is statistically more robust, but a drawback is that a
sufficient� embedding is already required for m+  1 instead of
m+ . At first sight it seems that for colored noise this definition
wouldD be preferable to Eq. ‘ 14’ because" of its apparent
symmetry.� 2
Î
However, after defining dimensions at half-
integer embedding spaces m+ ˆ “ m+ ” 1/2 by the mean
dq •„– ,� m+ ˆ —�˜š™ dq ›„œ ,� m+  1 ž�Ÿ dq  �¡ ,� m+ ¢ffi£ /2, ¤ 30fl ¥
and� correcting according to
dq ¦�§�¨ ,� m+ ˆ ©�ª
dq «�¬ ,� m+ ˆ ­¯® m+ ˆ °²±�³ ,� m+ ´
1 µM¶²·�¸ ,� m+ ¹ ,
� º 31fl »
itœ is readily seen that this definition is identical to Eq. ¼ 15½ ,�
whichD can hence be regarded as a symmetric correction
around� half-integer embedding dimensions.
WeŁ finally note that for strictly Gaussian noise, the rela-
tion�
dq ¾ ln� Cg
¿ ¿„À
,� m+ Á¯Â
dq ln Ã Ä m
+ Å�Æ�Ç�È
,� m+ É Ê 32fl Ë
holdsÓ exactly. This may be used to reveal spurious nonzero
dimensionl estimates caused by correlations of the noise.
E.
Ì On the use of Gaussian kernels
1. Calculating the derivative
One� might guess that a significant drawback of the use of
Gaussian� kernels is the apparent smearing of scales not
present' when using Heaviside kernels. However, the correla-
tion� sum calculated from step functions is not continuous: to
differentiatel it, one must use a finite difference over a con-
siderable� range of ÍÏÎ 7,27V Ð ,� which in fact also smears the
scales.� Since the correlation sum defined by Gaussian kernels
isœ differentiable, the latter form of smearing can be avoided
and� we can directly calculate dq (¯ Ñ ,� m+ )& as the derivative of the
correlationj sum
2SimilarÒ to the central difference for numerically approximating a
derivative.
GUIDO NOLTE, ANDREAS ZIEHE, AND KLAUS-ROBERT MU¨ LLER¼ PHYSICAL REVIEW E 64Ó 016112¾
016112-4

dq Ô„Õ ,� m+ Ö�×
dq ln Cg
¿ Ø�Ù
,� m+ Ú
dq ln Û�Ü¯Ý Þ
ßáàãâ
yä åZæ 2
Î
expd çéè�ê yä ëZì 2
Î
/4í î 2
Î ï
2D ð 2 ñóò expd ôéõ�ö yä ÷Zø 2/4í ù 2 ú
,�
û
33fl ü
whereD again the double index ý denotesl all pairs þ (¯ ií ,� jî )& ß ií
� jî � that� are included in the sum and yã ��� yã i � yã j/ . In all of
our` numerical simulations we use this formula to directly
calculatej the � uncorrected» � correlationj dimension.
2.� Transformation of Heaviside kernels to Gaussian kernels
A� correlation sum based on Gaussian kernels may be ex-
pressed' by a correlation sum based on step functions
Cg
¿ �

,� m+ ��
dq ��� f� ��� ,� ����� C ��ff�fi ,� m+ fl ffi 34fl �
withD C(¯ �! ,� m+ )& from Eq. " 4# . In order to find the correct
weightingD function f� (¯ $ ,� %�& )& , it is sufficient to express the
Gaussian� kernel by the Heaviside kernel. From
expd ')(
yã * 2
Î
4Ï + 2 , 0-
.
dq /�0 f� 1�2 ,� 3�46587:9<;�=?>A@ yã BDC ,� E 35fl F
itœ follows by partial integration that
f� G�H ,� I�J6K8LNM
O
2 P 2
Î expd QSR�T
2
4 U 2
Î V 36fl W
in agreement with X 18Y . The weight function3
Z
f� (¯ [ ,� \�] )& can be
used» to transform any quantity calculated from hard kernels
intoœ the respective ones calculated from Gaussian kernels.
Especially,^ it follows from
0-
_
dq `�a f� b�c ,� d�e6fhg�i DË jlk DË m 38fl n
that� the power law scaling—if it exists—will be the same for
both" correlation sums.
The relation between hard and Gaussian kernels could, in
principle,' be used to speed up the computation since the
correlationj sums according to Grassberger and Procaccia are
muchc simpler to calculate. However, there is a tradeoff: for
an� accurate calculation a fine o resolution is required that
partly' spoils the beneficial effect if the correlation sums are
calculatedj directly by implementing Eq. p 1q .
An� extremely efficient method to avoid this problem was
presented' in r 18s whereD the authors proposed to first calcu-
late� the histogram of t yä i u yä j/ v from
S
which the correlation sum
withD respect to any kernel can readily be calculated in neg-
ligible computer time. We would like to suggest a slight
modification of this method by calculating the histogram of
the� logarithmw of` the squared distances in order to ensure
sufficient� resolution also for small scales. Explicitly this
means that one rewrites the relevant sums as
x6y
f� z|{ yä }�~ 2  expd €�A‚ yä ƒ…„ 2/4í † 2 ‡
ˆ dzq f ‰ expŁ ‹ zŒ �Ž expŁ ‘ expŁ ’ zŒ “ /4í ” 2• –
— ˜6™›šœ
zŒ ž lnŸ   yä ¡…¢ 2 £ ¤ 39¥ ¦
with§ f� (¨ © yä ª�« 2)¬ ­ 1 for the denominator and f� (¨ ® yä ¯…° 2)¬ ±A² yä ³…´ 2 forµ
the¶ numerator of Eq. · 33¥ ¸ . The term in curly brackets can
now be approximated by the respective histogram and the
integral is finally approximated by the respective sum. Tak-
ing,¹ e.g., 100 values for a unit step of zŒ resultsº in essentially
exactŁ correlation sums.
III. SIMULATION RESULTS
A.» Dimension estimates in the presence
of white Gaussian noise
Numerical¼ results will be given mainly for the He´non
map,½ which is defined by
x¾ i ¿ 1 À 1 Á ax i
2 Ã bxÄ i Å 1 ,Æ Ç 40
È É
with§ aÂ Ê 1.4 and bÄ Ë 0.3Ì Í 4È Î . The time lag Ï forµ embedding
accordingÐ to Eq. Ñ 2Ò is set to 1. For convenience, all time
seriesÓ considered in this paper were normalized according to
x¾ i Ô
x¾ i
Õ
xÖ
,Æ × 41È Ø
whereÙ Ú xÖ denotes
Û
the standard deviation of the time series
(Ü x¾ i)
¬
. For additive noise, the normalization was done with
respectº to the noise-free data and for dynamical noise, with
respectº to the noisy data.
In order to evaluate our method we added white Gaussian
noiseÝ with standard deviation Þàß 0.1Ì to the time series cor-
respondingº to a noise level of 10%.
From the noisy data we compute uncorrected dimension
estimatesŁ with Eq. á 33â ã forµ embedding dimensions mä
å 2, . . . ,8 and subsequentlyÓ correct these estimates according
toæ Eq. ç 15è . Apart from Fig. 3 é whereÙ we compare N ê 500ë
andÐ N ì 20 000í ,Æ in all simulations Nî ï 5000ë time points are
usedð for the estimation of the invariants.
Inñ Fig. 1 we plot the results of uncorrected and corrected
estimatesŁ of the correlation dimension. Indeed, a proper scal-
ing and saturation behavior is completely ruined by the
3ò It should be noted that for fixed ó the maximum of fô (õ ö ,÷ ø8ù )ú
occurs at û8üþý ß 2 � implying a mismatch of scales. Replacing the
definition of the Gaussian kernel according to
exp �
y� 2
�
4 � 2
� � exp �
y� 2
�
2 � 2
� 37

ends up with a proper match of scales in the sense that Cg
� ( � ,m
)ú
gets the largest contribution from C� (õ ��� ,m
)ú at the same scale. Of
course, the specific choice is merely a convention, and in fact, the
present definition is slightly more convenient for the analytical cal-
culations.
NOISE ROBUST ESTIMATES OF CORRELATION . . . PHYSICAL REVIEW E 64� 016112�
016112-5

Gaussian� noise. In contrast, our corrected estimates show
scalingÓ and saturation at the correct dimension.
The correction breaks down if the scale � becomes� too
small:Ó the noisy correlation dimensions converge to the em-
bedding� dimensions and do not depend on the dimension of
theæ noise-free signal, and hence, solving for the latter be-
comes� ill defined. Note however, that for low embedding
dimensions,Û � may½ be considerably smaller than the noise
levelŸ while still allowing for a reasonable dimension esti-
mate.
The estimation becomes more and more difficult for
higher� embedding dimensions since then, the necessary rela-
tiveæ correction increases strongly. We note again that the
magnitude of the correction depends rather on the difference
of� embedding dimension and correlation dimension of the
noise-freeÝ signal than on the embedding dimension itself.
Thus,ff if the number of data points is correspondingly larger,
one� can expect to obtain similar results equally well also for
higher-dimensional dynamics.
Forfi other tests of the dimension estimation method we use
theæ time series obtained from the Ro¨sslerÓ and Lorenz systems
fl
25ffi of� differential equations that were then superimposed by
whiteÙ Gaussian noise with standard deviation �! 0.1.Ì For
continuous� systems the delay time " can� take arbitrary val-
ues.ð Here, we set #%$ 1 and &%' 0.25Ì for the Ro¨sslerÓ and Lo-
renz systems, respectively.
Theff results are shown in Fig. 2. Again we find very sat-
isfactory( bias removal. While none of the uncorrected di-
mension½ estimates shows scaling, the Lorenz system at least
approximatelyÐ saturates at the correct value, at )%* 0.5.Ì Re-
markably,½ for the noisy Ro¨sslerÓ system the correct dimension
cannot� even be anticipated before bias removal, but is nicely
recovered by our correction scheme.
For very large noise or for very few data the correction
stillÓ leads to qualitatively correct results. This can be seen in
Fig.fi 3 where we show the dimension estimates for the He´nonÝ
map with 50% noise, now using N + 20 000 data points and
withÙ 10% noise using N , 500ë data points. However, because
of� the relatively large fluctuations and the small scaling re-
FIG. 1. Uncorrected and corrected estimates of the correlation
dimension for embedding dimensions m - 2, . . . ,8 for the He´non
map in the presence of 10% white Gaussian noise. The true dimen-
sion D. / 1.21 is indicated by a dashed line. For simulations with
additive noise, the noise-free data, and with dynamical noise, the
noisy data were always scaled to standard deviation 1. Hence, any
quantity shown in this and in the following plots is dimensionless.
FIG. 2. Uncorrected and corrected estimates of the correlation
dimension for embedding dimensions m 0 2, . . . ,8 for the Ro¨ssler
and Lorenz system in the presence of 10% Gaussian noise. The
‘‘true’’ dimensions D. 1 2.052 3 44 for the Lorenz and D. 5 1.9 for the
Ro¨ssler system 6 estimated7 from 8 139;: are< indicated by dashed lines.
FIG. 3. Estimates of the correlation dimension for the He´non=
map with 50% noise level using N > 202 000 data points ? upper@ pan-
elsA and< with 10% noise using N B 500C data points D lower panelsE .F
GUIDO NOLTE, ANDREAS ZIEHE, AND KLAUS-ROBERT MU¨ LLERG PHYSICAL REVIEW E 64� 016112�
016112-6

gimeH we regard these examples as being at the limit of a
reasonable application of our method.
B. Noise level
Inñ Sec. II C we derived an estimator I (Ü mä ,Æ J )¬ for the stan-
dardÛ deviation of the noise. Like the dimension and K2 esti-Ł
mators,½ this construction has the advantage that in contrast to
fittingK procedures the consistency can be checked both in
termsæ of scaling and saturation properties.
In the upper left panel of Fig. 4 we show the noise esti-
mator½ for the He´nonÝ map with 10% white Gaussian noise. A
scalingÓ region is well established at scales in the order of the
noise level where the dependence of L on� the noise level is
maximal.½ Taking, e.g., the estimates in the middle of the
scalingÓ range at M%N 0.1051Ì results in a mean of O!P 0.0990Ì
withÙ a standard deviation of 0.003 in excellent agreement
withÙ the true value.
The estimate of the noise level becomes less stable for
smallerÓ noise since estimates at smaller scales are based on
fewerµ data. This can be seen in the upper right panel of Fig.
4 showing the result for 2% noise.
For comparison, we also plot the results for uniform and
colored� noise in the lower panels of Fig. 4. In both cases we
findK systematic but small deviations from the true noise
level. Uniform noise typically results in nice scaling behav-
ior with a small overestimation of the noise level, while in
theæ case of colored noise we observe a systematic underesti-
mation,½ which—analogous to the dimension estimates—is
more½ pronounced for small Q .
C.R K2 entropyS
Theff results for the KT 2• entropyŁ estimates are shown in Fig.
5.ë In contrast to the uncorrected estimates U left upper panelV
theæ corrected ones scale properly W right upper panelX . The
lowerŸ boundary of the scaling range grows for increasing
embeddingŁ dimensions, which is in fact a well-known prop-
ertyŁ also for noise-free data Y 22Z . A saturation behavior at the
correct� value is verifiable only for large embedding dimen-
sionsÓ (mä [ 10) as it also would have been expected from the
noise-freeÝ data \ 22] ^ .
The saturation is seen more clearly in the lower panels of
Fig.fi 5 where we show the KT 2• estimatesŁ as a function of the
embeddingŁ dimension for three fixed scales within the scal-
ing( regime. In the literature different values are given for KT 2•
of� the He´nonÝ map. Although it is not the primary goal of this
paper_ to settle this issue, our findings rather support K2•
` 0.325Ì as stated in a 24b thanæ K2 c 0.29
Ì from d 23e .
D. Other types of noise
For the derivation of the correction formulas we assumed
additiveÐ white Gaussian noise. In real-world data this as-
sumptionÓ will not hold exactly.
Inñ order to test the validity of our method also for noise
withÙ other probability distributions, we added uniform white
noise,Ý again with standard deviation f!g 0.1,Ì to the data gen-
eratedŁ by the He´nonÝ map and applied the same correction as
in the previous section. The upper panels of Fig. 6 show the
uncorrectedð and corrected dimension estimates. Though the
estimatesŁ are slightly worse, if compared with the Gaussian
noiseÝ case h cf.� Fig. 1i ,Æ both saturation and scaling are clearly
visiblej after correction.
Wek now address the case of nonwhite noise that, in prin-
ciple,� could be overcome by choosing a large value of lnm seeÓ
Eq. o 2prq or� by filtering the data appropriately before perform-
ing( the actual analysis in order to ‘‘whiten’’ the noise. How-
ever,Ł a too large s alsoÐ complicates the dimension estimation
sinceÓ due to the intrinsic chaotic nature of the dynamics,
functionalµ dependencies between consecutive data points are
diminished,Û and the correct filter to whiten the noise without
causing� severe phase distortions of the system itself is usu-
allyÐ unknown. Still, in order to get satisfactory results within
FIG. 4. Estimates of the respective noise levels t dashed lineu for
embedding dimensions m v 22 ,..., 8 for the He´non= map data. FIG. 5. Estimates of the correlation entropy K2� w 0.325
� x dashedy
linez for embedding dimensions m
{ 3,| . . . ,19 for the He´non= map in
the presence of 10% Gaussian noise. Upper panels: K} 2 as< a function
of ~ . Lower panels: K2 as< a function of embedding dimension for
€ 0.15,0.18,0.21 chosen from the scaling region.
NOISE ROBUST ESTIMATES OF CORRELATION . . . PHYSICAL REVIEW E 64� 016112�
016112-7

theæ proposed approach it would be advisable to avoid ex-
tremeæ deviations from the white-noise case since the latter
may½ ‘‘mimic low-dimensional chaotic attractors’’ ‚ 28] ƒ .
In order to check the robustness of our method against
non-iid noise, we added low pass filtered Gaussian noise to
theæ time series. The low pass filter was implemented by ap-
plying_ a moving average of order two to white Gaussian
noiseÝ with „!… 0.1,Ì
†
i ‡‰ˆ‹Ł i ŒŽ i  1  /2,
í ‘ 42’
“!”–• /í — 2 ˜ 0.071.Ì ™ 43š
The power spectrum of this colored noise then reads P(› œ )¬
Ÿž 1   cos(� ¡ )¬ ¢ /2,í where £¥¤§¦ 0,Ì ¨ª© is« the frequency and ¬ is«
the­ Nyquist frequency.
The results are shown in the lower panels of Fig. 6.
Again, we find a major improvement after applying the cor-
rection. However, if ® is smaller than the noise level we
observe� a systematic overestimate of the dimension in agree-
ment½ with the theoretical considerations stating that the ap-
proximations_ are more accurate for larger ¯ .
Wek finally present two examples using dynamical noise
that­ arises when the dynamical system itself and not merely
the­ measurement is disturbed by noise. This was realized by
aÐ small distortion of one variable of the Ro¨sslerÓ and Lorenz
systemÓ in each step of the integration of the differential
equations.Ł 4 Thoughff the estimates ° seeÓ Fig. 7± areÐ less stable
than­ for additive white Gaussian noise, we approximately
recover scaling and saturation at the correct values. Typi-
cally,� the correction leads to small but systematic underesti-
mates½ of the dimensions. Estimates of the noise level indi-
cate� that this dynamical noise roughly corresponds to 10%
additiveÐ noise. However, the lack of a clear scaling regime
can� readily serve as an indicator that the assumption of ad-
ditiveÛ white noise is inconsistent.
IV. CONCLUSION
Wek introduced a Gaussian kernel based method for reduc-
ing« the noise bias in estimates of correlation dimension and
K2• entropyŁ of dynamical system attractors.
In² contrast to most proposed methods and to all existing
exactŁ methods, our approach is localw in« scale space and re-
quires³ for each scale, only the knowledge of the function ´ :
the­ difference of the uncorrected dimension estimates for twoµ
adjacent embeddingŁ dimensions. Hence, both scaling and
saturationÓ can still be checked after bias removal. For prac-
tical­ purposes this latter property is highly desirable since in
most½ applications it is not clear, whether the time series un-
derÛ consideration is governed by deterministic chaotic dy-
namics or by an unstructured stochastic noise process.
Wek demonstrated the performance for various examples
usingð data from the He´nonÝ map and the Lorenz and Ro¨sslerÓ
system.Ó In all cases the noise level was chosen to destroy
scalingÓ and saturation at the true correlation dimension for
the­ uncorrected dimension estimate while these properties
could� be sufficiently recovered after bias removal. We could4The He´non map is unstable with respect to this perturbation.
FIG.¶ 6. Uncorrected · left panels¸ and corrected ¹ rightº panels»
estimates of the correlation dimension for embedding dimensions
m ¼ 2, . . . ,8 for the He´non map in the presence of uniformly distrib-
uted white noise, ½¿¾ 0.1 À upper panelsÁ and Gaussian distributed
colored noise, Ã¿Ä 0.071Å Æ lower panelsÇ . The true dimension D.
È 1.21 is always indicated by a dashed line.
FIG. 7. Estimates of correlation dimension for the Ro¨sslerÉ sys-
tem Ê left panelsË and for the Lorenz system Ì right panelsÍ perturbed
by dynamical noise. The Î estimates indicate a noise level of about
10%. Note, however, that no clear scaling regime can be observed
for the noise level estimates.
GUIDO NOLTE, ANDREAS ZIEHE, AND KLAUS-ROBERT MU¨ LLERÏ PHYSICAL REVIEW E 64Ð 016112Å
016112-8

showÓ experimentally that the proposed approach is not sen-
sitiveÓ to the distribution of the noise Ñ GaussianÒ versus uni-
formµ Ó . Experiments with dynamical noise led to qualitatively
correct� results with a small but systematic underestimate of
the­ dimension.
SpecialÔ emphasis was given to the problem of correlated
noise.Ý Also for this case we could derive an approximate
refinedº scaling law that turns out to depend on the eigenval-
uesð of the nontrivial noise covariance matrix in an
mÕ -dimensional embedding space. We could provide theoret-
ical« and experimental evidence that our method, which does
notÝ require knowledge of the noise characteristics itself, is
practicableÖ as long as the deviation from the white-noise case
is« not too large.
Estimation of the noise level and bias correction of K2
entropy× was achieved similarly. Again, Ø turned­ out to be
the­ crucial quantity sufficient to define a ‘‘noise function,’’
whichÙ should scale and saturate at the correct noise level,
andÐ to construct a bias-free estimator of KÚ 2Û . We could dem-
onstrate� a promising performance in case of the He´non map
even× though the estimate of KÚ 2Û in
«
the presence of noise is
generallyH considered to be an exceptionally difficult task
Ü
18Ý .
Future research will be devoted to applications of our es-
timation­ method to real-world data.
ACKNOWLEDGMENTS
K.-R.M.Þ and A.Z. acknowledge funding by the EU project
BLISSß à GrantÒ No. IST-1999-14190á . This study was sup-
portedÖ in part by the National Foundation for Functional
Brain Imaging â USAã ä .
APPENDIX
In case of colored noise, the probability distribution for
åçæ reads è henceforth omitting the index é on� ê )¬
pë ìîíðï�ñ
1
ò
detÛ ó R ô
exp× õ÷ö
R+ ø 1 ù
4 ú A1û
withÙ R+ givenH by Eq. ü 27] ý . In order to evaluate the expectation
of� the correlation sum
þ
Cg
ß ���
,Æ mÕ ����� ��

1
�
detÛ
R �
D� �
�
exp× ���
x �����ff� 2
Û
4fi fl 2 ffi
� R 1 !
4fi " A2
# $
withÙ D� % & k ' 1
m( d) * k ,Æ we reexpress the exponent as
+�,
x- .�/1032 2
Û
4 4 2 5
6 R+ 7 1 8
4fi 9;:
x- <>= 1 ? A@ A 1A@ B T C x- D
4 E 2
F�GIHKJ
A L Tx M>N 2
Û
4fi O 2 P
A3# Q
withÙ the definitions
ATA 1 RTS 2
Û
R U 1 V A4W
andÐ
X
A@ Y . Z A5# [
Usingã D� \^] D� _ /í det(A@ )¬ we find
`
Cg
ß a�b
,Æ mÕ c�d�e f
m(
detÛ A g detÛ R
hji
exp× k
x- l>m 1 n A@ o 1A@ p T q x- r
4 s 2
Û .
t
A6u
As we see, the presence of nonwhite noise has led to
nonspherical, ellipsoidal Gaussian kernels with an
mÕ -dimensional volume V givenv by
V w detx
1 y A z 1A { T
4fi | 2
Û
} 1/2
. ~ A7
Wek now assume that the correlation sum scales in the noise-
free case with the volume as € VD
 /‚ m(
. This is indeed an ap-
proximationÖ because the exact scaling law can in general
alsoƒ depend on the specific shape of the ellipsoid that varies
asƒ a function of „ . The length l… k of† the k
*
th­ axis is given by
the­ square root of the k* th­ eigenvalue of the matrix
in« the exponent of Eq. ‡ A6# ˆ : l… k ‰‹Ł (
› Œ
k
2 TŽ 2Û  withÙ  k
2 being‘
the­ k* th­ eigenvalue of R+ . For large ’ the­ ellipsoid becomes
spherical;“ especially, its shape becomes independent of ” .
Ignoring² • dependencex of the shape of the ellipsoid we
arriveƒ at
–
Cg
ß —�˜
,™ mÕ š�›�œ 
m( VD
 /‚ m(
detx A@ ž detx R+
. Ÿ A8#  
SinceÔ A@ is« merely a function of R+ weÙ may express ¡ Cg
ß (› ¢ ,™ mÕ )£ ¤
by‘ the eigenvalues ( ¥ k 2
Û
)£ , e.g.,
detx ¦ A@ §©¨ detx ª�« 1 ¬T­ 2
Û
R+ ® 1 ¯©°;±
k
²
k
2 ³T´ 2Û
µ
k
2
1/2
,™ ¶ A9# ·
leading¸ finally to
¹
Cg
ß º�»
,™ mÕ ¼�½�¾ ¿ m
( À
k Á 1
m( ÂÄÃ
k
2 ÅTÆ 2 Ç (È D É m( )/2Ê m(
. Ë A10# Ì
TheÍ calculation of the correlation dimension as given by Eqs.
Î
25Ï Ð andƒ Ñ 26Ï Ò is« now straightforward.
NOISE ROBUST ESTIMATES OF CORRELATION . . . PHYSICAL REVIEW E 64Ó 016112Ô
016112-9

Õ1Ö A. Babloyantz and A. Destexhe, Proc. Natl. Acad. Sci. U.S.A.
83× ,Ø 3513 Ù 1986Ú .Û
Ü
2Ý A.Þ Babloyantz, J. M. Salazar, and C. Nicolis, Phys. Lett. 111Aß ,
152 à 1985á .Û
â
3ã V.ä K. Yeragani, K. Srinivasan, S. Vempati, R. Pohl, and R.
Balon, J. Appl. Physiol. 75, 2429 å 1993æ .Û
ç
4è P.é Grassberger and I. Procaccia, Phys. Rev. Lett. 50, 346
ê
1983ë .
ì
5í P. Grassberger and I. Procaccia, Physica D 9î ,Ø 189 ï 1983ð .Û
ñ
6ò K. Pawelzik and H. G. Schuster, Phys. Rev. A 35,Ø 481 ó 1987ô .
õ
7ö H.÷ Kantz and T. Schreiber, Nonlinearø Time Series Analysis
ù
Cambridgeú University Press, Cambridge, England, 1997û .Û
ü
8ý L. A. Smith, Phys. Lett. A 133, 283 þ 1988ß .Û
�
9� A.Þ Stefanovska, S. Strle, and P. Kroselj, Phys. Lett. A 235� , 2Ø 4
�
1997� .
�
10� E. J. Kostelich and T. Schreiber, Phys. Rev. E 48,Ø 1752 � 1993� .Û

11
T. Schreiber, Phys. Rev. E 56,Ø 274 � 1997� .

12� D.� Kugiumtzis, Int. J. Bifurcation Chaos Appl. Sci. Eng. 6,
1283 � 1997� .Û
�
13� H. Oltmans and P. J. T. Verheijen, Phys. Rev. E 56, 1160
�
1997� .
�
14� M.� Mo¨ller,� W. Lange, F. Mitschke, N. B. Abraham, and U.
Hu¨bner,� Phys. Lett. A 138,Ø 176 ff 1989fi .
fl
15ffi E. Ott, E. D. Yorke, and J. A. Yorke, Physica D 16, 6Ø 2 � 1985 .
!
16" J.# C. Schouten, F. Takens, and C. M. van den Bleek, Phys.
Rev.$ E 50% , 1851 & 1994' .Û
(
17) C.ú Diks, Phys. Rev. E 53% ,Ø 4263 * 1996+ .Û
,
18- D.� Yu, M. Small, R. G. Harrison, and C. Diks, Phys. Rev. E
61Ó ,Ø 3750 . 2000/ .Û
0
191 T. Schreiber, Phys. Rev. E 48,Ø R13 2 19933 .
4
205 G.6 G. Szpiro, Physica D 65Ó ,Ø 289 7 19938 .Û
9
21: J.# Theiler, Phys. Rev. A 34, 2427 ; 1986< .
=
22> M.� Frank et al., Physica D 65Ó ,Ø 359 ? 1993@ .Û
A
23B P. Grassberger, R. Badii, and A. Politi, J. Stat. Phys. 51, 135
C
1988D .
E
24F P.é Grassberger and I. Procaccia, Physica D 13ß , 3Ø 4 G 1984H .
I
25J A.Þ Wolf, J. B. Swift, H. L. Swinney, and J. A. Vastano,
Physica D 16,Ø 285 K 1985L .Û
M
26N J.# Kurths and H. Herzel, Physica D 25� , 165 O 1987P .Û
Q
27R J.# Theiler and T. Lookman, Int. J. Bifurcation Chaos Appl. Sci.
Eng. 3S ,Ø 765 T 1993U .
V
28W P. E. Rapp, A. M. Albano, T. I. Schmah, and L. A. Farwell,
Phys.é Rev. E 47
X
, 2289 Y 1993Z .Û
GUIDO NOLTE, ANDREAS ZIEHE, AND KLAUS-ROBERT MU¨ LLER[ PHYSICAL REVIEW E 64Ó 016112Ô
016112-10Ô