# Modified correlation entropy estimation for a noisy chaotic time series.

**ABSTRACT** A method of estimating the Kolmogorov-Sinai (KS) entropy, herein referred to as the modified correlation entropy, is presented. The method can be applied to both noise-free and noisy chaotic time series. It has been applied to some clean and noisy data sets and the numerical results show that the modified correlation entropy is closer to the KS entropy of the nonlinear system calculated by the Lyapunov spectrum than the general correlation entropy. Moreover, the modified correlation entropy is more robust to noise than the correlation entropy.

**0**Bookmarks

**·**

**231**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**This project studies the dynamics of electromechanical systems, a Faraday disk coupled with a Bullard Dynamo. First the individual systems are stud-ied in detail. This is followed by the study of the coupled system. We find the fixed points, dynamical regimes for different set of parameters, and characterize the dynamics, by various measures of irregularity.07/2013, Degree: Summer Fellowship - [Show abstract] [Hide abstract]

**ABSTRACT:**Time series is widely exploited to study the innate character of the complex chaotic system. Existing chaotic models are weak in modeling accuracy because of adopting either error minimization strategy or an acceptable error to end the modeling process. Instead, interpolation can be very useful for solving differential equations with a small modeling error, but it is also very difficult to deal with arbitrary-dimensional series. In this paper, geometric theory is considered to reduce the modeling error, and a high-precision framework called Series-NonUniform Rational B-Spline (S-NURBS) model is developed to deal with arbitrary-dimensional series. The capability of the interpolation framework is proved in the validation part. Besides, we verify its reliability by interpolating Musa dataset. The main improvement of the proposed framework is that we are able to reduce the interpolation error by properly adjusting weights series step by step if more information is given. Meanwhile, these experiments also demonstrate that studying the physical system from a geometric perspective is feasible.Chaos (Woodbury, N.Y.) 09/2013; 23(3):033132. · 1.80 Impact Factor - SourceAvailable from: Osvaldo A Rosso[Show abstract] [Hide abstract]

**ABSTRACT:**The time evolution of the output of a semiconductor laser subject to delayed optical feedback can exhibit high-dimensional chaotic fluctuations. In this contribution, our aim is to quantify the degree of unpredictability of this hyperchaotic time evolution. To that end, we estimate permutation entropy, a novel information-theory-derived quantifier particularly robust in a noisy environment. The permutation entropy is defined as a functional of a symbolic probability distribution, evaluated using the Bandt-Pompe recipe to assign a probability distribution function to the time series generated by the chaotic system. This measure quantifies the diversity of orderings present in the associated time series. In order to evaluate the performance of this novel quantifier, we compare with the results obtained by using a more standard chaos quantifier, namely the Kolmogorov-Sinai entropy. Here, we present numerical results showing that the permutation entropy, evaluated at specific time-scales involved in the chaotic regime of the semiconductor laser subject to optical feedback, give valuable information about the degree of unpredictability of the chaotic laser dynamics. The influence of additive observational noise on the proposed tool is also investigated.IEEE Journal of Selected Topics in Quantum Electronics 01/2011; 17(5):1250-1257. · 4.08 Impact Factor

Page 1

Modified correlation entropy estimation for a noisy chaotic time series

A. W. Jayawardena,1,a?Pengcheng Xu,2and W. K. Li3

1International Centre for Water Hazard and Risk Management,

Public Works Research Institute, 305-8516 Tsukuba, Japan

2Academy of Mathematics and System Sciences, Chinese Academy of Sciences, Beijing 100080, China

3Department of Statistics and Actuarial Science, The University of Hong Kong, Hong Kong, Hong Kong

?Received 6 April 2009; accepted 16 March 2010; published online 22 April 2010?

A method of estimating the Kolmogorov–Sinai ?KS? entropy, herein referred to as the modified

correlation entropy, is presented. The method can be applied to both noise-free and noisy chaotic

time series. It has been applied to some clean and noisy data sets and the numerical results show

that the modified correlation entropy is closer to the KS entropy of the nonlinear system calculated

by the Lyapunov spectrum than the general correlation entropy. Moreover, the modified correlation

entropy is more robust to noise than the correlation entropy. © 2010 American Institute of Physics.

?doi:10.1063/1.3382013?

Correlation entropy (CE), which is an approximation of

the Kolmogorov–Sinai (KS) entropy, is an important

quantity that can be used to identify chaos in a nonlinear

dynamical system. It gives an indication of the predict-

ability of the nonlinear time series since the inverse of the

KS entropy evaluates the maximum time step for predic-

tion. Estimating the CE accurately is important for the

simulation and forecasting of a chaotic time series. How-

ever, this is made difficult by the presence of noise which

blurs the signal of the time series, thereby introducing

errors in the calculation of the correlation sum. To over-

come this difficulty, we introduce a new quantity, herein

referred to as the modified correlation entropy (MCE), as

an approximation of the KS entropy. For noise-free data,

the MCE is equivalent to the general CE. For noisy cha-

otic data, the MCE is closer to the KS entropy than the

general CE.

I. INTRODUCTION

Study of chaos in time series has seen a rapid growth

during the past two decades or so. Many types of time series

which appear to be stochastic superficially are in fact of a

deterministic nature of low dimension. If such types can be

identified, short-term predictions can be made using local

models rather than stochastic global models. The identifica-

tion of chaos involves the determination of certain invariants

such as the correlation dimension, Lyapunov exponent, en-

tropies, among others. They are usually carried out in a re-

constructed phase space of the original time series. By

Takens’s embedding theorem ?Takens, 1981?, a nonlinear

time series can be embedded into a delay phase space by a

special embedding dimension and an arbitrary choice of the

time delay.

Important characteristics of a chaotic time series are that

it is initial value sensitive, consists of a topological mixture,

has dense periodic orbits, and is fractal. The sensitivity of

initial value can be evaluated by the largest Lyapunov expo-

nent while fractal structure can be validated by estimating

the fractal dimension. The topological mixture rate of a non-

linear time series can be evaluated by the dynamical entropy.

The concept of dynamical entropy, which was first intro-

duced by Kolmogorov ?1958? and Sinai ?1959?, provides a

method to describe the mixing rate of a deterministic system.

Hence it is called the “KS” entropy, which is based on an

invariant measure of the attractor and it is a “measure of

theoretic entropy” of a nonlinear system. Later, the concept

of “topological entropy” ?T-Entropy? that does not employ a

metric for the system was introduced by Adler et al. ?1965?.

Given the evolution function of the dynamical system, the

T-Entropy or the KS entropy can be calculated. However, for

nonlinear time series, the evolution function is unknown, and

therefore the T-Entropy or the KS entropy cannot be calcu-

lated from the definitions. Several methods have been devel-

oped to estimate the KS entropy for a chaotic time series.

Grassberger and Procaccia ?1983? introduced a method to

approximate the KS entropy for a chaotic time series using

the correlation sum ?correlation integral?. The estimation of

the entropy by this method is then called the CE. Another

entropy measure for a time series is its “information en-

tropy,” which is also called the “Shannon entropy” since

it was first introduced by Shannon ?1948? and Weaver and

Shannon ?1949?. A generalized entropy for a nonlinear sys-

tem is also introduced according to the q-deformed algebra

and special functions developed by Renyi ?1971?, and it is

called the “Renyi entropy.” The information entropy and the

CE have been proved to be special cases of the Renyi en-

tropy. Methods have been developed to calculate various en-

tropy measures for a chaotic time series. Grassberger and

Procaccia ?1983? provided a simple formula to estimate the

CE for a chaotic time series, which has been applied widely.

Kantz and Schürmann ?1996? introduced a method to calcu-

late a finite order entropy to approximate the KS entropy.

Schürmann and Grassberger ?1996? provided a method to

estimate the information entropy for a symbolic sequence,

which can also be applied for a chaotic time series. Recently,

a?Author to whom correspondence should be addressed. Electronic mail:

hrecjaw@hkucc.hku.hk.

CHAOS 20, 023104 ?2010?

1054-1500/2010/20?2?/023104/11/$30.00© 2010 American Institute of Physics

20, 023104-1

Page 2

the above methods have been employed to estimate the en-

tropy measure for some special cases, such as chaotic time

series with small data size ?Bonachela et al., 2008?.

However, all chaotic time series are blurred by noise.

The presence of additive noise strongly affects the evolution

of the correlation sum, which in turn affects the estimation of

the CE. The correlation sum for a chaotic time series with

additive Gaussian noise has been discussed by several au-

thors. Most of the correlation integral equations depend non-

linearly on the KS entropy. Hence, it is difficult to estimate

the KS entropy by these correlation integral equations.

In this study, by employing the correlation integral

equation obtained by Diks ?1999? ?p. 123? and Oltmans and

Verheijen ?1997? for a chaotic time series with additive

Gaussian noise, we provide a method to estimate the KS

entropy, herein referred to as the “MCE,” which has the ad-

vantage that it is similar to the CE and that it can be calcu-

lated by the correlation sum directly without any statistical

methods. It is also robust to noise, i.e., the presence of addi-

tive Gaussian noise does not affect the estimation of the KS

entropy. Hence, the MCE is suitable for all chaotic time se-

ries regardless of whether they are clean or noisy.

The application of the method is demonstrated with four

chaotic time series in this study: two artificial and two real

world. The numerical results show that the MCE is a robust

estimation for the KS entropy. The KS entropy is also esti-

mated by using the Lyapunov spectrum, and numerical re-

sults for the four data sets used show that the MCE is closer

to the KS entropy than the CE.

The flow of this paper is arranged as follows. In Sec. II,

some definitions of various entropies are introduced. In Sec.

III, the correlation sum is introduced for a clean and a noisy

chaotic time series. In Sec. IV, a method to estimate the KS

entropy for a clean or a noisy chaotic time series is devel-

oped. In Sec. V, the estimation method for the Lyapunov

spectrum is introduced to estimate the KS entropy. In Sec.

VI, the proposed estimation method is applied to several data

sets, and the results and discussion are presented in Sec. VII.

II. ENTROPY OF A DYNAMICAL SYSTEM

There are several types of entropies defined in the litera-

ture. In the context of thermodynamics, entropy refers to the

amount of “disorder” in the system—the higher the entropy,

the higher the amount of disorder. For a closed thermody-

namic system, it is the measure of the amount of thermal

energy not available to do mechanical work. In statistical

mechanics, it refers to the amount of uncertainty in the

system. In information theory, it is a measure of the uncer-

tainty associated with a random variable. Shannon entropy

?Shannon, 1948? refers to a measure of the average informa-

tion content that is missing by not knowing the value of the

random variable. In statistical thermodynamics, it measures

the degree to which the probability of the system is spread

out over all possible substates.

In a time series, entropy is an important entity as its

inverse gives the time scale relevant for the predictability of

the system. It also provides the topological information about

the folding process. Several definitions of entropies can be

found in the literature ?e.g., Kantz and Schreiber ?2003??, and

some of them are briefly given here.

In information theory, the entropy of a random variable

is defined as

K?Z? = −?

i

piln pi,

?1?

where piis the probability of the random variable Z taking a

specified value zi?i.e., Pr?Z=zi?=pi?. In thermodynamics, a

similar expression is

K?Z? = − kB?

i

piln pi,

?2?

where kBis the Boltzmann constant.

The order-q Renyi entropy is defined as ?Renyi, 1971?

Kq?P?? =

1

1 − qln?pi

q,

?3?

where piis the fraction of the measure contained in the par-

tition Pi??? of side length ??. The Shannon entropy, evalu-

ated by the l’Hospital rule for q=1, is

K1?P?? = −?

i

piln pi.

?4?

Block entropies of block size n are defined as

Kq?n,P?? =

1

1 − qln ?

i1,i2,...,im

pi1,i2,...,in

q

,

?5?

where pi1,i2,...,inare the joint probabilities that at an arbitrary

time n, the observable falls into the interval Ii1, at time

n+1, to the interval Ii2, and so on, where Itis the interval.

Then, the order-q entropies are

kq= supPlim

n→?

n→??Kq?n + 1,P?? − Kq?n,P???,

where the supremum supPindicates that one has to maximize

over all possible partitions P?and usually implies the limit

?→0.

In Eq. ?6?, k1?when q=1? is the KS entropy and k0

?when q=0? is the T-Entropy. KS entropy is also sometimes

called the metric entropy or simply Kolmogorov entropy. KS

entropy has a numerical value of zero for nonchaotic systems

and positive values for chaotic systems. It can also be

thought of as the additional information obtained by observ-

ing the state of the system at a certain time given a priori

knowledge of the entire past. Numerically, entropies are

computed for a finite order q, which in the limit as

q→? converges to the KS entropy. However, there are prob-

lems in estimating KS entropy by using Eq. ?6?, as it would

require box counting which would be difficult in a high di-

mensional phase space. There are also difficulties arising

from the limitations of the data set as it is nontrivial to obtain

the limit process as ?→0.

1

nKq?n,??

= supPlim

?6?

023104-2Jayawardena, Xu, and LiChaos 20, 023104 ?2010?

Page 3

III. CORRELATION SUM

Since it is difficult to calculate the probability density by

counting boxes, a substitute method, which is called the rela-

tive integral method, is usually employed to estimate the

dynamic entropy for a chaotic time series. Considering a

chaotic time series ?xn?, 1?n?Ns, where Nsis the length of

the time series embedded into a delay phase space with em-

bedding dimension m and time delay ?, a point in the recon-

structed phase space can be denoted by

Xi= ?xi,xi+?, ... ,xi+?m−1???.

?7?

The correlation sum of the time series is then given by

Cm?r? =

1

N?N − 1??

i=1

N

?

j?i

N

H?r − ?Xi− Xj??,

?8?

where Xiand Xjare two different points in the reconstructed

phase space, N is the number of sample points in the recon-

structed phase space, and H?x? is the Heaviside function

H?x? =?

0,

x ? 0

1, x ? 0.?

In this study, ?•? is the Euclidean norm.

The correlation sum of a chaotic time series is a very

important entity. It can be used to estimate many related

parameters. It is therefore necessary to find a “good”

description of the correlation sum in order to estimate such

parameters accurately. For a noise-free chaotic time series,

Frank et al. ?1993? proved that the correlation sum behaves

as

Cm?r? = ??exp?− m?K???r/?m?Dfor r → 0,m → ?, ?9?

where ? is a constant, D is the correlation dimension, K is

the KS entropy ?order 2 entropy?, m is the embedding dimen-

sion, and ? is the time delay.

For a chaotic time series with additive Gaussian noise,

Diks ?1999? ?p. 123? and Oltmans and Verheijen ?1997?

proved that the correlation integral satisfies

Cm?r? =?e−m?Km−D/2?D−m2−mrm

??m/2 + 1?

?M?m − D

2

,m

2+ 1,−

r2

4?2?,

?10?

where ? is a constant ?same as in Eq. ?9??, ? is the Gaussian

noise level, and M?a,b,z? is the Kummer’s confluent

hypergeometric function, which has the following integral

representation:

M?a,b,z? =

??b?

??a???b − a??

0

1

eztta−1?1 − t?b−a−1dt,

?11?

where

a =m − D

2

,

b =m

2+ 1,

?12?

z = −

r2

4?2.

The Kummer’s confluent hypergeometric function also satis-

fies the following equation:

d

dzM?a,b,z? =a

bM?a + 1,b + 1,z?,

?13?

where

a + 1 =m − D

2

+ 1 =?m + 2? − D

2

and

b + 1 =m

2+ 2 =m + 2

2

+ 1.

The correlation dimension and KS entropy, together with the

noise level of the chaotic time series, can be estimated theo-

retically by applying Eqs. ?9? and ?10?. Diks ?1999? sug-

gested that for fixed embedding dimension, Eq. ?10? can be

applied to obtain a nonlinear maximum likelihood estimate

of the correlation dimension D and the noise level ?. Fur-

thermore, for large embedding dimensions, the estimation of

the KS entropy for noisy time series is given by ?Diks, 1999?

?ln?

K =1

Cm?r?

Cm+1?r??−1

?ln?

r

2?m??.

?14?

The estimation of the KS entropy by Eq. ?14? strongly de-

pends upon the estimation of the noise level, which is very

difficult for noisy chaotic time series. Hence, the use of Eq.

?14? cannot provide a better estimation of the KS entropy for

a noisy chaotic time series.

Equation ?14? suggests a relationship between the corre-

lation sums Cm?r? and Cm+2?r?, which will then be used to

estimate the KS entropy for a noisy chaotic time series.

IV. MODIFIED CORRELATION ENTROPY

By comparing the correlation sum ?Eq. ?9?? for two dif-

ferent values of the embedding dimension, m and m+2, we

have, when r?0,

Cm?r?

Cm+2?r?= exp?2?K??1 +2

m?

D/2

,

?15?

which can be written as

023104-3 Correlation entropy estimationChaos 20, 023104 ?2010?

Page 4

lnCm?r?

Cm+2?r?= 2?K + ln?1 +2

The CE K2as defined by Diks ?1999? ?p. 111? is

1

Cm?r?

m?

D/2

.

?16?

K2=

2?ln?

Cm+2?r??.

?17?

When the time series is noise-free, Eq. ?15? holds, and Eqs.

?16? and ?17? then simplify to a relationship between the CE

K2and the KS entropy K as

K2= K +

1

2?ln??1 +2

m?

D/2?,

?18?

which converges to K as m→?, i.e.,

limm→?K2= K.

For a noisy chaotic time series, by comparing the correlation

sum ?Eq. ?10?? for two different values of the embedding

dimension, m and m+2, we have

Cm?r?

Cm+2?r?=

m−D/2

e−2?K?m + 2?−D/22−2?−2r2

= exp?2?K??m + 2

??m/2 + 2?

??m/2 + 1?

M?a,b,z?

M?a + 1,b + 1,z?

= exp?2?K??m + 2

m?

D/22?2?m − D?M?a,b,z?

r2dM?a,b,z?/dzm?

D/2

2?2?m − D?

r2d ln?M?a,b,z??/dz,

?19?

where z=−r2/?2?2?, which has been defined in Eq. ?12?.

Hence, we obtain

exp?2?K??m + 2

m?

D/2

?m − D?

=

Cm?r?

Cm+2?r?

r2

2?2

d ln?M?a,b,z??

dz

.

?20?

Taking natural logarithms for both sides of Eq. ?10? gives

ln?Cm?r?? = ln??e−m?Km−D/22−m?D−m? + m ln?r?

− ln???m/2 + 1??

+ ln?M?m − D

2

,m

2+ 1,−

r2

4?2??.

?21?

Differentiating Eq. ?21? with respect to r, we obtain

dzln?M?m − D

??−

d

drln?Cm?r?? =m

r+d

2

,m

2+ 1,−

r2

4?2??

r

2?2?,

from which we obtain

r2

2?2

d

dzln?M?a,b,z?? = m − rd

drln?Cm?r??.

?22?

Substituting Eq. ?22? into Eq. ?20? gives

TABLE I. Statistics of data sets used in this study. Note: Under the noise level column, C means that the data

set is clean; A/B indicates that A is the added noise level and B is the actual noise level; R indicates that the data

set is raw and NR-1 indicates that the data set has been noise reduced by method 1, while NR-2 indicates that

the data set has been noise reduced by the method 2. ?The actual noise level ?2=1/N?n=1

x ¯nare the clean and noisy time series, respectively.?

N?xn−x ¯n?2, where xnand

DataLength Maximum MinimumMeanVariance Noise level

Lorenz A

Lorenz B

Lorenz C

Lorenz D

Rössler A

Rössler B

Rössler C

Rössler D

Mekong A

Mekong B

Mekong C

Chao A

Chao B

Chao C

5000

5000

5000

5000

5000

5000

5000

5000

4292

4292

4292

5844

5844

5844

18.02

18.21

19.27

20.33

17.29

17.98

18.97

20.17

?18

?19

?20.12

?21.19

?14.59

?15.28

?16.18

?17.1

891

951

903

71

232

95

0.3074

0.3091

0.3109

0.3127

0.13

0.132

0.134

0.136

7.8977

7.911

7.9561

8.0325

8.2

8.21

8.245

8.313

C

0.5/0.5020

1.0/1.004

1.5/1.5060

C

0.5/0.5020

1.0/1.004

1.5/1.5060

R

NR-1

NR-2

R

NR-1

NR-2

21 000

19 447

19 091

4320

4200

3927

4302

4297

4737

627.4

622

626.9

2624.8

2609

3177

470.5

440

647

023104-4Jayawardena, Xu, and Li Chaos 20, 023104 ?2010?

Page 5

exp?2?K??m + 2

Cm+2?r??m − rd

m?

D/2

?m − D?

drln?Cm?r???.

=

Cm?r?

?23a?

Taking natural logarithms for both sides of the above equa-

tion gives

2?K + ln?m + 2

m?

D/2

= ln?

Cm?r?

Cm+2?r??m − rd

− ln?m − D?.

drln?Cm?r????

?23b?

Note that the left-hand side of Eq. ?23b? is just the right-hand

side of Eq. ?16?. Since the right-hand side of Eq. ?23b? does

not depend on K, we can define a new CE for the noisy time

series by

K¯2=

1

2?ln?

2?ln??m −d ln?Cm?r??

Cm?r?

Cm+2?r??

+

1

d ln r???m − D??.

?24?

We call the CE estimated by Eq. ?24? the “MCE” for a cha-

otic time series. When the chaotic time series is noise-free,

by using Eq. ?9?, or using Eqs. ?10? and ?11? ?Jayawardena

et al., 2008?, it can be shown that

D =d ln?Cm?r??

d ln?r?

.

?25?

Equation ?24? ensures that when the chaotic time series is

noise-free, the MCE and the CE ?Eq. ?17?? are equivalent,

i.e.,

FIG. 1. ?Color online? CE and MCE vs radius r for Lorenz data ?embedding

dimension m=20, time delay ?=1 for 0.1?r?0.3; ? indicates the noise

level?.

FIG. 2. ?Color online? CE and MCE vs radius r for Rössler data ?embedding

dimension m=20, time delay ?=1 for 0.1?r?0.3; ? indicates the noise

level?.

FIG. 3. ?Color online? CE and MCE vs radius r for Mekong data ?embed-

ding dimension m=20, time delay ?=1 for 0.1?r?0.3; NR-1: noise re-

duced by method 1, NR-2: noise reduced by method 2?.

FIG. 4. ?Color online? CE and MCE vs radius r for Chao Phraya data

?embedding dimension m=20, time delay ?=1 for 0.1?r?0.3; NR-1: noise

reduced by method 1, NR-2: noise reduced by method 2?.

023104-5 Correlation entropy estimation Chaos 20, 023104 ?2010?

Page 6

K¯2= K2.

?26?

The estimation of the MCE by Eq. ?24? needs the correlation

dimension of the chaotic time series, which can be given

a priori, or be estimated using the correlation sum. In the

former case it can be given approximately as

D ? md− 1,

?27?

where mdis the minimum embedding dimension of the cha-

otic time series, which can be obtained by the false nearest

neighbor ?FNN? method ?Kantz and Schreiber, 2003?. In the

latter case, it can be estimated by employing the correlation

sum using Eq. ?8?. Given several values r1,r2,...,rL, the

corresponding correlation sums are then obtained as

Cm?r1?,Cm?r2?,...,Cm?rL?. Then the correlation dimension is

approximated by

D ?1

L?

i=1

L

d ln?Cm?ri??

d ln?ri?

.

?28?

In this study, the first method ?Eq. ?27?? is used.

Note that the estimation process for the MCE does not

involve the noise level of the noisy chaotic time series.

Hence, the above estimation method would be easier for a

noisy chaotic time series than the estimation method based

on Eq. ?14?. Furthermore, the MCE is more robust to noise

than the general CE obtained by Eq. ?18?.

V. KS ENTROPY AND THE LYAPUNOV SPECTRUM

In a dynamical system, the Lyapunov exponent charac-

terizes the rate of separation of initially nearby trajectories.

The rate of separation can be different for different initial

separations, thus leading to a whole spectrum of Lyapunov

exponents, with the number of such exponents being equal to

the dimension of the phase space. The largest of them deter-

mines the predictability of the system and a positive largest

exponent gives an indication that the system is chaotic. For

an evolutionary system, the spectrum of Lyapunov exponents

??1,?2,...,?n? depends upon the starting point x0and is de-

fined from the Jacobian matrix as follows:

?j= lim

N→?

1

N?J?F?N??x0??ej?,

?29?

(b) (a)

(c) (d)

FIG. 5. ?Color online? CE, MCE, and LE vs embedding dimension for Lorenz data with four different noise levels ?a for clean data; b for ?=0.5; c for ?

=1.0; and d for ?=1.5?.

023104-6 Jayawardena, Xu, and Li Chaos 20, 023104 ?2010?

Page 7

J?F?N??x0?? =?dF?N??x?

dx?

x0

,

?30?

where J?F? is the Jacobian matrix of the nonlinear function

F, F?n?is the nth iteration of the nonlinear evolution function,

and ?ej?,j=1,2,...,m is a set of orthogonal basis of the

tangent space at x0. The evolution function relates the future

value to the present value according to the equation

xn+1= F?xn?,

?31?

where ?xn?n=1

space of the chaotic time series.

For a conservative system, the sum of all Lyapunov ex-

ponents is zero. For a dissipative system, it is negative. Ac-

cording to the theorem of Pesin ?1977?, the sum of all posi-

tive Lyapunov exponents gives an estimate of the KS

entropy. The Lyapunov spectrum estimated as described be-

low is used to estimate the KS entropy to compare with the

MCE.

N

is the trajectory in the reconstructed phase

The Jacobian of Eqs. ?29? and ?30?, by the chain rule,

can be written as

JF?N??x0? = ?JF?xN−1?? · ?JF?N−1??x0??

= ?JF?xN−1?? · ?JF?xN−2?? · ?JF?N−2??x0?? = ¯

= ?JF?xN−1?? · ?JF?xN−2?? ¯ ?JF?x1?? · ?JF?x0??

= An· An−1¯ A2· A1,

?32?

where An=JF?xn−1?,1?n?N. Using the QR decomposition,

matrix A can be decomposed into two matrices: an orthogo-

nal matrix Q and an upper triangular matrix R. We define

matrices Qn,Rnvia

An+1Qn= Qn+1Rn+1,

?33?

where Q0=I, the identity matrix. Then, the Lyapunov spec-

trum of the system is given by

(b)(a)

(c)(d)

FIG. 6. ?Color online? CE, MCE, and LE vs embedding dimension for Rössler data with four different noise levels ?a for clean data; b for ?=0.5; c for ?

=1.0; and d for ?=1.5?.

023104-7 Correlation entropy estimation Chaos 20, 023104 ?2010?

Page 8

?j= lim

N→?

1

N?

n=1

N

ln??Rn?j,j???,

?34?

where Rn?j,j? is the element of matrix Rnat jth row and jth

column. The Jacobian matrix at each time point in the trajec-

tory of the nonlinear system is evaluated by employing the

least-squares method to some nearest neighbors of the corre-

sponding points in the reconstructed phase space. The above

procedure follows that developed by Brown et al. ?1991?.

VI. APPLICATION

The proposed method is then applied to four data sets to

estimate the MCE: two artificial chaotic time series and two

real-world chaotic time series. The artificial chaotic times

series are generated by the Lorenz and Rössler equations.

The Lorenz map is ?Lorenz, 1963? defined by the equations

?

z ˙=

x ˙=

y ˙= − xz + rx − y

??y − x?

xy − bz,?

?35?

and it becomes chaotic for ?=10, r=28, and b=8/3. Equa-

tion ?35? is a three dimensional differential system, which

can be solved numerically using the fourth order Runge–

Kutta method with a time step of 0.005 and initial conditions

of

x?0? = 12.5,y?0? = 2.5,z?0? = 1.5.

The values of the x-coordinate are then recorded as the real-

ized time series. To ensure that the values of the time series

are in the chaotic attractor, the first 4000 values are dis-

carded. Then, a time series of length 5000 is generated for

Eq. ?35? by

xn= x?100+ 0.2n?,

Gaussian noise with noise levels ??? 0.5, 1.0, and 1.5 are

then added to the series.

The Rössler system is defined by the equations ?Rössler,

1976?

1 ? n ? 5000.

(b)(a)

(c)

FIG. 7. ?Color online? CE, MCE, and LE vs embedding dimension for Mekong data with three different noise levels ?a for raw data; b for noise reduced by

method 1; c for noise reduced by method 2?.

023104-8 Jayawardena, Xu, and LiChaos 20, 023104 ?2010?

Page 9

?

x ˙=

y ˙=

− y − z

x + ay

z ˙= b − z?x − c?,?

and it becomes chaotic for a=0.15, b=0.2, and c=10. Equa-

tion ?36? is also a three dimensional differential system

which can be solved numerically using the fourth order

Runge–Kutta method with a time step of 0.02 and initial

conditions

?36?

x?0? = 0.1,y?0? = 0.1,z?0? = 0.1.

The values of the x-coordinate are then recorded as the real-

ized time series. To ensure that the values of the time series

are in the chaotic attractor, the first 5000 points are dis-

carded. Then, a time series of length 5000 is generated for

Eq. ?36? by

xn= x?100+ 0.5n?,1 ? n ? 5000.

Gaussian noise with noise levels ??? 0.5, 1.0, and 1.5 are

then added to the series.

The two real-world time series come from the field of

hydrology. They are the flow measurements made at two

gauging stations across two major rivers in Asia, namely, the

Mekong, which is a transboundary river that runs through six

countries ?China, Myanmar, Lao, Thailand, Cambodia, and

Vietnam?, and the Chao Phraya that runs through Thailand.

For Mekong, daily discharges measured at Nong Khai ?loca-

tion: 17.87° North, 102.72° East; basin area: 302 000 km2;

GRDC Reference No. 2969090? for the period of April

1980–December 1991, and for Chao Phraya, daily discharges

measured at Nakhon Sawan ?location: 15.67° North, 100.2°

East; basin area: 110 569 km2; GRDC Reference No.

2964100? for the period of April 1978–March 1994 were

used in the study.

Two noise reduction methods, described elsewhere

?Jayawardena and Gurung, 2000?, are applied to the two raw

data sets. The first noise removal method ?noise reduced by

method 1? follows that of Grassberger et al. ?1993?, while the

second method ?noise reduced by method 2? follows that of

(b) (a)

(c)

FIG. 8. ?Color online? CE, MCE, and LE vs embedding dimension for Chao Phraya data with three different noise levels ?a for raw data; b for noise reduced

by method 1; c for noise reduced by method 2?.

023104-9 Correlation entropy estimationChaos 20, 023104 ?2010?

Page 10

Schreiber and Grassberger ?1991?. Some statistics of these

four data sets are given in Table I. However, for convenience

of selecting the correlation radius r in calculating the corre-

lation sum using Eq. ?8?, all of the above data sets have been

rescaled into the interval ?0,1?.

VII. RESULTS AND DISCUSSION

In this study, a new approximating quantity for the KS

entropy, the MCE, is introduced for a chaotic time series.

This quantity is more robust to noise than the general CE and

is estimated by employing the correlation sum. Hence, the

estimation process strongly depends on the calculation of the

correlation sum. The correlation sum depends on several pa-

rameters: the amount of the sample points N, the embedding

dimension m, the time delay ?, and the correlation radius r.

The effects of these parameters on the estimation of the cor-

relation sum are discussed by Ramsey and Yuan ?1990?, and

they note that the sample size of the chaotic time series

should not be smaller than 2000. However, they also pointed

out that when the embedding dimension is large, the sample

size should also be large to ensure that the correlation sum

results in a small bias. In this study, 5000 sample points each

for Lorenz and Rössler data, 4292 sample points for the

Mekong data, and 5844 sample points for the Chao Phraya

data were used to estimate their respective correlation sums.

The embedding dimension for the Lorenz data and the

Rössler data is 3, since they are generated from a three di-

mensional chaotic attractor. Based on a previous study

?Jayawardena et al., 2002? that uses the false nearest neigh-

borhood method, each of the two real-world data sets has

embedding dimension of 3. By Takens’ embedding theorem

?Takens, 1981?, the time delay for a chaotic time series can

be chosen arbitrarily. In this study, it is set as 1.

The correlation radius r is also important for the corre-

lation sum calculation. Too small values of r would lead to

no neighbors for a point while a large value of r would miss

the microstructure of the chaotic attractor. In this study,

the correlation sums were calculated for 0.1?r?0.3 using

rescaled data.

The numerical results of the comparison between the CE

and the MCE, both calculated by using correlation sums, for

all the data sets used in this study are shown in Figs. 1–4. In

the simulation, the embedding dimension is 20 and the time

delay is 1 for all the data sets. Figure 1 shows the results for

the Lorenz data for four noise levels: clean data and noisy

data with noise levels 0.5, 1.0, and 1.5. They show that the

curves for the MCE and the CE are almost identical for the

clean Lorenz data set. For noisy data, the curves of the MCE

are still close to the curve for the clean data, while the curves

of the CE are far apart from the curve of clean data espe-

cially for small values of r. These results demonstrate that

the MCE is more robust to noise than the CE. A similar

conclusion can be made for the Rössler data as well, which

are shown in Fig. 2. The results for the Mekong and Chao

Phraya data, which are shown in Figs. 3 and 4, also indicate

the same pattern. It is also seen that the MCE is more stable

and that it varies in a smaller range compared with the CE.

Equations ?18? and ?24? show that the MCE and the CE

converge to the KS entropy of the system when the embed-

ding dimension is large enough. For comparison, the KS en-

tropy is also estimated by the Lyapunov spectrum method, as

described in Sec. V. This is referred to as Lyapunov entropy

in this context. Figures 5 and 6 show the variation in the

different entropies ?CE, MCE, and Lyapunov entropy ?LE??

as functions of the embedding dimension for different noise

levels for the Lorenz and Rössler data. The corresponding

results for the Mekong and Chao Phraya data are shown in

Figs. 7 and 8. All the four data sets show that the MCEs are

closer to the LE than the CE especially for large value of the

embedding dimension. The three sets of graphs for the

Mekong and Chao Phraya data correspond to raw data, noise

reduced-1, and noise reduced-2.

VIII. CONCLUSION

In this study, a new method of estimating the KS entropy

for a chaotic time series is presented. It is applicable to clean

as well as noisy data. Both theoretical analysis and numerical

results, as demonstrated by the application of the method to

Lorenz data, Rössler data, Mekong data, and Chao Phraya

data, show that the MCE is more robust to noise than the CE.

The numerical results show that the MCE is closer to the KS

entropy for both noisy and noise-free time series data.

APPENDIX A: ALGORITHM 1 „FOR MODIFIED

CORRELATION DIMENSION…

?1? Re-scale the time series ?xn?n=1

time series, ?x ¯n?n=1

?2? Estimate the minimum embedding dimension mdfor the

re-scaled time series by FNN method.

?3? Given an embedding dimension m ?m?md? and a time

delay ?, embed the chaotic time series into the delay phase

space. Points in the delay phase space are

?Xn?,1?n?N−?m−1??.

?4? Give several values of r, 0.1=r1?r2?¯ri?¯rL=0.3,

calculate the correlation sum Cm?ri? for the time series by

Eq. ?8?.

?5? Increase the embedding dimension to m+2, embed the time

series ?x ¯n?n=1

into a new delay phase space. Points in the

new delay phase space are ?Yn?,1?n?N−?m+2−1??.

?6? Give several values of r, 0.1=r1?r2?¯ri?¯rL=0.3,

calculate the correlation sum Cm+2?ri? for the time series by

Eq. ?8?,

?7? Calculate d ln?Cm?r??/d ln?r? by

d ln?Cm?ri??

d ln?ri?

ln?ri? − ln?ri−1?

N

to ?0, 1?, and obtain a new

N.

N

=ln?Cm?ri?? − ln?Cm?ri−1??

,1 ? i ? L

?8? Calculate

K2?i? =

1

2?lnCm+2?ri?

Cm?ri?,1 ? i ? L

?9? Calculate

K¯2?i? =

1

2?lnCm+2?ri?

Cm?ri?

+

1

2?ln?m − d ln?Cm?ri??/d ln?ri?

m − md+ 1

?,

1 ? i ? L

023104-10 Jayawardena, Xu, and LiChaos 20, 023104 ?2010?

Page 11

APPENDIX B: ALGORITHM 2 „FOR LYAPUNOV

SPECTRUM…

?1? Given a trajectory ?xn?,1?n?N.

?2? Let Q0=I, identity matrix.

?3? For every xn, find its NB?assumed to be 20 for all data

sets? nearest neighbors in the reconstructed phase space,

?xn

?4? Estimate parameters for local linear model

m

k?, 1?k?NB.

xn+m?=a+?

i=1

bixn+?i−1??

by the least square method using the NBnearest neighbors.

?5? Set Matrix

An=?

0

0

0

]

1

0

0

]

0

1

0

]

¯

¯

¯

?

0

0

0

]

b1 b2 b3 ¯ bm?

?6? QR decomposition: QkRk=AkQk−1

?7? ?j=1/N?ln?Rn?j,j?,?

j=1,2,.....N

Adler, R. L., Konheim, A. G., and McAndrew, M. H., “Topological en-

tropy,” Trans. Am. Math. Soc. 114, 309–319 ?1965?.

Bonachela, J. A., Hinrichsen, H., and Munoz, M., “Entropy estimations of

small data set,” J. Phys. A: Math. Theor. 41, 202001 ?2008?.

Brown, R., Bryant, P., and Abarbanel, H. D., “Computing the Lyapunov

spectrum of a dynamical system from an observed time series,” Phys.

Rev. A 43, 2787–2806 ?1991?.

Diks, C., Nonlinear Time Series Analysis, Methods and Applications ?World

Scientific, Singapore, 1999?.

Frank, M., Blank, H.-R., Heindl, J., Kaltenhauser, M., Kochner, H.,

Kreissche, W., Muller, N., Pocher, S., Sporer, R., and Wagner, T.,

“Improvement of K2entropy calculations by means of dimension scaled

distances,” Physica D 65, 359–364 ?1993?.

Grassberger, P., Hegger, R., Kantz, H., Schaffrath, C., and Schreiber, T.,

“On noise reduction methods for chaotic data,” Chaos 3, 127–141

?1993?.

Grassberger, P. and Procaccia, I., “Estimation of the Kolmogorov entropy

from a chaotic signal,” Phys. Rev. A 28, 2591–2593 ?1983?.

Jayawardena, A. W. and Gurung, A. B., “Noise reduction and prediction of

hydrometeorological time series: Dynamical systems approach vs. sto-

chastic approach,” J. Hydrol. 228, 242 ?2000?.

Jayawardena, A. W., Li, W. K., and Xu, P., “Neighbourhood selection for

local modeling and prediction of hydrological time series,” J. Hydrol.

258, 40–57 ?2002?.

Jayawardena, A. W., Xu, P. C., and Li, W. K., “A method of estimating the

noise level in a chaotic time series,” Chaos 18, 023115 ?2008?.

Kantz, H. and Schreiber, T., Nonlinear Time Series Analysis ?Cambridge

University Press, Cambridge, England, 2003?.

Kantz, H. and Schürmann, T., “Enlarger scaling ranges for the KS-entropy

and information dimension,” Chaos 6, 167–171 ?1996?.

Kolmogorov, A. N., “A new invariant of transitive dynamical systems,”

Dokl. Akad. Nauk SSSR 119, 861–864 ?1958?.

Lorenz, E. N., “Deterministic non-periodic flow,” J. Atmos. Sci. 20, 130–

141 ?1963?.

Oltmans, H. and Verheijen, P. J. T., “Influence of noise on power-law scaling

functions and an algorithm for dimension estimations,” Phys. Rev. E 56,

1160–1170 ?1997?.

Pesin, Y. B., “Characteristic Lyapunov exponents and smooth ergodic

theory,” Russ. Math. Surveys 32, 55–112 ?1977?.

Ramsey, J. B. and Yuan, H. J., “The statistical properties of dimension

calculations using small data sets,” Nonlinearity 3, 155–176 ?1990?.

Renyi, A., Probability Theory ?North-Holland, Amsterdam, 1971?.

Rössler, O. E., “An equation for continuous chaos,” Phys. Lett. 57A, 397–

398 ?1976?.

Schreiber, T. and Grassberger, P., “A simple noise-reduction method for real

data,” Phys. Lett. A 160, 411–418 ?1991?.

Schürmann, T. and Grassberger, P., “Entropy estimation of symbol se-

quences,” Chaos 6, 414–427 ?1996?.

Shannon, C. E., “A mathematical theory of communication,” Bell Syst.

Tech. J. 27, 379–423 and 623–656 ?1948?.

Sinai, A. G., “On the concept of entropy of a dynamical system,” Dokl.

Akad. Nauk SSSR 142, 768–776 ?1959?.

Takens, F., “Detecting strange attractors in turbulence,” Lect. Notes Math.

898, 366–381 ?1981?.

Weaver, W. and Shannon, C. E., The Mathematical Theory of Communica-

tion ?University of Illinois Press, Urbana, IL, 1949?.

023104-11Correlation entropy estimation Chaos 20, 023104 ?2010?