Page 1

AN IMPROVED PROPORTIONATE NLMS ALGORITHM BASED ON THE l0 NORM

Constantin Paleologu1, Jacob Benesty2, and Silviu Ciochin?1

1: University Politehnica of Bucharest, Telecommunications Department

Iuliu Maniu Blvd., 1-3, Bucharest, Romania

e-mail: {pale, silviu}@comm.pub.ro

2: Universite du Quebec, INRS-EMT, Montreal, QC H5A 1K6, Canada

e-mail: benesty@emt.inrs.ca

ABSTRACT

The proportionate normalized least-mean-square (PNLMS)

algorithm was developed in the context of network echo

cancellation. It has been proven to be efficient when the

echo path is sparse, which is not always the case in real-

world echo cancellation. The improved PNLMS (IPNLMS)

algorithm is less sensitive to the sparseness character of the

echo path. This algorithm uses the l1 norm to exploit

sparseness of the impulse response that needs to be

identified. In this paper, we propose an IPNLMS algorithm

based on the l0 norm, which represents a better measure of

sparseness than the l1 norm. Simulation results prove that

the proposed algorithm outperforms the original IPNLMS

algorithm.

Index Terms— Echo cancellation, adaptive filters,

proportionate normalized least-mean-square (PNLMS)

algorithm, sparseness.

1. INTRODUCTION

Echo cancellation is one of the most popular applications of

adaptive filtering [1]. The basic approach is to build a model

of the echo path impulse response using an adaptive filter.

Theoretically, various kinds of adaptive algorithms are

applicable for echo cancellation [2]. However, for many

good reasons, the normalized least-mean-square (NLMS)

algorithm is frequently the algorithm of choice in this

important application.

In order to enhance the performance of the NLMS

algorithm for network echo cancellation, a proportionate

NLMS (PNLMS) algorithm has been proposed in [3]. It

takes advantage of the fact that the network echo path is

sparse in nature, i.e., only a small portion of the coefficients

is significantly different from zero (active coefficients). The

idea behind the PNLMS algorithm is to update each

coefficient of the filter independently of the others by

adjusting the adaptation step-size in proportion to the

magnitude of the estimated filter coefficient. It redistributes

the adaptation gains among all coefficients and emphasizes

the large ones in order to speed up their convergence,

achieving a fast initial convergence rate. Unfortunately,

after this initial phase, the convergence rate of the PNLMS

algorithm slows down significantly, even becoming slower

than NLMS. In order to deal with this problem, several

versions of the PNLMS algorithm were proposed (e.g., see

[4] and the references therein). Most of them still assume

the sparse character of the echo path impulse response.

However, in real-world echo cancellation scenarios, it is

difficult to a priori know the sparseness “degree” of the

echo path. Motivated by this consideration, an improved

PNLMS (IPNLMS) algorithm was proposed in [5]. This

algorithm better takes advantage of the “proportionate”

idea, by using the l1 norm to exploit sparseness of the

impulse response that needs to be identified. Consequently,

it behaves well even in the case when the echo path is non-

sparse or quasi-sparse (e.g., like in acoustic echo

cancellation).

The motivation behind this work is that the l0 norm can

represent an even better measure of sparseness than the l1

norm [6]. Thus, we propose an IPNLMS algorithm based on

the l0 norm. The remainder of the paper is organized as

follows. Section 2 presents the backgrounds of the PNLMS-

based algorithms. The proposed IPNLMS algorithm based

on the l0 norm is derived in Section 3. Experimental results

are shown in Section 4. Finally, Section 5 concludes this

work.

2. PNLMS ALGORITHMS BACKGROUND

Let us consider an echo canceller configuration, where we

try to model an echo path with impulse response h = [h0,

h1,…, hL–1]T using an adaptive filter, ?(n) = [?0(n), ?1(n),…,

?L–1(n)]T. Superscript T denotes transposition, L is the length

of the systems, and n is the time index. The desired signal

for the adaptive filter is y(n) = hTx(n) + v(n), where x(n) =

[x(n), x(n–1),…, x(n–L+1)]T is a real-valued vector

containing the L most recent samples of the input signal

(i.e., far-end signal) and v(n) is the near-end signal.

The proportionate-type

summarized by the following equations:

NLMS algorithms are

This work was supported by the UEFISCSU Romania under Grants PN-

II-PCE-“Idei” no. 65/01.10.2007 and no. 331/01.10.2007.

309 978-1-4244-4296-6/10/$25.00 ©2010 IEEEICASSP 2010

Page 2

, (1)

? ?

e n

? ?

y n

?? ? ?

1

x

ˆ

h

T

nn

???

? ?

n

??

?? ? ? ? ?

? ? ?

nn

G

? ? ?

1

x

1

ˆ

h

ˆ

h

1

T

nn e n

?

n

n

?

?

?

?

???

G

x

x

, (2)

where ? is the normalized step-size parameter and ? is the

regularization. The main idea behind these algorithms is to

assign an individual step-size to each filter coefficient, using

the diagonal matrix G(n), in such a way that a larger

coefficient receives a larger increment, thus increasing the

convergence rate of that coefficient. Consequently, the

active coefficients are adjusted faster than non-active

coefficients (i.e., small or zero coefficients), so that this type

of algorithms converge faster than NLMS especially for

sparse impulse responses (i.e., when only a small percentage

of coefficients is significant). Regarding (2), it can be

noticed that the NLMS algorithm is obtained when G(n) = I,

where I is the identity matrix.

In the case of the original PNLMS algorithm [3], the

diagonal elements of G(n), denoted here by gl(n), with

0 ? l ? L – 1, are calculated using the following procedure:

? ?? ?

0

maxmax,

h n

???

?

?

? ?

1

l

L

g n

?

?

Parameters ? and ? are positive numbers with typical values

? = 5/L and ? = 0.01. The constant ? prevents the very small

coefficients from stalling and the parameter ? regularizes the

updating when all coefficients are zero at initialization. The

main limitation of the PNLMS algorithm is that it assigns

too much adaptation gains to large coefficients to speed up

their convergence at the cost of small coefficients. Thus,

after an initial fast convergence phase, this algorithm slows

down and becomes even slower than NLMS.

The IPNLMS algorithm proposed in [5] uses a

smoother choice as compared to (3), i.e.,

? ?

1

l

n

L

where –1 ? ? < 1 and

? ?

1

0l

?

is the l1 norm of the vector ?(n). Therefore, substituting (5)

into (4) and taking (6) into account, we find that

need to identify. A better measure can be the l0 norm, since

it is the natural mathematical measure of sparsity [7]. The l0

norm of a vector is equal to the number of its non-zero

components.

Let us consider the vector

the function

? ?

0,

?

the l0 norm of the vector a is

L

?

??

a

? ?

n

? ?

?

1

ˆˆ

h

ˆ

,,,

ln

?

?

?

?

Llh n

??

, (3)

? ?

? ?

n

0

l

i

i

n

?

?

?

?

. (4)

? ? ????? ?

1

ˆ

h

ˆ

h

1

n

???????

ln , (5)

? ?

n

1

ˆ

ˆ

h

L

l

nh

?

??

(6)

? ???

? ?

? ?

1

0

ˆ

h n

1

1

2

ˆ

2

l

l

L

?

i

i

g n

L

h n

?

?

?

?

?

?

???

?

. (7)

For practical reasons, a very small positive constant ? was

added to the denominator of the second term of (7), in order

to avoid division by zero (especially at the beginning of the

adaptation when all the filter taps are initialized to zero). It

can be checked that for ? = –1 the IPNLMS and NLMS

algorithms are equivalent, while for ? close to 1 the

IPNLMS algorithm behaves like the PNLMS algorithm. In

practice, good choices for the parameter ? are 0 or –0.5.

Also, it is very important to correctly choose the

regularization parameters of the PNLMS-type algorithms. It

was explained in [5] that the relation between the

regularization parameters of the discussed algorithms is

?IPNLMS = ?PNLMS(1–?)/2 = ?NLMS(1–?)/2L. The IPNLMS

algorithm performs better than PNLMS for both sparse and

non-sparse impulse responses.

3. IPNLMS ALGORITHM BASED ON l0 NORM

It can be noticed that the IPNLMS algorithm uses the l1

norm to exploit the sparseness of the impulse response we

. Defining

??

011

,,,

T

L

a aa?

?

a

?

(8)

1, 0,

0

l

l

l

a

a

f a

?

?

?

??

? ?

f a

1

0

0

l

l

?

. (9)

The function f(al) is not continuous and the fact that many

elements of the vector a can be very small but not exactly

equal to zero, it is better to approximate it by a smooth and

continuous function. A good approximation is [6]

? ?

l

f a

? ?

where ? is a large positive value. Therefore,

1

lim1

l

?

Now we can use this norm to estimate the elements of G(n).

Following the idea of the IPNLMS algorithm, we have

? ?

0

1

l

n

L

Next, substituting (12) into (4) and taking (11) into account,

we find that

1,

la

e

??

(10)

1

0

00

1

ll

L

?

L

?

aa

l

e

?

?

??

?

???

?

e

?

?

??

?

?

?

????

?

?.

a

(11)

? ? ????

? ?

ˆ

ˆ

h

11

lh n

n

e

?

???

?

?

??

?????

?

?

?. (12)

310

Page 3

? ???

? ?

? ?

ˆ

h n

1

ˆ

0

11

?

?

?

1

2

21

l

i

l

L

?

h n

i

e

g n

L

e

?

?

?

?

?

?

?

?

?

??

???

?

?

?

??

. (13)

The very small positive constant ? plays the same role as in

(7), i.e., to avoid division by zero. Using (13) instead of (7)

we obtain the IPNLMS algorithm based on the l0 norm.

Finally, some practical issues should be outlined. First,

the choice of the parameter ? is important. In principle, it

should depend on the energy distribution of the impulse

response that needs to be identified; a dispersive system

could require a larger value for ?. Several considerations

regarding the choice of this parameter can be found in [6].

Also, we can obtain some “a priori” information about the

impulse response by using a regular IPNLMS algorithm in

the first N iterations (with N > L); then, having a rough

model of the echo path, we can evaluate its sparseness

degree using a certain measure, e.g.,

? ?

1N

? ? ?? ?

??

? ?

h N

0

1

L

ˆ

h

ˆ

(14)

and choose the value of ? accordingly. It is obvious that the

value of the parameter from (14) is between 0 and 1. Note

that the closer this measure is to 1, the sparser is the impulse

response. On the contrary, the closer the measure is to 0, the

denser or less sparse is the impulse response.

Second, the evaluation of the exponential term from

(13) could be problematic in practical implementations. A

possible solution is based on the first order Taylor series

expansions of exponential functions [6]. Another practical

option is to use a look-up table.

4. SIMULATION RESULTS

For the experiments, we have chosen two impulse

responses (Fig. 1) of length L = 512 (using an 8 kHz

sampling rate). The first one (Fig. 1a) is a network echo path

(sparse), according to ITU-T G.168 [8]; the second one (Fig.

1b) is a measured acoustic echo path (quasi-sparse). The

input signal, x(n), is either a white Gaussian noise or a

speech signal. An independent white Gaussian noise, v(n),

is added to the output of the echo path, resulting the desired

signal, y(n); the signal-to-noise ratio (SNR) is 30 dB.

It would be very interesting to compare the proposed

algorithm with most of the PNLMS-type algorithms but it is

beyond the scope of this paper. For this reason we choose to

compare the IPNLMS algorithm based on the l0 norm

(referred in the following as IPNLMS-l0) with the classical

PNLMS algorithm [3] and with the original IPNLMS

algorithm [5]. These two algorithms are the benchmarks to

which most PNLMS-type algorithms are compared.

Besides, it is important to notice the improvements of the

proposed IPNLMS-l0 as compared to the IPNLMS

algorithm, taking into account the improvements of the

IPNLMS as compared to the PNLMS.

050100150200250300350400450500

-0.1

0

0.1

0.2

Samples

(b)

(a)

0 50 100150200 250300 350400 450 500

-0.01

-0.005

0

0.005

0.01

Samples

Fig. 1. (a) Sparse network echo path according to ITU-T G.168;

(b) measured acoustic echo path (quasi-sparse).

The measure of performance is the normalized

misalignment (in dB), defined as 20log10(||h – ?(n)||2/||h||2),

where ||•||2 denotes the l2 norm; the results are averaged over

20 independent trials. The normalized step-size for all the

algorithms is ? = 0.2. The regularization parameter for the

PNLMS algorithm is ?PNLMS = 20?x

algorithms use ?IPNLMS = 20?x

signal variance). For the IPNLMS algorithms we fix ? = 0.

First, we evaluate the impact of the parameter ? on the

performance of the proposed IPNLMS-l0 algorithm. The

experiments are performed for both impulse responses from

Fig. 1; the results are presented in Fig. 2, using a white

Gaussian noise as input signal. In both cases it can be

noticed that the performance is improved when the value of

? increases, but up to a value which seems to be closer to

the number of active coefficients (i.e., ? = 50 in Fig. 2a and

? = 500 in Fig. 2b). This could be a motivation for using the

approach related to (14) in order to choose the value of ?.

Fig. 3 compares the performance of the PNLMS,

IPNLMS, and IPNLMS-l0 algorithms for both sparse and

quasi-sparse impulse responses; the input signal is a white

Gaussian noise. The IPNLMS-l0 algorithm uses ? = 50 in

Fig. 3a and ? = 500 in Fig. 3b. It can be noticed that the

proposed algorithm outperforms IPNLMS in both cases, but

the differences are more significant for the sparse impulse

response (Fig. 3a).

In Fig. 4, the tracking capabilities of the algorithms are

tested. The impulse responses from Fig. 1 are shifted to the

right by 12 samples. The other conditions are the same as in

the previous experiment (Fig. 3). According to this

simulation, the IPNLMS-l0 algorithm has a similar tracking

reaction as compared to the IPNLMS algorithm.

Finally, the speech input is used in the last experiment

(Fig. 5). The other conditions are the same as in Fig. 4. It is

clear that the IPNLMS-l0 algorithm outperforms the other

algorithms. In the case of the sparse impulse response (Fig.

5a), it is interesting to notice that the improvement of the

IPNLMS-l0 over the IPNLMS is similar with the

improvement of the IPNLMS over the PNLMS.

2/L, while the IPNLMS

2/2L (where

? is the input

2

x

311

Page 4

00.10.20.3 0.4

Time (seconds)

(b)

0.50.60.70.80.91

-40

-30

-20

-10

0

Misalignment (dB)

(a)

? = 10

? = 50

? = 100

? = 500

0 0.2 0.4 0.6 0.8

Time (seconds)

11.21.41.61.82

-40

-30

-20

-10

0

Misalignment (dB)

? = 50

? = 500

? = 2500

? = 5000

Fig. 2. Misalignment of the IPNLMS-l0 algorithm for different

values of the parameter ?. (a) Impulse response from Fig. 1a; (b)

Impulse response from Fig. 1b. The input signal is a white

Gaussian noise.

0 0.20.40.60.8

Time (seconds)

(b)

1 1.21.41.61.82

-40

-30

-20

-10

0

Misalignment (dB)

(a)

PNLMS

IPNLMS

IPNLMS-l0

0 0.20.40.6 0.8

Time (seconds)

11.21.41.6 1.82

-40

-30

-20

-10

0

Misalignment (dB)

PNLMS

IPNLMS

IPNLMS-l0

Fig. 3. Misalignment of the PNLMS, IPNLMS, and IPNLMS-l0

algorithms. (a) Impulse response from Fig. 1a, ? = 50; (b) Impulse

response from Fig. 1b, ? = 500. The input signal is a white

Gaussian noise.

0 0.51 1.52 2.53

-40

-30

-20

-10

0

Time (seconds)

(b)

Misalignment (dB)

(a)

PNLMS

IPNLMS

IPNLMS-l0

00.511.522.53

-40

-30

-20

-10

0

Time (seconds)

Misalignment (dB)

PNLMS

IPNLMS

IPNLMS-l0

Fig. 4. Misalignment of the PNLMS, IPNLMS, and IPNLMS-l0

algorithms. Echo path changes at time 1.5. Other conditions as in

Fig. 3.

0123456

-30

-20

-10

0

Time (seconds)

(b)

Misalignment (dB)

(a)

PNLMS

IPNLMS

IPNLMS-l0

0123456

-15

-10

-5

0

Time (seconds)

Misalignment (dB)

PNLMS

IPNLMS

IPNLMS-l0

Fig. 5. Misalignment of the PNLMS, IPNLMS, and IPNLMS-l0

algorithms. The input signal is speech. Echo path changes at time 3.

Other conditions as in Fig. 3.

5. CONCLUSIONS AND PERSPECTIVES

In this paper we proposed a new proportionate-type

NLMS algorithm. It is based on the IPNLMS algorithm but

it uses the l0 norm to exploit the sparseness of the system

that needs to be identified. Simulation results indicate that

the IPNLMS-l0 algorithm outperforms IPNLMS especially

for sparse impulse responses. The main challenge associated

with the IPNLMS-l0 algorithm is to find a practical way to

choose the value of the parameter ?. Several strategies for

this purpose were also discussed in the paper. This idea can

obviously be extended to the affine projection algorithm.

6. REFERENCES

[1] J. Benesty, T. Gaensler, D. R. Morgan, M. M. Sondhi, and S.

L. Gay, Advances in Network and Acoustic Echo Cancellation,

Springer-Verlag, Berlin, Germany, 2001.

[2] J. Benesty and Y. Huang, Eds., Adaptive Signal Processing–

Applications to Real-World Problems, Springer-Verlag, Berlin,

Germany, 2003.

[3] D. L. Duttweiler, “Proportionate normalized least-mean-

squares adaptation in echo cancelers”, IEEE Trans. Speech and

Audio Processing, vol. 8, pp. 508–518, Sept. 2000.

[4] H. Deng and M. Doroslova?ki, “Proportionate adaptive

algorithms for network echo cancellation,” IEEE Trans. Signal

Processing, vol. 54, pp. 1794–1803, May 2006.

[5] J. Benesty and S. L. Gay, “An improved PNLMS algorithm,”

Proc. IEEE ICASSP, 2002, pp. II-1881–II-1884.

[6] Y. Gu, J. Jin, and S. Mei, “l0 norm constraint LMS algorithm

for sparse system identification,” IEEE Signal Processing Letters,

vol. 16, pp. 774–777, Sept. 2009.

[7] D. Donoho, “Compressed sensing,” IEEE Trans. Information

Theory, vol. 52, no. 4, pp. 1289–1306, Apr. 2006.

[8] Digital Network Echo Cancellers, ITU-T Rec. G.168, 2002.

312