Page 1

STRUCTURED LEAST SQUARES WITH BOUNDED DATA UNCERTAINTIES

M. Pilanci1, O. Arikan1, B. Oguz2, M.C. Pinar3

1Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey

2Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA

3Department of Industrial Engineering, Bilkent University, Ankara, Turkey

ABSTRACT

In many signal processing applications the core problem re-

duces to a linear system of equations. Coefficient matrix un-

certainties create a significant challenge in obtaining reliable

solutions. In this paper, we present a novel formulation for

solving a system of noise contaminated linear equations while

preserving the structure of the coefficient matrix. The pro-

posed method has advantages over the known Structured To-

tal Least Squares (STLS) techniques in utilizing additional in-

formation about the uncertainties and robustness in ill-posed

problems. Numericalcomparisonsaregiventoillustratethese

advantages in two applications: signal restoration problem

with an uncertain model and frequency estimation of multi-

ple sinusoids embedded in white noise.

Index Terms— total least squares, robust solutions, in-

verse problems, structured perturbations, bounded data un-

certainties

1. INTRODUCTION

In various signal processing applications such as deconvolu-

tion, signal modeling, frequency estimation and system iden-

tification, it is important to produce robust estimates for an

unknown vector ˆ x from a set of measurements y. Typically,

a linear model is used to relate the unknowns to the available

measurements: y = Hx + w, where the matrix H ∈ Rm×n

describes the linear relationship and w is an additive noise

vector. There are many well known approaches to provide es-

timates ˆ x. For instance, if x is a random vector with known

first and second order statistics, the Wiener estimator, which

minimizes the mean-squared error (MSE) over all linear esti-

mators, is a reasonable choice. In the absence of such a statis-

ticalinformationonx, leastsquarestechniquesarecommonly

used.

In many applications the elements of matrix H are also

subject to errors since they are results of some other measure-

ments or an imprecise model. It has been shown that if the er-

rors in H and w are both independent identically distributed

Gaussian noise, the Maximum Likelihood (ML) estimate for

x is provided by the Total Least Squares (TLS) technique,

which ”corrects” the system with minimum perturbation so

that it is consistent [1]. However, in many applications H has

a certain structure, such as Toeplitz and Structured Total Least

Squares (STLS) techniques have been developed to perform

structured perturbations [2].

A major drawback of both the TLS and the STLS tech-

niques is that, in trying to reach to a consistent system, they

can produce unacceptably large perturbations on H and y.

Another significant problem of TLS arises in nonzero resid-

ualproblemsinwhichtheoriginalsystemisinconsistent, may

be due to lower order linear modeling or actual nonlinear re-

lationship between the unknowns and the measurement. In

these cases the TLS solution may be more sensitive than the

LS solution and it is necessary to relax the consistency con-

dition, and incorporate perturbation bounds [1]. For this pur-

pose, two alternative formulations have been proposed. In

Min-Max formulation, which is also referred to as Bounded

DataUncertainties(BDU)orRobustLeastSquares(RLS)[3],

ˆ x is chosen as a minimizer of the maximum error over the

set of allowed perturbations. In Min-Min formulation, which

is referred to as Bounded Errors-in-Variables Model [4], ˆ x is

chosen as a minimizer of the minimum error over the set of al-

lowed perturbations. Therefore, Min-Max approach provides

more conservative estimates than the estimates obtained by

the Min-Min approach.

In this paper, we formulate a new Min-Min type approach,

the Structured Least Squares with Bounded Data Uncertain-

ties (SLS-BDU), to overcome the sensitivity problems in

STLS methods. In the SLS-BDU approach the residual norm

?(H + ΔH)x − (y + Δy)? subject to bounded and struc-

tured perturbations is minimized with respect to x as well as

the perturbations ΔH and Δy. Hence, the consistency is

not forced, and the sensitivity of the solution is kept under

control with the perturbation bounds. Before proceeding with

the details of the proposed approach, we first present a review

on TLS, STLS, Min-Min and Min-Max approaches. Then, on

two different applications, we report results of a comparison

study. Finally, the drawn conclusions are presented.

3261 978-1-4244-2354-5/09/$25.00 ©2009 IEEEICASSP 2009

Page 2

2. REVIEW: TOTAL LEAST SQUARES AND THE

STRUCTURED TOTAL LEAST SQUARES

Giventheoverdeterminedlinearsystemofequations, Hx ≈y,

where both H and y may have imprecisions, TLS produces x

as the minimum norm solution to (H + ΔH)x = (y + Δy)

where [ΔHΔy] is chosen to be minimum norm perturbation

on the original system which results in a consistent system.

The TLS problem can be solved using the Singular Value

Decomposition (SVD) as [1]:

xTLS= (HTH − σ2

n+1I)−1HTy ,

(1)

where σn+1is the smallest singular value of [H y] and sub-

tracted to remove the bias introduced by the error in H. How-

ever, the subtraction of σ2

deregulates the inverse operation, hence results in sensitivity

issues.

In the Structured Total Least Squares (STLS) formulation

the problem becomes,

min

n+1I from the diagonal of HTH

ΔH,Δy,x?ΔH Δy?F, s.t.(H+ΔH)x = (y+Δy) and

[ΔH Δy] has the same structure as [A b] .

This problem is non-convex and known to be NP-hard

and developed solution methods are based on local optimiza-

tion. When the matrices are ill conditioned the solution has a

huge norm and variance since STLS introduce deregulariza-

tion similar to TLS.

3. REVIEW: MIN-MAX AND MIN-MIN

METHODOLOGY

3.1. Robust Least Squares

One of the Min-Max techniques is known as the Robust Least

Squares (RLS) which generates estimate to x as a solution to

the following optimization problem:

min

x

max

?[ΔH Δy]?F≤ρ?(H + ΔH)x − (y + Δy)? .

(2)

RLS minimizes the worst case residual over a set of pertur-

bations with bounded Frobenius norm. As the bound ρ gets

larger, the obtained solutions deviate more from the least

squares solution. Hence, the RLS approach trades accuracy

for robustness.

SRLS is the structured version of RLS with ΔH =

p?

lems can be obtained using convex, second-order cone pro-

gramming [3].

i=1δiHiand solutions to both the RLS and the SRLS prob-

3.2. Bounded Errors-in-Variables Model

One of the Min-Min techniques is known as the Bounded

Errors-in-Variables Model, where the inner maximization of

the RLS cost function is replaced with a minimization over

the allowed perturbations:

min

x

min

?[ΔH]?F≤ηH

?[Δy]?2≤ηy

?(H + ΔH)x − (y + Δy)? .

As opposed to the cautious approach in the Min-Max tech-

niques, thistechniquehasanoptimisticapproachandsearches

for the most favorable perturbation in the allowed set of per-

turbations. In this sense it is closer to the TLS technique, but

more robust since it does not pursue the consistency as in TLS

resulting in sensitivity issues. However, unlike the Min-Max

case, the Min-Min approach may be degenerate if the residual

becomes zero [4]. The nondegenerate and unstructured case

has the same form of the TLS solution

xMin−Min= (HTH − γI)−1HTy,

for some positive valued γ which depends on the perturbation

bounds. For small enough bounds on the perturbations, it can

be shown that the value of γ is less than that of σ2

TLS solution given in Eqn. 1. [4]. Thus, the deregularization

of the Min-Min solution is less than that of the TLS, resulting

in more robust solutions.

n+1in the

4. PROPOSED STRUCTURED LEAST SQUARES

WITH BOUNDED DATA UNCERTAINTIES

APPROACH

The SLS-BDU approach is a structured Min-Min approach,

that is developed to provide more robust solutions than the

STLS technique. Although the STLS utilizes structured per-

turbations, because it seeks consistency, the perturbations can

be unreasonably large even if a penalty on ?x? is added to

the objective. In many signal processing applications pertur-

bations beyond some bounds cannot be justified. Therefore

in our proposed approach, we want to consider perturbations

that are within a given tolerable bound only. The following

cases illustrate the need for the bounded perturbations:

1. The given linear equations may be inadequate to rep-

resent the observed phenomenon, e.g., wrong model,

nonlinear data, where seeking consistency of equations

is not appropriate.

2. Some elements of the matrix may be exactly known

or given with confidence intervals, e.g., econometric or

mechanical models.

3. Forcing the consistency in ill-posed problems may re-

sult a very sensitive estimator and the mean-squared er-

ror is not desirable as it will be shown in numerical

examples.

3262

Page 3

In SLS-BDU approach, we propose to use the following lin-

early structured version of the Bounded Errors-in-Variables

optimization:

?????(H +

Hiand yiwith αi’s determining the amount of perturbation.

The SLS-BDU formulation allows bounds defined over any

convex set. Here, for the sake of simplicity in the presenta-

tion, we only consider a weighted norm bound on the α with

a positive definite weighting matrix W.

The SLS-BDU optimization given in Eqn. 3. is noncon-

vex. However, as we will show next, an iterative algorithm

can be used to a find a local minimum of it. For this purpose,

first define:

min

x

min

?Wα?≤ρ

p

?

i=1

αiHi)x − (y +

p

?

i=1

αiyi)

?????

. (3)

Similar to the SRLS formulation, the structure is encoded to

H(α) = H+

p

?

i=1

αiHi,

y(α) = y+

p

?

i=1

αiyi,α = [α1...αp]T.

(4)

Then, simplify the SLS-BDU optimization given in Eqn. 3.

as:

min

x

min

?Wα?≤ρJ(x,α) ,

(5)

where J(x,α) is defined as ?H(α)x − y(α)?. For a fixed α,

minimization of J(x,α) with respect to x becomes a convex

ordinary least squares problem which can be solved easily.

Now we will show that for a fixed x minimization of J(x,α)

with respect to α is also a convex optimization problem.

min

?Wα?≤ρJ(x,α) =

where ?(x) = Hx − y, hi = Hix. Hence, for a fixed x

minimization of J(x,α) with respect to α becomes:

min

?Wα?≤ρ??(x) + [(h1− y1)...(hp− yp)]α?

min

?Wα?≤ρ

??(x) + Uα? ,

(6)

where U = [(h1− y1)...(hp− yp)]. This final form is a

Constrained Least Squares problem which can be solved by

using the method of Lagrange multipliers [5].

The above derived convexity results enables us to use the

following iterative optimization algorithm to converge to a lo-

cal minimum of the SLS-BDU optimization given in Eqn. 3.:

Step 1 Set ˆ α0= 0, and ˆ x0= (HTH)−1HTy, ˆ α0= 0.

Step 2 For k ≥ 1, by using the method of Lagrange multi-

pliers update ˆ αk+1as the solution to (6).

Step 3 Set ˆ xk+1 = (H(ˆ αk)TH(ˆ αk))−1HTy(ˆ αk) where

H(α) and y(α) are defined in Eqn.4.

Step 4 Repeat steps 2 and 3, until ?ˆ xk− ˆ xk−1? ≤ ε,

where ε is a user defined threshold of convergence. If

problems are encountered in evaluating ˆ xk, one can use

QR decomposition or Tikhonov regularization.

0510 15

time

2025 30

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

h[n]

Observed impulse response

True impulse response

Fig. 1. Nominal and actual impulse responses are shown in

solid and dashed lines respectively.

?b/btrue

0.20.6

?xtrue− xLS?/?xtrue?

?xtrue− xSLS−BDU?/?xtrue?

?Htrue− H?F/?Htrue?F

?Htrue− HSLS−BDU?F/?Htrue?F

Table 1. xtrue, xLS and xSLS−BDU correspond to actual

signal and estimates, Htrue, H, HSLS−BDU correspond to

actual, nominal and corrected matrices respectively.

0.0820

0.0274

0.1072

0.0655

0.2123

0.1279

0.2589

0.1284

5. NUMERICAL EXAMPLES

5.1. Signal Restoration with an Uncertain Kernel

Suppose that the observed signal is y[n] =

L−1

?

k=0

x[n−k]h[k]+

w[n] , n = 0,..N − 1 where

?

is the kernel of convolution with bounded data uncertainties

on amplitudes | δai |≤ ?aiand dampings | δbi |≤ ?bi, i =

1,...,Np. x[n] is the signal to be estimated and w[n] is white

Gaussian noise. The uncertainties in bi’s can be linearized by

a first order approximation, e−(bi+δbi)n≈ e−bin(1 − δbin) ,

and the uncertain matrix representation becomes,

h[n] =

Np

i=1

(ai+ δai)e−(bi+δbi)ncos(win + φi)

y = (H +

Np

?

i=1

αiHi)x + w ,

with the constraint ?Wα?∞≤ ?, where Hiare fixed Toeplitz

matrices.

Suppose that we observe the nominal impulse response

shown in Fig. 1. and have a priori bounds on the uncertainty.

Structured Least Squares with Bounded Data Uncertainties

3263

Page 4

051015 20 25

time

30 35 4045 50

−0.5

0

0.5

1

1.5

2

2.5

3

3.5

x[n]

True Signal

LS Estimate

SLS−BDU estimate

Fig. 2. Actual and restored signals are shown in dashed and

solid lines respectively.

min

i

0.8653

8.6991e-7

0.0187

Ei

max

i

1.0044

7.6497e+9

1.0889

Ei

mean( Ei)

LS0.9345

9.1162e+7

0.6396

STLS

SLS-BDU

Table 2. Minimum, Maximum and Mean Relative Errors for

LS, STLS and SLS-BDU

corrects the system in given perturbation bounds and restores

the original signal with better accuracy as shown in Fig. 2.

and Table 2. Note that if the uncertainty is not bounded as

in STLS, the approximation may not be valid and the corre-

sponding estimator is not desirable.

5.2. Frequency Estimation of Multiple Sinusoids

Linear prediction equations can be solved to estimate the pa-

rameters of multiple sinusoids and it is shown that STLS es-

timator corresponds to the ML estimator when noise is nor-

mally distributed [6]. Consider the case where parameters of

two sinusoids which are close in frequency need to be esti-

mated with frequencies f1= 0.32 Hz and f2= 0.30 Hz in

white noise wn:

x(n) = cos(2πf1n)+cos(2πf2n)+wn,n = 0,1,...,99.

Wesettheconstraintontheperturbationsas?α? ≤ δ such

that there exists an energy bound on the observed signal. The

relative estimation error Ei??xtrue−x[i]?

and the proposed SLS-BDU estimators are evaluated in inde-

pendent trials at 23 dB SNR and plotted in Fig. 3. As it can be

seen in Table 2 when the consistency condition is relaxed as

in SLS-BDU, the sensitivity problem of STLS is avoided sig-

nificantly without adding a regularization term and therefore

preserving details in the signals which can be resolved.

?xtrue?

of LS, STLS [2]

01020304050

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

trial number

relative error

STSL vs SLS−BDU

LS

STLS

SLS−BDU

Fig. 3. Relative Estimation Error of LS, STLS and SLS-BDU

in 50 independent trials. Frequently ?xSTLS? attains huge

values because of ill conditioning.

6. CONCLUSIONS

A new robust estimation technique is proposed for the solu-

tion of structured linear system of equations with bounded

data uncertainties. Numerical examples showed that the pro-

posed SLS-BDU technique achieves better mean-squared er-

ror and utilizes additional information about the uncertainties.

An iterative algorithm to compute the proposed estimator is

shown to be accurate and efficient. Our formulation can be

used to obtain robust and accurate results in many other sig-

nal processing applications, especially in commonly occur-

ring ill-posed problems with significant sensitivity issues.

7. REFERENCES

[1] S. Van Huffel and P. Lemmerling, “Total Least Squares

and Errors-in-Variables Modeling Analysis, Algorithms and

Applications“, Kluwer Academic, 2002.

[2] I. Markovsky, S. Van Huffel, and R. Pintelon, “Block-

Toeplitz/Hankel Structured Total Least Squares“, Tech. Rep.

03–135,2003.

[3] L. El-Ghaoui, H. Lebret. “Robust solutions to least-square

problems to uncertain data matrices“, SIAM J. Matrix Anal.

Appl. 18, 1035–1064 (1997).

[4] S. Chandrasekaran, M. Gu, A. H. Sayed, and K. E. Schu-

bert, “The degenerate bounded error-in-variables model“,

SIAM J. Matrix Anal. Appl., vol. 23, pp. 138–166, 2001.

[5] G. H. Golub and U. von Matt, “Quadratically constrained

least squares and quadratic problems“, Numer. Math., 59

(1991), pp. 561–580.

[6] T. Abatzoglou, J. Mendel, and G. Harada, “The con-

strained total least squares technique and its application to

harmonic superresolution“, IEEE Trans. Signal Process., 39

(1991), pp. 1070–1087.

3264