Structured least squares with bounded data uncertainties
ABSTRACT In many signal processing applications the core problem reduces to a linear system of equations. Coefficient matrix uncertainties create a significant challenge in obtaining reliable solutions. In this paper, we present a novel formulation for solving a system of noise contaminated linear equations while preserving the structure of the coefficient matrix. The proposed method has advantages over the known Structured Total Least Squares (STLS) techniques in utilizing additional information about the uncertainties and robustness in ill-posed problems. Numerical comparisons are given to illustrate these advantages in two applications: signal restoration problem with an uncertain model and frequency estimation of multiple sinusoids embedded in white noise.
- SourceAvailable from: Orhan Arikan
Conference Paper: Recovery of sparse perturbations in Least Squares problems.[Show abstract] [Hide abstract]
ABSTRACT: We show that the exact recovery of sparse perturbations on the coefficient matrix in overdetermined Least Squares problems is possible for a large class of perturbation structures. The well established theory of Compressed Sensing enables us to prove that if the perturbation structure is sufficiently incoherent, then exact or stable recovery can be achieved using linear programming. We derive sufficiency conditions for both exact and stable recovery using known results of � 0/� 1 equivalence. However the problem turns out to be more complicated than the usual setting used in various sparse reconstruction problems. We propose and solve an optimization criterion and its convex relaxation to recover the perturbation and the solution to the Least Squares problem simultaneously. Then we demonstrate with numerical examples that the proposed method is able to recover the perturbation and the unknown exactly with high probability. The performance of the proposed technique is compared in blind identification of sparse multipath channels.Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2011, May 22-27, 2011, Prague Congress Center, Prague, Czech Republic; 01/2011 · 4.63 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: A novel approach is proposed to provide robust and accurate estimates for linear regression problems when both the measurement vector and the coefficient matrix are structured and subject to errors or uncertainty. A new analytic formulation is developed in terms of the gradient flow of the residual norm to analyze and provide estimates to the regression. The presented analysis enables us to establish theoretical performance guarantees to compare with existing methods and also offers a criterion to choose the regularization parameter autonomously. Theoretical results and simulations in applications such as blind identification, multiple frequency estimation and deconvolution show that the proposed technique outperforms alternative methods in mean-squared error for a significant range of signal-to-noise ratio values.IEEE Transactions on Signal Processing 06/2010; · 2.81 Impact Factor
STRUCTURED LEAST SQUARES WITH BOUNDED DATA UNCERTAINTIES
M. Pilanci1, O. Arikan1, B. Oguz2, M.C. Pinar3
1Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey
2Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
3Department of Industrial Engineering, Bilkent University, Ankara, Turkey
In many signal processing applications the core problem re-
duces to a linear system of equations. Coefficient matrix un-
certainties create a significant challenge in obtaining reliable
solutions. In this paper, we present a novel formulation for
solving a system of noise contaminated linear equations while
preserving the structure of the coefficient matrix. The pro-
posed method has advantages over the known Structured To-
tal Least Squares (STLS) techniques in utilizing additional in-
formation about the uncertainties and robustness in ill-posed
advantages in two applications: signal restoration problem
with an uncertain model and frequency estimation of multi-
ple sinusoids embedded in white noise.
Index Terms— total least squares, robust solutions, in-
verse problems, structured perturbations, bounded data un-
In various signal processing applications such as deconvolu-
tion, signal modeling, frequency estimation and system iden-
tification, it is important to produce robust estimates for an
unknown vector ˆ x from a set of measurements y. Typically,
a linear model is used to relate the unknowns to the available
measurements: y = Hx + w, where the matrix H ∈ Rm×n
describes the linear relationship and w is an additive noise
vector. There are many well known approaches to provide es-
timates ˆ x. For instance, if x is a random vector with known
first and second order statistics, the Wiener estimator, which
minimizes the mean-squared error (MSE) over all linear esti-
mators, is a reasonable choice. In the absence of such a statis-
In many applications the elements of matrix H are also
subject to errors since they are results of some other measure-
ments or an imprecise model. It has been shown that if the er-
rors in H and w are both independent identically distributed
Gaussian noise, the Maximum Likelihood (ML) estimate for
x is provided by the Total Least Squares (TLS) technique,
which ”corrects” the system with minimum perturbation so
that it is consistent . However, in many applications H has
a certain structure, such as Toeplitz and Structured Total Least
Squares (STLS) techniques have been developed to perform
structured perturbations .
A major drawback of both the TLS and the STLS tech-
niques is that, in trying to reach to a consistent system, they
can produce unacceptably large perturbations on H and y.
Another significant problem of TLS arises in nonzero resid-
be due to lower order linear modeling or actual nonlinear re-
lationship between the unknowns and the measurement. In
these cases the TLS solution may be more sensitive than the
LS solution and it is necessary to relax the consistency con-
dition, and incorporate perturbation bounds . For this pur-
pose, two alternative formulations have been proposed. In
Min-Max formulation, which is also referred to as Bounded
ˆ x is chosen as a minimizer of the maximum error over the
set of allowed perturbations. In Min-Min formulation, which
is referred to as Bounded Errors-in-Variables Model , ˆ x is
chosen as a minimizer of the minimum error over the set of al-
lowed perturbations. Therefore, Min-Max approach provides
more conservative estimates than the estimates obtained by
the Min-Min approach.
In this paper, we formulate a new Min-Min type approach,
the Structured Least Squares with Bounded Data Uncertain-
ties (SLS-BDU), to overcome the sensitivity problems in
STLS methods. In the SLS-BDU approach the residual norm
?(H + ΔH)x − (y + Δy)? subject to bounded and struc-
tured perturbations is minimized with respect to x as well as
the perturbations ΔH and Δy. Hence, the consistency is
not forced, and the sensitivity of the solution is kept under
control with the perturbation bounds. Before proceeding with
the details of the proposed approach, we first present a review
on TLS, STLS, Min-Min and Min-Max approaches. Then, on
two different applications, we report results of a comparison
study. Finally, the drawn conclusions are presented.
3261 978-1-4244-2354-5/09/$25.00 ©2009 IEEEICASSP 2009
2. REVIEW: TOTAL LEAST SQUARES AND THE
STRUCTURED TOTAL LEAST SQUARES
Giventheoverdeterminedlinearsystemofequations, Hx ≈y,
where both H and y may have imprecisions, TLS produces x
as the minimum norm solution to (H + ΔH)x = (y + Δy)
where [ΔHΔy] is chosen to be minimum norm perturbation
on the original system which results in a consistent system.
The TLS problem can be solved using the Singular Value
Decomposition (SVD) as :
xTLS= (HTH − σ2
where σn+1is the smallest singular value of [H y] and sub-
tracted to remove the bias introduced by the error in H. How-
ever, the subtraction of σ2
deregulates the inverse operation, hence results in sensitivity
In the Structured Total Least Squares (STLS) formulation
the problem becomes,
n+1I from the diagonal of HTH
ΔH,Δy,x?ΔH Δy?F, s.t.(H+ΔH)x = (y+Δy) and
[ΔH Δy] has the same structure as [A b] .
This problem is non-convex and known to be NP-hard
and developed solution methods are based on local optimiza-
tion. When the matrices are ill conditioned the solution has a
huge norm and variance since STLS introduce deregulariza-
tion similar to TLS.
3. REVIEW: MIN-MAX AND MIN-MIN
3.1. Robust Least Squares
One of the Min-Max techniques is known as the Robust Least
Squares (RLS) which generates estimate to x as a solution to
the following optimization problem:
?[ΔH Δy]?F≤ρ?(H + ΔH)x − (y + Δy)? .
RLS minimizes the worst case residual over a set of pertur-
bations with bounded Frobenius norm. As the bound ρ gets
larger, the obtained solutions deviate more from the least
squares solution. Hence, the RLS approach trades accuracy
SRLS is the structured version of RLS with ΔH =
lems can be obtained using convex, second-order cone pro-
i=1δiHiand solutions to both the RLS and the SRLS prob-
3.2. Bounded Errors-in-Variables Model
One of the Min-Min techniques is known as the Bounded
Errors-in-Variables Model, where the inner maximization of
the RLS cost function is replaced with a minimization over
the allowed perturbations:
?(H + ΔH)x − (y + Δy)? .
As opposed to the cautious approach in the Min-Max tech-
for the most favorable perturbation in the allowed set of per-
turbations. In this sense it is closer to the TLS technique, but
more robust since it does not pursue the consistency as in TLS
resulting in sensitivity issues. However, unlike the Min-Max
case, the Min-Min approach may be degenerate if the residual
becomes zero . The nondegenerate and unstructured case
has the same form of the TLS solution
xMin−Min= (HTH − γI)−1HTy,
for some positive valued γ which depends on the perturbation
bounds. For small enough bounds on the perturbations, it can
be shown that the value of γ is less than that of σ2
TLS solution given in Eqn. 1. . Thus, the deregularization
of the Min-Min solution is less than that of the TLS, resulting
in more robust solutions.
4. PROPOSED STRUCTURED LEAST SQUARES
WITH BOUNDED DATA UNCERTAINTIES
The SLS-BDU approach is a structured Min-Min approach,
that is developed to provide more robust solutions than the
STLS technique. Although the STLS utilizes structured per-
turbations, because it seeks consistency, the perturbations can
be unreasonably large even if a penalty on ?x? is added to
the objective. In many signal processing applications pertur-
bations beyond some bounds cannot be justified. Therefore
in our proposed approach, we want to consider perturbations
that are within a given tolerable bound only. The following
cases illustrate the need for the bounded perturbations:
1. The given linear equations may be inadequate to rep-
resent the observed phenomenon, e.g., wrong model,
nonlinear data, where seeking consistency of equations
is not appropriate.
2. Some elements of the matrix may be exactly known
or given with confidence intervals, e.g., econometric or
3. Forcing the consistency in ill-posed problems may re-
sult a very sensitive estimator and the mean-squared er-
ror is not desirable as it will be shown in numerical
In SLS-BDU approach, we propose to use the following lin-
early structured version of the Bounded Errors-in-Variables
Hiand yiwith αi’s determining the amount of perturbation.
The SLS-BDU formulation allows bounds defined over any
convex set. Here, for the sake of simplicity in the presenta-
tion, we only consider a weighted norm bound on the α with
a positive definite weighting matrix W.
The SLS-BDU optimization given in Eqn. 3. is noncon-
vex. However, as we will show next, an iterative algorithm
can be used to a find a local minimum of it. For this purpose,
αiHi)x − (y +
Similar to the SRLS formulation, the structure is encoded to
H(α) = H+
y(α) = y+
αiyi,α = [α1...αp]T.
Then, simplify the SLS-BDU optimization given in Eqn. 3.
where J(x,α) is defined as ?H(α)x − y(α)?. For a fixed α,
minimization of J(x,α) with respect to x becomes a convex
ordinary least squares problem which can be solved easily.
Now we will show that for a fixed x minimization of J(x,α)
with respect to α is also a convex optimization problem.
where ?(x) = Hx − y, hi = Hix. Hence, for a fixed x
minimization of J(x,α) with respect to α becomes:
?Wα?≤ρ??(x) + [(h1− y1)...(hp− yp)]α?
??(x) + Uα? ,
where U = [(h1− y1)...(hp− yp)]. This final form is a
Constrained Least Squares problem which can be solved by
using the method of Lagrange multipliers .
The above derived convexity results enables us to use the
following iterative optimization algorithm to converge to a lo-
cal minimum of the SLS-BDU optimization given in Eqn. 3.:
Step 1 Set ˆ α0= 0, and ˆ x0= (HTH)−1HTy, ˆ α0= 0.
Step 2 For k ≥ 1, by using the method of Lagrange multi-
pliers update ˆ αk+1as the solution to (6).
Step 3 Set ˆ xk+1 = (H(ˆ αk)TH(ˆ αk))−1HTy(ˆ αk) where
H(α) and y(α) are defined in Eqn.4.
Step 4 Repeat steps 2 and 3, until ?ˆ xk− ˆ xk−1? ≤ ε,
where ε is a user defined threshold of convergence. If
problems are encountered in evaluating ˆ xk, one can use
QR decomposition or Tikhonov regularization.
Observed impulse response
True impulse response
Fig. 1. Nominal and actual impulse responses are shown in
solid and dashed lines respectively.
Table 1. xtrue, xLS and xSLS−BDU correspond to actual
signal and estimates, Htrue, H, HSLS−BDU correspond to
actual, nominal and corrected matrices respectively.
5. NUMERICAL EXAMPLES
5.1. Signal Restoration with an Uncertain Kernel
Suppose that the observed signal is y[n] =
w[n] , n = 0,..N − 1 where
is the kernel of convolution with bounded data uncertainties
on amplitudes | δai |≤ ?aiand dampings | δbi |≤ ?bi, i =
1,...,Np. x[n] is the signal to be estimated and w[n] is white
Gaussian noise. The uncertainties in bi’s can be linearized by
a first order approximation, e−(bi+δbi)n≈ e−bin(1 − δbin) ,
and the uncertain matrix representation becomes,
(ai+ δai)e−(bi+δbi)ncos(win + φi)
y = (H +
αiHi)x + w ,
with the constraint ?Wα?∞≤ ?, where Hiare fixed Toeplitz
Suppose that we observe the nominal impulse response
shown in Fig. 1. and have a priori bounds on the uncertainty.
Structured Least Squares with Bounded Data Uncertainties
051015 20 25
30 35 4045 50
Fig. 2. Actual and restored signals are shown in dashed and
solid lines respectively.
Table 2. Minimum, Maximum and Mean Relative Errors for
LS, STLS and SLS-BDU
corrects the system in given perturbation bounds and restores
the original signal with better accuracy as shown in Fig. 2.
and Table 2. Note that if the uncertainty is not bounded as
in STLS, the approximation may not be valid and the corre-
sponding estimator is not desirable.
5.2. Frequency Estimation of Multiple Sinusoids
Linear prediction equations can be solved to estimate the pa-
rameters of multiple sinusoids and it is shown that STLS es-
timator corresponds to the ML estimator when noise is nor-
mally distributed . Consider the case where parameters of
two sinusoids which are close in frequency need to be esti-
mated with frequencies f1= 0.32 Hz and f2= 0.30 Hz in
white noise wn:
x(n) = cos(2πf1n)+cos(2πf2n)+wn,n = 0,1,...,99.
Wesettheconstraintontheperturbationsas?α? ≤ δ such
that there exists an energy bound on the observed signal. The
relative estimation error Ei??xtrue−x[i]?
and the proposed SLS-BDU estimators are evaluated in inde-
pendent trials at 23 dB SNR and plotted in Fig. 3. As it can be
seen in Table 2 when the consistency condition is relaxed as
in SLS-BDU, the sensitivity problem of STLS is avoided sig-
nificantly without adding a regularization term and therefore
preserving details in the signals which can be resolved.
of LS, STLS 
STSL vs SLS−BDU
Fig. 3. Relative Estimation Error of LS, STLS and SLS-BDU
in 50 independent trials. Frequently ?xSTLS? attains huge
values because of ill conditioning.
A new robust estimation technique is proposed for the solu-
tion of structured linear system of equations with bounded
data uncertainties. Numerical examples showed that the pro-
posed SLS-BDU technique achieves better mean-squared er-
ror and utilizes additional information about the uncertainties.
An iterative algorithm to compute the proposed estimator is
shown to be accurate and efficient. Our formulation can be
used to obtain robust and accurate results in many other sig-
nal processing applications, especially in commonly occur-
ring ill-posed problems with significant sensitivity issues.
 S. Van Huffel and P. Lemmerling, “Total Least Squares
and Errors-in-Variables Modeling Analysis, Algorithms and
Applications“, Kluwer Academic, 2002.
 I. Markovsky, S. Van Huffel, and R. Pintelon, “Block-
Toeplitz/Hankel Structured Total Least Squares“, Tech. Rep.
 L. El-Ghaoui, H. Lebret. “Robust solutions to least-square
problems to uncertain data matrices“, SIAM J. Matrix Anal.
Appl. 18, 1035–1064 (1997).
 S. Chandrasekaran, M. Gu, A. H. Sayed, and K. E. Schu-
bert, “The degenerate bounded error-in-variables model“,
SIAM J. Matrix Anal. Appl., vol. 23, pp. 138–166, 2001.
 G. H. Golub and U. von Matt, “Quadratically constrained
least squares and quadratic problems“, Numer. Math., 59
(1991), pp. 561–580.
 T. Abatzoglou, J. Mendel, and G. Harada, “The con-
strained total least squares technique and its application to
harmonic superresolution“, IEEE Trans. Signal Process., 39
(1991), pp. 1070–1087.