Necessary and Sufficient Conditions for Sparsity Pattern Recovery

Abstract

The paper considers the problem of detecting the sparsity pattern of a k -sparse vector in BBR n from m random noisy measurements. A new necessary condition on the number of measurements for asymptotically reliable detection with maximum-likelihood (ML) estimation and Gaussian measurement matrices is derived. This necessary condition for ML detection is compared against a sufficient condition for simple maximum correlation (MC) or thresholding algorithms. The analysis shows that the gap between thresholding and ML can be described by a simple expression in terms of the total signal-to-noise ratio (SNR), with the gap growing with increasing SNR. Thresholding is also compared against the more sophisticated Lasso and orthogonal matching pursuit (OMP) methods. At high SNRs, it is shown that the gap between Lasso and OMP over thresholding is described by the range of powers of the nonzero component values of the unknown signals. Specifically, the key benefit of Lasso and OMP over thresholding is the ability of Lasso and OMP to detect signals with relatively small components.

Full text

Page 1

arXiv:0804.1839v1 [cs.IT] 11 Apr 2008
NECESSARY AND SUFFICIENT CONDITIONS FOR SPARSITY PATTERN RECOVERY
Necessary and Sufficient Conditions on
Sparsity Pattern Recovery
1
Alyson K. Fletcher, Member, IEEE, Sundeep Rangan,
and Vivek K Goyal, Senior Member, IEEE
Abstract
The problem of detecting the sparsity pattern of a k-sparse vector in Rnfrom m random noisy measurements is
of interest in many areas such as system identification, denoising, pattern recognition, and compressed sensing. This
paper addresses the scaling of the number of measurements m, with signal dimension n and sparsity-level nonzeros
k, for asymptotically-reliable detection. We show a necessary condition for perfect recovery at any given SNR for
all algorithms, regardless of complexity, is m = Ω(k log(n − k)) measurements. Conversely, it is shown that this
scaling of Ω(k log(n − k)) measurements is sufficient for a remarkably simple “maximum correlation” estimator.
Hence this scaling is optimal and does not require more sophisticated techniques such as lasso or matching pursuit.
The constants for both the necessary and sufficient conditions are precisely defined in terms of the minimum-to-
average ratio of the nonzero components and the SNR. The necessary condition improves upon previous results
for maximum likelihood estimation. For lasso, it also provides a necessary condition at any SNR and for low SNR
improves upon previous work. The sufficient condition provides the first asymptotically-reliable detection guarantee
at finite SNR.
Index Terms
compressed sensing, convex optimization, lasso, maximum likelihood estimation, random matrices, random
projections, regression, sparse approximation, sparsity, subset selection
I. INTRODUCTION
Suppose one is given an observation y ∈ Rmthat was generated through y = Ax+d, where A ∈ Rm×n
is known and d ∈ Rmis an additive noise vector with a known distribution. It may be desirable for an
estimate of x to have a small number of nonzero components. An intuitive example is when one wants
to choose a small subset from a large number of possibly-related factors that linearly influence a vector
of observed data. Each factor corresponds to a column of A, and one wishes to find a small subset of
columns with which to form a linear combination that closely matches the observed data y. This is the
subset selection problem in (linear) regression [1], and it gives no reason to penalize large values for the
nonzero components.
In this paper, we assume that the true signal x has k nonzero entries and that k is known when estimating
x from y. We are concerned with establishing necessary and sufficient conditions for the recovery of the
positions of the nonzero entries of x, which we call the sparsity pattern. Once the sparsity pattern is
correct, n−k columns of A can be ignored and the stability of the solution is well understood; however,
we do not study any other performance criterion.
This work was supported in part by a University of California President’s Postdoctoral Fellowship, NSF CAREER Grant CCF-643836,
and the Centre Bernoulli at´Ecole Polytechnique F´ ed´ erale de Lausanne.
A. K. Fletcher (email: alyson@eecs.berkeley.edu) is with the Department of Electrical Engineering and Computer Sciences, University of
California, Berkeley.
S. Rangan (email: srangan@qualcomm.com) is with Qualcomm Technologies, Bedminster, NJ.
V. K. Goyal (email: vgoyal@mit.edu) is with the Department of Electrical Engineering and Computer Science and the Research Laboratory
of Electronics, Massachusetts Institute of Technology.

Page 2

2NECESSARY AND SUFFICIENT CONDITIONS FOR SPARSITY PATTERN RECOVERY
A. Previous Work
Sparsity pattern recovery (or more simply, sparsity recovery) has received considerable attention in a
variety guises. Most transparent from our formulation is the connection to sparse approximation. In a
typical sparse approximation problem, one is given data y ∈ Rm, dictionary1A ∈ Rm×n, and tolerance
ǫ > 0. The aim is to find ˆ x with the fewest number of nonzero entries among those satisfying ?Aˆ x−y? ≤ ǫ.
This problem is NP-hard [3] but greedy heuristics (matching pursuit [2] and its variants) and convex
relaxations (basis pursuit [4], lasso [5] and others) can be effective under certain conditions on A and
y [6]–[8]. Scaling laws for sparsity recovery with any A were first given in [9].
More recently, the concept of “sensing” sparse x through multiplication by a suitable random matrix A,
with measurement error d, has been termed compressed sensing [10]–[12]. This has popularized the study
of sparse approximation with respect to random dictionaries, which was considered also in [13]. Results
are generally phrased as the asymptotic scaling of the number of measurements m (the length of y) needed
for sparsity recovery to succeed with high probability, as a function of the other problem parameters. More
specifically, most results are sufficient conditions for specific tractable recovery algorithms to succeed. For
example, if A has i.i.d. Gaussian entries and d = 0, then m ≍ 2k log(n/k) dictates the minimum scaling
at which basis pursuit succeeds with high probability [14]. With nonzero noise variance, necessary and
sufficient conditions for the success of lasso in this setting have the asymptotic scaling [15]
m ≍ 2k log(n − k) + k + 1.
(1)
To understand the ultimate limits of sparsity recovery, while also casting light on the efficacy of lasso
or orthogonal matching pursuit (OMP), it is of interest to determine necessary and sufficient conditions
for an optimal recovery algorithm to succeed. Of course, since it is sufficient for lasso, the condition
(1) is sufficient for an optimal algorithm. Is it close to a necessary condition? We address precisely this
question by proving a necessary condition that differs from (1) by a factor that is constant with respect
to n and k while depending on the signal-to-noise ratio (SNR) and mean-to-average ratio (MAR), which
will be defined precisely in Section II. Furthermore, we present an extremely simple algorithm for which
a sufficient condition for sparsity recovery is similarly within a constant factor of (1).
Previous necessary conditions had been based on information-theoretic analyses such as the capacity
arguments in [16], [17] and a use of Fano’s inequality in [18]. More recent publications with necessary
conditions include [19]–[22]. As described in Section III, our new necessary conditions are stronger than
the previous results.
Table I previews our main results and places (1) in context. The measurement model and parameters
MAR and SNR are defined in the following section. Arbitrarily small constants have been omitted, and
the last column—labeled simply SNR → ∞—is more specifically for MAR > ǫ > 0 for some fixed ǫ and
SNR = Ω(k).
B. Paper Organization
The setting is formalized in Section II. In particular, we define our concepts of signal-to-noise ratio and
mean-to-average ratio; our results clarify the roles of these quantities in the sparsity recovery problem.
Necessary conditions for success of any algorithm are considered in Section III. There we present a new
necessary condition and compare it to previous results and numerical experiments. Section IV introduces
a very simple recovery algorithm for the purpose of showing that a sufficient condition for its success is
rather weak—it has the same dependence on n and k as (1). Conclusions are given in Section V, and
proofs appear in the Appendix.
1The term seems to have originated in [2] and may apply to A or the columns of A as a set.

Page 3

FLETCHER, RANGAN AND GOYAL3
finite SNRSNR → ∞
Any algorithm must fail
m <
2
MAR·SNRk log(n − k) + k − 1
Theorem 1
m ≤ k
(elementary)
Necessary and
sufficient for lasso
unknown (expressions above
and right are necessary)
m ≍ 2k log(n − k) + k + 1
Wainwright [15]
Sufficient for maximum
correlation estimator (8)
m >
8(1+SNR)
MAR·SNRk log(n − k)
Theorem 2
m >
8
MARk log(n − k)
from Theorem 2
TABLE I
SUMMARY OF RESULTS ON MEASUREMENT SCALING FOR RELIABLE SPARSITY RECOVERY
(SEE BODY FOR DEFINITIONS AND TECHNICAL LIMITATIONS)
II. PROBLEM STATEMENT
Consider estimating a k-sparse vector x ∈ Rnthrough a vector of observations,
y = Ax + d,
(2)
where A ∈ Rm×nis a random matrix with i.i.d. N(0,1/m) entries and d ∈ Rmis i.i.d. unit-variance
Gaussian noise. Denote the sparsity pattern of x (positions of nonzero entries) by the set Itrue, which is
a k-element subset of the set of indices {1, 2, ..., n}. Estimates of the sparsity pattern will be denoted
byˆI with subscripts indicating the type of estimator. We seek conditions under which there exists an
estimator such thatˆI = Itruewith high probability.
In addition to the signal dimensions, m, n and k, we will show that there are two variables that dictate
the ability to detect the sparsity pattern reliably: the SNR, and what we will call the minimum-to-average
ratio.
The SNR is defined by
SNR =E[?Ax?2]
E[?d?2]
Since we are considering x as an unknown deterministic vector, the SNR can be further simplified as
follows: The entries of A are i.i.d. N(0,1/m), so columns ai∈ Rmand aj∈ Rmof A satisfy E[a′
Therefore, the signal energy is given by
=E[?Ax?2]
m
.
(3)
iaj] = δij.
E??Ax?2?
= E
?
?
?
j∈Itrue
ajxj?2
?
=
??
??
i,j∈Itrue
E[a′
iajxixj]
=
i,j∈Itrue
xixjδij = ?x?2.
Substituting into the definition (3), the SNR is given by
SNR =1
m?x?2.
(4)
The minimum-to-average ratio of x is defined as
MAR =minj∈Itrue|xj|2
?x?2/k
.
(5)
Since ?x?2/k is the average of {|xj|2| j ∈ Itrue}, MAR ∈ (0,1] with the upper limit occurring when all
the nonzero entries of x have the same magnitude.

Page 4

4NECESSARY AND SUFFICIENT CONDITIONS FOR SPARSITY PATTERN RECOVERY
Remarks: Other works use a variety of normalizations, e.g.: the entries of A have variance 1/n in [12],
[20]; the entries of A have unit variance and the variance of d is a variable σ2in [15], [18], [21], [22];
and our scaling of A and a noise variance of σ2are used in [23]. This necessitates great care in comparing
results.
Some results involve
MAR · SNR =k
While a similar quantity affects a regularization weight sequence in [15], there it does not affect the
number of measurements required for the success of lasso.2The magnitude of the smallest nonzero entry
of x is also prominent in the phrasing of results in [21], [22].
mmin
j∈Itrue|xj|2.
III. NECESSARY CONDITION FOR SPARSITY RECOVERY
We first consider sparsity recovery without being concerned with computational complexity of the
estimation algorithm. Since the vector x ∈ Rnis k-sparse, the vector Ax belongs to one of L =
subspaces spanned by k of the n columns of A. Estimation of the sparsity pattern is the selection of one
of these subspaces, and since the noise d is Gaussian, the probability of error is minimized by choosing
the subspace closest to the observed vector y. This results in the maximum likelihood (ML) estimate.
Mathematically, the ML estimator can be described as follows. Given a subset J ⊆ {1, 2, ..., n}, let
PJy denote the orthogonal projection of the vector y onto the subspace spanned by the vectors {aj| j ∈ J}.
The ML estimate of the sparsity pattern is
?n
k
?
ˆIML= argmax
J : |J|=k
?PJy?2,
where |J| denotes the cardinality of J. That is, the ML estimate is the set of k indices such that the
subspace spanned by the corresponding columns of A contain the maximum signal energy of y.
Since the number of subspaces, L, grows exponentially in n and k, an exhaustive search is computa-
tionally infeasible. However, the performance of ML estimation provides a lower bound on the number of
measurements needed by any algorithm that cannot exploit a priori information on x other than it being
k-sparse.
Theorem 1: Let k = k(n) and m = m(n) vary with n such that limn→∞k(n) = ∞ and
2 − δ
MAR · SNRk log(n − k) + k − 1
for some δ > 0. Then even the ML estimator asymptotically cannot detect the sparsity pattern, i.e.,
m(n) <
(6)
lim
n→∞Pr
?ˆIML= Itrue
?
= 0.
Proof: See Appendix B.
The theorem shows that for fixed SNR and MAR, the scaling m = Ω(k log(n − k)) is necessary for
reliable sparsity pattern recovery. The next section will show that this scaling can be achieved with an
extremely simple method.
Remarks:
1) The theorem applies for any k(n) such that limn→∞k(n) = ∞, including both cases with k = o(n)
and k = Θ(n). In particular, under linear sparsity (k = αn for some constant α), the theorem shows
that
m ≍
measurements are necessary for sparsity recovery. Similarly, if m/n is bounded above by a constant,
then sparsity recovery will certainly fail unless k = O(n/logn).
2α
MAR · SNRnlogn
2The formulation of [15] makes SNR = Θ(n), which obscures the effect of the noise level. See also the second remark following
Theorem 2.

Page 5

FLETCHER, RANGAN AND GOYAL5
2) In the case of MAR· SNR < 1, the bound (6) improves upon the necessary condition of [15] for the
asymptotic success of lasso by the factor (MAR · SNR)−1.
3) The bound (6) can be compared against the information-theoretic bounds mentioned earlier. The
tightest of these bounds is in [17] and shows that the problem dimensions must satisfy
2
mlog2
?n
k
?
≤ log2(1 + SNR) − αlog2(1 +SNR
α
),
(7)
where α = k/n is the sparsity ratio. For large n and k, the bound can be rearranged as
m ≥2h(α)
α
?
log2(1 + SNR) − αlog2(1 +
SNR
α
)
?−1
k,
where h(·) is the binary entropy function. In particular, when the sparsity ratio α is fixed, the bound
shows only that m needs to grow at least linearly with k. In contrast, Theorem 1 shows that with
fixed sparsity ratio m = Ω(k log(n−k)) is necessary for reliable sparsity recovery. Thus, the bound
in Theorem 1 is significantly tighter and reveals that the previous information-theoretic necessary
conditions from [16]–[18], [21], [22] are overly optimistic.
4) Results more similar to Theorem 1—based on direct analyses of error events rather than information-
theoretic arguments—appeared in [19], [20]. The previous results showed that with fixed SNR as
defined here, sparsity recovery with m = Θ(k) must fail. The more refined analysis in this paper
gives the additional log(n − k) factor and the precise dependence on MAR · SNR.
5) Theorem 1 is not contradicted by the relevant sufficient condition of [21], [22]. That sufficient
condition holds for scaling that gives linear sparsity and MAR·SNR = Ω(√nlogn). For MAR·SNR =
√nlogn, Theorem 1 shows that fewer than m ≍ 2√k logk measurements will cause ML decoding
to fail, while [22, Thm. 3.1] shows that a typicality-based decoder will succeed with m = Θ(k)
measurements.
6) Note that the necessary condition of [18] is proven for MAR = 1. Theorem 1 gives a bound that
increases for smaller MAR; this suggests (though does not prove, since the condition is merely
necessary) that smaller MAR makes the problem harder.
Numerical validation: Computational confirmation of Theorem 1 is technically impossible, and even
qualitative support is hard to obtain because of the high complexity of ML detection. Nevertheless, we
may obtain some evidence through Monte Carlo simulation.
Fig. 1 shows the probability of success of ML detection for n = 20 as k, m, SNR, and MAR are varied,
with each point representing at least 500 independent trials. Each subpanel gives simulation results for
k ∈ {1,2,...,5} and m ∈ {1,2,...,40} for one (SNR, MAR) pair. Signals with MAR < 1 are created
by having one small nonzero component and k − 1 equal, larger nonzero components. Overlaid on the
color-intensity plots is a black curve representing (6).
Taking any one column of one subpanel from bottom to top shows that as m is increased, there
is a transition from ML failing to ML succeeding. One can see that (6) follows the failure-success
transition qualitatively. In particular, the empirical dependence on SNR and MAR approximately follows
(6). Empirically, for the (small) value of n = 20, it seems that with MAR·SNR held fixed, sparsity recovery
becomes easier as SNR increases (and MAR decreases).
Less extensive Monte Carlo simulations for n = 40 are reported in Fig. 2. The results are qualitatively
similar. As might be expected, the transition from low to high probability of successful recovery as a
function of m appears more sharp at n = 40 than at n = 20.
IV. SUFFICIENT CONDITION WITH MAXIMUM CORRELATION DETECTION
Consider the following simple estimator. As before, let ajbe the jth column of the random matrix A.
Define the maximum correlation (MC) estimate as
ˆIMC=?j : |a′
jy| is one of the k largest values of |a′
iy|?.
(8)