No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Exact Sparse Nonnegative Least Squares
... Cohen and Gillis (2019) proposed a method that solves exactly the k-sparse NNLS subproblems using a bruteforce approach. Nadisic et al. (2020) extended this work by replacing the bruteforce subroutine by a dedicated branch-and-bound algorithm. To the best of our knowledge, no existing work considered a matrix-wise 0 constraint. ...
... The algorithm Arborescent (Nadisic et al. 2020) is a branch-and-bound algorithm designed to solve exactly k-sparse NNLS problems. In a nutshell, Arborescent enumerates the possible supports (that is, the possible patterns of zeros) on a search tree, as shown in Fig. 2. ...
... For one subproblem, that is, to generate one Pareto front, we have that • The cost of Arborescent depends on the number of nodes explored in the branch and bound. In the worst case, it is of the same order as the bruteforce algorithm, and requires to solve r k NNLS subproblems, while, in the best case, it is of the order of r (Nadisic et al. 2020). Empirically, the cost is far from the worst case but grows faster than linear with r. • The cost of NNOMP is in O(mr 4 ) operations (Yaghoobi et al. 2015). ...
Nonnegative least squares problems with multiple right-hand sides (MNNLS) arise in models that rely on additive linear combinations. In particular, they are at the core of most nonnegative matrix factorization algorithms and have many applications. The nonnegativity constraint is known to naturally favor sparsity, that is, solutions with few non-zero entries. However, it is often useful to further enhance this sparsity, as it improves the interpretability of the results and helps reducing noise, which leads to the sparse MNNLS problem. In this paper, as opposed to most previous works that enforce sparsity column- or row-wise, we first introduce a novel formulation for sparse MNNLS, with a matrix-wise sparsity constraint. Then, we present a two-step algorithm to tackle this problem. The first step divides sparse MNNLS in subproblems, one per column of the original problem. It then uses different algorithms to produce, either exactly or approximately, a Pareto front for each subproblem, that is, to produce a set of solutions representing different tradeoffs between reconstruction error and sparsity. The second step selects solutions among these Pareto fronts in order to build a sparsity-constrained matrix that minimizes the reconstruction error. We perform experiments on facial and hyperspectral images, and we show that our proposed two-step approach provides more accurate results than state-of-the-art sparse coding heuristics applied both column-wise and globally.
... Sparse NMF is usually solved by extending standard NMF algorithms with a regularization such as the 1 penalty [9,11], or constraints on some sparsity measure, like the one introduced in [10]. Recently, exact k-sparse methods based on the 0 -"norm" have been used for NMF, using a brute-force approach [4], or a dedicated branch-and-bound algorithm [16]. They allow the explicit definition of a maximum number (usually noted k) of non-zero entries per column of H. ...
... In particular, a brute-force approach could tackle this problem by solving O(r k ) NNLS problems. However, this combinatorial subproblem can be solved exactly and at a reasonable cost by dedicated branch-and-bound algorithms, such as arborescent [16], given that r is sufficiently small, which is typically the case in practice. Even when k is fixed, the following result shows that no provably correct algorithm exists for solving SSNMF in polynomial time (unless P=NP): ...
... On Lines 3 to 5, we apply a postprocessing to the selected points by checking whether they are k-sparse combinations of other selected points; this is a k-sparse NNLS problem solved with arborescent [16]. If they are, then they cannot be vertices and they are discarded, such as point M 1 in Figure 1b which belongs to the segment [W 1 , W 4 ]. ...
We propose a new variant of nonnegative matrix factorization (NMF), combining separability and sparsity assumptions. Separability requires that the columns of the first NMF factor are equal to columns of the input matrix, while sparsity requires that the columns of the second NMF factor are sparse. We call this variant sparse separable NMF (SSNMF), which we prove to be NP-complete, as opposed to separable NMF which can be solved in polynomial time. The main motivation to consider this new model is to handle underdetermined blind source separation problems, such as multispectral image unmixing. We introduce an algorithm to solve SSNMF, based on the successive nonnegative projection algorithm (SNPA, an effective algorithm for separable NMF), and an exact sparse nonnegative least squares solver. We prove that, in noiseless settings and under mild assumptions, our algorithm recovers the true underlying sources. This is illustrated by experiments on synthetic data sets and the unmixing of a multispectral image.
... Sparse NMF is usually solved by extending standard NMF algorithms with a regularization such as the 1 penalty [9,11], or constraints on some sparsity measure, like the one introduced in [10]. Recently, exact k-sparse methods based on the 0 -"norm" have been used for NMF, using a brute-force approach [4], or a dedicated branch-and-bound algorithm [16]. They allow the explicit definition of a maximum number (usually noted k) of non-zero entries per column of H. ...
... In particular, a brute-force approach could tackle this problem by solving O(r k ) NNLS problems. However, this combinatorial subproblem can be solved exactly and at a reasonable cost by dedicated branch-and-bound algorithms, such as arborescent [16], given that r is sufficiently small, which is typically the case in practice. Even when k is fixed, the following result shows that no provably correct algorithm exists for solving SSNMF in polynomial time (unless P=NP): ...
... On Lines 3 to 5, we apply a postprocessing to the selected points by checking whether they are k-sparse combinations of other selected points; this is a k-sparse NNLS problem solved with arborescent [16]. If they are, then they cannot be vertices and they are discarded, such as point M 1 in Figure 1b which belongs to the segment [W 1 , W 4 ]. ...
We propose a new variant of nonnegative matrix factorization (NMF), combining separability and sparsity assumptions. Separability requires that the columns of the first NMF factor are equal to columns of the input matrix, while sparsity requires that the columns of the second NMF factor are sparse. We call this variant sparse separable NMF (SSNMF), which we prove to be NP-complete, as opposed to separable NMF which can be solved in polynomial time. The main motivation to consider this new model is to handle underdetermined blind source separation problems, such as multispectral image unmixing. We introduce an algorithm to solve SSNMF, based on the successive nonnegative projection algorithm (SNPA, an effective algorithm for separable NMF), and an exact sparse nonnegative least squares solver. We prove that, in noiseless settings and under mild assumptions, our algorithm recovers the true underlying sources. This is illustrated by experiments on synthetic data sets and the unmixing of a multispectral image.
... Analysis of similar sparsity constrained problems using gauge function is given in Chandrasekaran et al. (2012) which also proposes a MIP-based solution. In a recent study Nadisic et al. (2020), it was also proposed to adapt the Branch and Bound algorithm to solve this problem. The thesis of the present paper is that one can solve to optimality a large number of instances by using a judicious enumeration strategy. ...
... Comparison with Arborescent One of the main alternatives to our algorithm is another enumeration algorithm called Arborescent Nadisic et al. (2020). That approach uses a branch and bound idea to solve the sparse NNLS problem. ...
Computing sparse solutions to overdetermined linear systems is a ubiquitous problem in several fields such as regression analysis, signal and image processing, information theory and machine learning. Additional non-negativity constraints in the solution are useful for interpretability. Most of the previous research efforts aimed at approximating the sparsity constrained linear least squares problem, and/or finding local solutions by means of descent algorithms. The objective of the present paper is to report on an efficient and modular implicit enumeration algorithm to find provably optimal solutions to the NP-hard problem of sparsity-constrained non-negative least squares. We focus on the problem where the system is assumed to be over-determined where the matrix has full column rank. Numerical results with real test data as well as comparisons of competing methods and an application to hyperspectral imaging are reported. Finally, we present a Python library implementation of our algorithm.
... We now focus on heuristics to find candidate solutions to MSC. Similarly to sparse coding, MSC is an NP-hard problem for which obtaining the global solution typically requires costly algorithms [64,65]. Therefore, in the following section, several heuristics are proposed that aim at finding good sparse approximations in reasonable time. ...
Constrained tensor and matrix factorization models allow to extract interpretable patterns from multiway data. Therefore crafting efficient algorithms for constrained low-rank approximations is nowadays an important research topic. This work deals with columns of factor matrices of a low-rank approximation being sparse in a known and possibly overcomplete basis, a model coined as Dictionary-based Low-Rank Approximation (DLRA). While earlier contributions focused on finding factor columns inside a dictionary of candidate columns, i.e., one-sparse approximations, this work is the first to tackle DLRA with sparsity larger than one. I propose to focus on the sparse-coding subproblem coined Mixed Sparse-Coding (MSC) that emerges when solving DLRA with an alternating optimization strategy. Several algorithms based on sparse-coding heuristics (greedy methods, convex relaxations) are provided to solve MSC. The performance of these heuristics is evaluated on simulated data. Then, I show how to adapt an efficient MSC solver based on the LASSO to compute Dictionary-based Matrix Factorization and Canonical Polyadic Decomposition in the context of hyperspectral image processing and chemometrics. These experiments suggest that DLRA extends the modeling capabilities of low-rank approximations, helps reducing estimation variance and enhances the identifiability and interpretability of estimated factors.
... We now focus on heuristics to find candidate solutions to MSC. Similarly to sparse coding, MSC is an NP-hard problem for which obtaining the global solution typically requires costly algorithms (Bourguignon et al., 2015;Nadisic et al., 2020). Therefore, in the following section, several heuristics are proposed that aim at finding good sparse approximations in reasonable time. ...
Constrained tensor and matrix factorization models allow to extract interpretable patterns from multiway data. Therefore identifiability properties and efficient algorithms for constrained low-rank approximations are nowadays important research topics. This work deals with columns of factor matrices of a low-rank approximation being sparse in a known and possibly overcomplete basis, a model coined as Dictionary-based Low-Rank Approximation (DLRA). While earlier contributions focused on finding factor columns inside a dictionary of candidate columns, i.e. one-sparse approximations, this work is the first to tackle DLRA with sparsity larger than one. I propose to focus on the sparse-coding subproblem coined Mixed Sparse-Coding (MSC) that emerges when solving DLRA with an alternating optimization strategy. Several algorithms based on sparse-coding heuristics (greedy methods, convex relaxations) are provided to solve MSC. The performance of these heuristics is evaluated on simulated data. Then, I show how to adapt an efficient MSC solver based on the LASSO to compute Dictionary-based Matrix Factorization and Canonical Polyadic Decomposition in the context of hyperspectral image processing and chemometrics. These experiments suggest that DLRA extends the modeling capabilities of low-rank approximations, helps reducing estimation variance and enhances the identifiability and interpretability of estimated factors.
... Greedy schemes are standard techniques for sparse approximation, with a relatively low computing cost compared to exact methods [4,28], while convex relaxation yields another important branch of approximate methods [33,14]. ...
This paper proposes an exact recovery analysis of greedy algorithms for non-negative sparse representations. Orthogonal greedy algorithms such as Orthogonal Matching Pursuit (OMP) and Orthogonal Least Squares (OLS) consist of gradually increasing the solution support and updating the nonzero coefficients in the least squares sense. From a theoretical viewpoint, greedy algorithms have been extensively studied in terms of exact support recovery. In contrast, the exact recovery analysis of their non-negative extensions (NNOMP, NNOLS) remains an open problem. We show that when the mutual coherence μ is lower than 12K−1, the iterates of NNOMP / NNOLS coincide with those of OMP / OLS, respectively, the latter being known to reach K-step exact recovery. Our analysis heavily relies on a sign preservation property satisfied by OMP and OLS. This property is of stand-alone interest and constitutes our second important contribution. Finally, we provide an extended discussion of the main challenges of deriving improved analyses for correlated dictionaries.
... Estimating early re ections knowing their number can be recast as a sparse coding problem. Unfortunately, problems in this class are not necessarily easier to solve, but it was shown that they produce more accurate solutions [Bourguignon et al. 2015;Nadisic et al. 2020]. ...
Most of audio signal processing methods regard reverberation and in particular acoustic echoes as a nuisance. However, they convey important spatial and semantic information about sound sources and, based on this, recent echo-aware methods have been proposed. In this work, we focus on two directions. First, we study how to estimate acoustic echoes blindly from microphone recordings. Two approaches are proposed, one leveraging on continuous dictionaries, one using recent deep learning techniques. Then, we focus on extending existing methods in audio scene analysis to their echo-aware forms. The Multichannel NMF framework for audio source separation, the SRP-PHAT localization method, and the MVDR beamformer for speech enhancement are all extended to their echo-aware versions.
... Cohen and Gillis (2019) proposed a method that solves exactly the k-sparse NNLS subproblems, using a bruteforce approach. Nadisic et al (2020) extended this work by replacing the bruteforce subroutine by a dedicated branch-and-bound algorithm. ...
Nonnegative least squares (NNLS) problems arise in models that rely on additive linear combinations. In particular, they are at the core of nonnegative matrix factorization (NMF) algorithms. The nonnegativity constraint is known to naturally favor sparsity, that is, solutions with few non-zero entries. However, it is often useful to further enhance this sparsity, as it improves the interpretability of the results and helps reducing noise. While the -"norm", equal to the number of non-zeros entries in a vector, is a natural sparsity measure, its combinatorial nature makes it difficult to use in practical optimization schemes. Most existing approaches thus rely either on its convex surrogate, the -norm, or on heuristics such as greedy algorithms. In the case of multiple right-hand sides NNLS (MNNLS), which are used within NMF algorithms, sparsity is often enforced column- or row-wise, and the fact that the solution is a matrix is not exploited. In this paper, we first introduce a novel formulation for sparse MNNLS, with a matrix-wise sparsity constraint. Then, we present a two-step algorithm to tackle this problem. The first step uses a homotopy algorithm to produce the whole regularization path for all the -penalized NNLS problems arising in MNNLS, that is, to produce a set of solutions representing different tradeoffs between reconstruction error and sparsity. The second step selects solutions among these paths in order to build a sparsity-constrained matrix that minimizes the reconstruction error. We illustrate the advantages of our proposed algorithm for the unmixing of facial and hyperspectral images.
... Theoretical results on NNLS have shown that, especially when the dimension of Q is large (large K and E in our case), the solutions tend to be very sparse [58,59,60]. The optimization problem can be solved by the widely used algorithm in [61] (see also a faster version in [62]), as well as by concurrent work on exact sparse NNLS [63] and even strong GPU accelerations could be exploited [64,65] ...
Auxiliary particle filters (APFs) are a class of sequential Monte Carlo (SMC) methods for Bayesian inference in state-space models. In their original derivation, APFs operate in an extended space using an auxiliary variable to improve the inference. Later works have re-interpreted APFs from a multiple importance sampling perspective. In this perspective, the proposal is a mixture composed of kernels and weights that are selected by taking into account the latest observation. In this work, we further exploit this perspective by proposing an online, flexible framework for APFs that adapts the mixture proposal by convex optimization and allows for a controllable computational complexity. We minimize the discrepancy between the proposal and the filtering distribution at a set of relevant points, which are chosen by leveraging the structure of SMC. We compare our method to state-of-the-art particle filters, showing better performance in challenging and widely used dynamical models.
Orthogonal greedy algorithms are popular sparse signal reconstruction algorithms. Their principle is to select atoms one by one. A series of unconstrained least-square subproblems of gradually increasing size is solved to compute the approximation coefficients, which is efficiently performed using a fast recursive update scheme. When dealing with nonnegative sparse signal reconstruction, a series of
least-squares subproblems have to be solved. Fast implementation becomes tricky since each subproblem does not have a closedform solution anymore. Recently, non-negative extensions of the classical orthogonal matching pursuit and orthogonal least squares algorithms were proposed, using slow (i.e., non-recursive) or recursive but inexact implementations. In this paper, we revisit these algorithms in a unified way. We define a class of non-negative orthogonal algorithms and exhibit their structural properties. We propose a fast and exact implementation based on the active-set resolution of non-negative least-squares and exploiting warm start initializations. The algorithms are assessed in terms of accuracy and computational complexity for a sparse spike deconvolution problem. We also present an application to near-infrared spectra decomposition.
In this paper we consider a variant of the dictionary learning problem where the dictionary has full rank, the coefficients have a fixed sparsity level, and both the coefficients and the dictionary are nonnegative. It is equivalent to k-sparse nonnegative matrix factorization (K-NMF). This model is encountered in source separation where nonnegative linear combinations of a few components generate the data points (samples), such as in hyperspectral images. We first discuss the impact of nonnegativity on the identifiability of low-rank sparse component analysis (LRSCA), building upon recent advances. Then, as a main contribution, we propose two algorithms to train K-NMF: one based on alternating optimization and exact sparse coding, the other based on a nonnegative variant of K-subspace. We show on noiseless simulated data that our methods outperform by a large margin the state of the art. Finally, we apply our methods for the spectral unmixing of a hyperspectral image.
This software package is a MATLAB implementation of infeasible path-following algorithms for solving standard semidefinite programs (SDP). Mehrotra-type predictor-corrector variants are included. Analogous algorithms for the homogeneous formulation of the standard SDP are also implemented. Four types of search directions are available, namely, the AHO, HKM, NT, and GT directions. A few classes of SDP problems are included as well. Numerical results for these classes show that our algorithms are fairly efficient and robust on problems with dimensions of the order of a hundred.
Blind hyperspectral unmixing (HU), also known as unsupervised HU, is one of the most prominent research topics in signal processing (SP) for hyperspectral remote sensing [1], [2]. Blind HU aims at identifying materials present in a captured scene, as well as their compositions, by using high spectral resolution of hyperspectral images. It is a blind source separation (BSS) problem from a SP viewpoint. Research on this topic started in the 1990s in geoscience and remote sensing [3]-[7], enabled by technological advances in hyperspectral sensing at the time. In recent years, blind HU has attracted much interest from other fields such as SP, machine learning, and optimization, and the subsequent cross-disciplinary research activities have made blind HU a vibrant topic. The resulting impact is not just on remote sensing - blind HU has provided a unique problem scenario that inspired researchers from different fields to devise novel blind SP methods. In fact, one may say that blind HU has established a new branch of BSS approaches not seen in classical BSS studies. In particular, the convex geometry concepts - discovered by early remote sensing researchers through empirical observations [3]-[7] and refined by later research - are elegant and very different from statistical independence-based BSS approaches established in the SP field. Moreover, the latest research on blind HU is rapidly adopting advanced techniques, such as those in sparse SP and optimization. The present development of blind HU seems to be converging to a point where the lines between remote sensing-originated ideas and advanced SP and optimization concepts are no longer clear, and insights from both sides would be used to establish better methods.
This letter demonstrates that sparse recovery can be achieved by an -minimization ersatz easily implemented using a conventional nonnegative least squares algorithm. A connection with orthogonal matching pursuit is also highlighted. The preliminary results call for more investigations on the potential of the method and on its relations to classical sparse recovery algorithms.
Nonnegative matrix factorization (NMF) has become a widely used tool for the
analysis of high-dimensional data as it automatically extracts sparse and
meaningful features from a set of nonnegative data vectors. We first illustrate
this property of NMF on three applications, in image processing, text mining
and hyperspectral imaging --this is the why. Then we address the problem of
solving NMF, which is NP-hard in general. We review some standard NMF
algorithms, and also present a recent subclass of NMF problems, referred to as
near-separable NMF, that can be solved efficiently (that is, in polynomial
time), even in the presence of noise --this is the how. Finally, we briefly
describe some problems in mathematics and computer science closely related to
NMF via the nonnegative rank.
In this paper, we study the nonnegative matrix factorization problem under
the separability assumption (that is, there exists a cone spanned by a small
subset of the columns of the input nonnegative data matrix containing all
columns), which is equivalent to the hyperspectral unmixing problem under the
linear mixing model and the pure-pixel assumption. We present a family of fast
recursive algorithms, and prove they are robust under any small perturbations
of the input data matrix. This family generalizes several existing
hyperspectral unmixing algorithms and hence provides for the first time a
theoretical justification of their better practical performance.
This software package is a Matlab implementation of infeasible path-following al-gorithms for solving standard semideenite programming (SDP) problems. Mehrotra-type predictor-corrector variants are included. Analogous algorithms for the homoge-neous formulation of the standard SDP problem are also implemented. Four types of search directions are available, namely, the AHO, HKM, NT, and GT directions. A few classes of SDP problems are included as well. Numerical results for these classes show that our algorithms are fairly eecient and robust on problems with dimensions of the order of a few hundreds.
This paper discusses computational experiments with linear optimization problems involving semidefinite, quadratic, and linear
cone constraints (SQLPs). Many test problems of this type are solved using a new release of SDPT3, a Matlab implementation
of infeasible primal-dual path-following algorithms. The software developed by the authors uses Mehrotra-type predictor-corrector
variants of interior-point methods and two types of search directions: the HKM and NT directions. A discussion of implementation
details is provided and computational results on problems from the SDPLIB and DIMACS Challenge collections are reported.
This paper describes an algorithm for cardinality-constrained quadratic optimization problems, which are convex quadratic
programming problems with a limit on the number of non-zeros in the optimal solution. In particular, we consider problems
of subset selection in regression and portfolio selection in asset management and propose branch-and-bound based algorithms
that take advantage of the special structure of these problems. We compare our tailored methods against CPLEX’s quadratic
mixed-integer solver and conclude that the proposed algorithms have practical advantages for the special class of problems
we consider.
We describe graph implementations, a generic method for representing a convex function via its epigraph, described in a disciplined convex programming framework.
This simple and natural idea allows a very wide variety of smooth and nonsmooth convex programs to be easily specified and
efficiently solved, using interiorpoint methods for smooth or cone convex programs.
Effective unmixing of hyperspectral data cube under a noisy scenario has been a challenging research problem in remote sensing arena. A branch of existing hyperspectral unmixing algorithms is based on Craig's criterion, which states that the vertices of the minimum-volume simplex enclosing the hyperspectral data should yield high fidelity estimates of the endmember signatures associated with the data cloud. Recently, we have developed a minimum-volume enclosing simplex (MVES) algorithm based on Craig's criterion and validated that the MVES algorithm is very useful to unmix highly mixed hyperspectral data. However, the presence of noise in the observations expands the actual data cloud, and as a consequence, the endmember estimates obtained by applying Craig-criterion-based algorithms to the noisy data may no longer be in close proximity to the true endmember signatures. In this paper, we propose a robust MVES (RMVES) algorithm that accounts for the noise effects in the observations by employing chance constraints. These chance constraints in turn control the volume of the resulting simplex. Under the Gaussian noise assumption, the chance-constrained MVES problem can be formulated into a deterministic nonlinear program. The problem can then be conveniently handled by alternating optimization, in which each subproblem involved is handled by using sequential quadratic programming solvers. The proposed RMVES is compared with several existing benchmark algorithms, including its predecessor, the MVES algorithm. Monte Carlo simulations and real hyperspectral data experiments are presented to demonstrate the efficacy of the proposed RMVES algorithm.
Nonnegative matrix factorization (NMF) has become a very popular technique in
machine learning because it automatically extracts meaningful features through
a sparse and part-based representation. However, NMF has the drawback of being
highly ill-posed, that is, there typically exist many different but equivalent
factorizations. In this paper, we introduce a completely new way to obtaining
more well-posed NMF problems whose solutions are sparser. Our technique is
based on the preprocessing of the nonnegative input data matrix, and relies on
the theory of M-matrices and the geometric interpretation of NMF. This approach
provably leads to optimal and sparse solutions under the separability
assumption of Donoho and Stodden (NIPS, 2003), and, for rank-three matrices,
makes the number of exact factorizations finite. We illustrate the
effectiveness of our technique on several image datasets.
Imaging spectrometers measure electromagnetic energy scattered in their
instantaneous field view in hundreds or thousands of spectral channels with
higher spectral resolution than multispectral cameras. Imaging spectrometers
are therefore often referred to as hyperspectral cameras (HSCs). Higher
spectral resolution enables material identification via spectroscopic analysis,
which facilitates countless applications that require identifying materials in
scenarios unsuitable for classical spectroscopic analysis. Due to low spatial
resolution of HSCs, microscopic material mixing, and multiple scattering,
spectra measured by HSCs are mixtures of spectra of materials in a scene. Thus,
accurate estimation requires unmixing. Pixels are assumed to be mixtures of a
few materials, called endmembers. Unmixing involves estimating all or some of:
the number of endmembers, their spectral signatures, and their abundances at
each pixel. Unmixing is a challenging, ill-posed inverse problem because of
model inaccuracies, observation noise, environmental conditions, endmember
variability, and data set size. Researchers have devised and investigated many
models searching for robust, stable, tractable, and accurate unmixing
algorithms. This paper presents an overview of unmixing methods from the time
of Keshava and Mustard's unmixing tutorial [1] to the present. Mixing models
are first discussed. Signal-subspace, geometrical, statistical, sparsity-based,
and spatial-contextual unmixing algorithms are described. Mathematical problems
and potential solutions are described. Algorithm characteristics are
illustrated experimentally.
The purpose of this correspondence is to generalize a result by Donoho and Huo and Elad and Bruckstein on sparse representations of signals in a union of two orthonormal bases for RN. We consider general (redundant) dictionaries for RN, and derive sufficient conditions for having unique sparse representations of signals in such dictionaries. The special case where the dictionary is given by the union of L≥2 orthonormal bases for RN is studied in more detail. In particular, it is proved that the result of Donoho and Huo, concerning the replacement of the ℓ0 optimization problem with a linear programming problem when searching for sparse representations, has an analog for dictionaries that may be highly redundant.
. We present computational experience with a branch-and-cut algorithm to solve quadratic programming problems where there is an upper bound on the number of positive variables. Such problems arise in financial applications. The algorithm solves the largest real-life problems in a few minutes of run-time. 1 Introduction. We are interested in optimization problems QMIP of the form: min x T Qx + c T x s.t. Ax b (1) jsupp(x)j K (2) 0 x j u j ; all j (3) where x is an n-vector, Q is a symmetric positive-semidefinite matrix, supp(x) = fj : x j ? 0g and K is a positive integer. Problems of this type are of interest in portfolio optimization. Briefly, variables in the problem correspond to commodities to be bought, the objective is a measure of "risk", the constraints (1) prescribe levels of "performance", and constraint (2) specifies that not too many 1 different types of commodities can be chosen. All data is derived from statistical information. A good deal of previous work ha...
Finding solutions to least-squares problems with low cardinality has found many applications, including portfolio optimization, subset selection in statistics, and inverse problems in signal processing. Although most works consider local approaches that scale with high-dimensional problems, some others address its global optimization via mixed integer programming (MIP) reformulations. We propose dedicated branch-and-bound methods for the exact resolution of moderate-size, yet difficult, sparse optimization problems, through three possible formulations: cardinality-constrained and cardinality-penalized least-squares, and cardinality minimization under quadratic constraints. A specific tree exploration strategy is built. Continuous relaxation problems involved at each node are reformulated as ℓ1-norm-based optimization problems, for which a dedicated algorithm is designed. The obtained certified solutions are shown to better estimate sparsity patterns than standard methods on simulated variable selection problems involving highly correlated variables. Problem instances selecting up to 24 components among 100 variables, and up to 15 components among 1000 variables, can be solved in less than 1000 s. Unguaranteed solutions obtained by limiting the computing time to 1s are also shown to provide competitive estimates. Our algorithms strongly outperform the CPLEX MIP solver as the dimension increases, especially for quadratically-constrained problems. The source codes are made freely available online.
We propose a new method for estimation in linear models. The ‘lasso’ minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree‐based models are briefly described.
Sparse approximation addresses the problem of approximately fitting a linear model with a solution having as few non-zero components as possible. While most sparse estimation algorithms rely on suboptimal formulations, this work studies the performance of exact optimization of l
0
-norm-based problems through Mixed-Integer Programs (MIPs). Nine different sparse optimization problems are formulated based on l
1
, l
2
or l
∞
data misfit measures, and involving whether constrained or penalized formulations. For each problem, MIP reformulations allow exact optimization, with optimality proof, for moderate-size yet difficult sparse estimation problems. Algorithmic efficiency of all formulations is evaluated on sparse deconvolution problems. This study promotes error-constrained minimization of the l
0
norm as the most efficient choice when associated with l
1
and l
∞
misfits, while the l
2
misfit is more efficiently optimized with sparsity-constrained and sparsity-penalized problems. Exact l
0
-norm optimization is shown to outperform classical methods in terms of solution quality, both for over- and underdetermined problems. Numerical simulations emphasize the relevance of the different lp fitting possibilities as a function of the noise statistical distribution. Such exact approaches are shown to be an efficient alternative, in moderate dimension, to classical (suboptimal) sparse approximation algorithms with l
2
data misfit. They also provide an algorithmic solution to less common sparse optimization problems based on l
1
and l
∞
misfits. For each formulation, simulated test problems are proposed where optima have been successfully computed. Data and optimal solutions are made available as potential benchmarks for evaluating other sparse approximation methods.
Non-negative matrix factorization (NMF) is a recently developed technique for finding parts-based, linear representations of non-negative data. Although it has successfully been applied in several applications, it does not always result in parts-based representations. In this paper, we show how explicitly incorporating the notion of 'sparseness' improves the found decompositions. Additionally, we provide complete MATLAB code both for standard NMF and for our extension. Our hope is that this will further the application of these methods to solving novel data-analysis problems.
At the intersection of mathematics, engineering, and computer science sits the thriving field of compressive sensing. Based on the premise that data acquisition and compression can be performed simultaneously, compressive sensing finds applications in imaging, signal processing, and many other domains. In the areas of applied mathematics, electrical engineering, and theoretical computer science, an explosion of research activity has already followed the theoretical results that highlighted the efficiency of the basic principles. The elegant ideas behind these principles are also of independent interest to pure mathematicians.A Mathematical Introduction to Compressive Sensing gives a detailed account of the core theory upon which the field is build. With only moderate prerequisites, it is an excellent textbook for graduate courses in mathematics, engineering, and computer science. It also serves as a reliable resource for practitioners and researchers in these disciplines who want to acquire a careful understanding of the subject. A Mathematical Introduction to Compressive Sensing uses a mathematical perspective to present the core of the theory underlying compressive sensing.
We investigate fast methods that allow to quickly eliminate variables
(features) in supervised learning problems involving a convex loss
function and a -norm penalty, leading to a potentially substantial
reduction in the number of variables prior to running the supervised
learning algorithm. The methods are not heuristic: they only eliminate
features that are {\em guaranteed} to be absent after solving the
learning problem. Our framework applies to a large class of problems,
including support vector machine classification, logistic regression and
least-squares. The complexity of the feature elimination step is
negligible compared to the typical computational effort involved in the
sparse supervised learning problem: it grows linearly with the number of
features times the number of examples, with much better count if data is
sparse. We apply our method to data sets arising in text classification
and observe a dramatic reduction of the dimensionality, hence in
computational effort required to solve the learning problem, especially
when very sparse classifiers are sought. Our method allows to
immediately extend the scope of existing algorithms, allowing us to run
them on data sets of sizes that were out of their reach before.
We describe a recursive algorithm to compute representations of
functions with respect to nonorthogonal and possibly overcomplete
dictionaries of elementary building blocks e.g. affine (wavelet) frames.
We propose a modification to the matching pursuit algorithm of Mallat
and Zhang (1992) that maintains full backward orthogonality of the
residual (error) at every step and thereby leads to improved
convergence. We refer to this modified algorithm as orthogonal matching
pursuit (OMP). It is shown that all additional computation required for
the OMP algorithm may be performed recursively
Hyperspectral instruments acquire electromagnetic energy scattered within their ground instantaneous field view in hundreds of spectral channels with high spectral resolution. Very often, however, owing to low spatial resolution of the scanner or to the presence of intimate mixtures (mixing of the materials at a very small scale) in the scene, the spectral vectors (collection of signals acquired at different spectral bands from a given pixel) acquired by the hyperspectral scanners are actually mixtures of the spectral signatures of the materials present in the scene. Given a set of mixed spectral vectors, spectral mixture analysis (or spectral unmixing) aims at estimating the number of reference materials, also called endmembers, their spectral signatures, and their fractional abundances. Spectral unmixing is, thus, a source separation problem. This paper presents an overview of the principal research directions in hyperspectral unmixing. The paper is organized into six main topics: i) mixing models, ii) signal subspace identification, iii) geometrical-based spectral unmixing, iv) statistical-based spectral unmixing, v) sparse regression based unmixing, and vi) spatial-contextual information. For each topic, we summarize what is the mathematical problem involved and give relevant pointers to state-of-the-art algorithms to address these problems.
Identification algorithms based on the well-known linear least squares methods of gaussian elimination, Cholesky decomposition, classical Gram-Schmidt, modified Gram-Schmidt, Householder transformation, Givens method, and singular value decomposition are reviewed. The classical Gram-Schmidt, modified Gram-Schmidt, and Householder transformation algorithms are then extended to combine structure determination, or which terms to include in the model, and parameter estimation in a very simple and efficient manner for a class of multivariate discrete-time non-linear stochastic systems which are linear in the parameters.
Non-negative sparse coding is a method for decomposing multivariate data into non-negative sparse components. We briefly describe the motivation behind this type of data representation and its relation to standard sparse coding and non-negative matrix factorization. We then give a simple yet efficient multiplicative algorithm for finding the optimal values of the hidden components. In addition, we show how the basis vectors can be learned from the observed data. Simulations demonstrate the effectiveness of the proposed method.
A signal processing perspective on hyperspectral unmixing: Insights from remote sensing
- Wing-Kin
- José M Ma
- Tsung-Han Bioucas-Dias
- Nicolas Chan
- Paul Gillis
- Antonio J Gader
- Plaza
A comparison of block pivoting and interior-point algorithms for linear least squares problems with nonnegative variables