-
[show abstract]
[hide abstract]
ABSTRACT: We consider transmission over a binary-input additive white Gaussian noise
channel using low-density parity-check codes. One of the most popular
techniques for decoding low-density parity-check codes is the linear
programming decoder. In general, the linear programming decoder is suboptimal.
I.e., the word error rate is higher than the optimal, maximum a posteriori
decoder.
In this paper we present a systematic approach to enhance the linear program
decoder. More precisely, in the cases where the linear program outputs a
fractional solution, we give a simple algorithm to identify frustrated cycles
which cause the output of the linear program to be fractional. Then adding
these cycles, adaptively to the basic linear program, we show improved word
error rate performance.
05/2011;
-
[show abstract]
[hide abstract]
ABSTRACT: We present and study linear programming based detectors for two-dimensional
intersymbol interference channels. Interesting instances of two-dimensional
intersymbol interference channels are magnetic storage, optical storage and
Wyner's cellular network model.
We show that the optimal maximum a posteriori detection in such channels
lends itself to a natural linear programming based sub-optimal detector. We
call this the Pairwise linear program detector. Our experiments show that the
Pairwise linear program detector performs poorly. We then propose two methods
to strengthen our detector. These detectors are based on systematically
enhancing the Pairwise linear program. The first one, the Block linear program
detector adds higher order potential functions in an {\em exhaustive} manner,
as constraints, to the Pairwise linear program detector. We show by experiments
that the Block linear program detector has performance close to the optimal
detector. We then develop another detector by
{\em adaptively} adding frustrated cycles to the Pairwise linear program
detector. Empirically, this detector also has performance close to the optimal
one and turns out to be less complex then the Block linear program detector.
02/2011;
-
[show abstract]
[hide abstract]
ABSTRACT: Inference and learning of graphical models are both well-studied problems in statistics and machine learning that have found many applications in science and engineering. However, exact inference is intractable in general graphical models, which suggests the problem of seeking the best approximation to a collection of random variables within some tractable family of graphical models. In this paper, we focus our attention on the class of planar Ising models, for which inference is tractable using techniques of statistical physics [Kac and Ward; Kasteleyn]. Based on these techniques and recent methods for planarity testing and planar embedding [Chrobak and Payne], we propose a simple greedy algorithm for learning the best planar Ising model to approximate an arbitrary collection of binary random variables (possibly from sample data). Given the set of all pairwise correlations among variables, we select a planar graph and optimal planar Ising model defined on this graph to best approximate that set of correlations. We demonstrate our method in some simulations and for the application of modeling senate voting records. Comment: 11 pages, 4 figures, Submitted to 14th International Conference on Artificial Intelligence and Statistics (AISTATS 2011)
11/2010;
-
[show abstract]
[hide abstract]
ABSTRACT: We propose an optimization approach to design cost-effective electrical power transmission networks. That is, we aim to select both the network structure and the line conductances (line sizes) so as to optimize the trade-off between network efficiency (low power dissipation within the transmission network) and the cost to build the network. We begin with a convex optimization method based on the paper ``Minimizing Effective Resistance of a Graph'' [Ghosh, Boyd \& Saberi]. We show that this (DC) resistive network method can be adapted to the context of AC power flow. However, that does not address the combinatorial aspect of selecting network structure. We approach this problem as selecting a subgraph within an over-complete network, posed as minimizing the (convex) network power dissipation plus a non-convex cost on line conductances that encourages sparse networks where many line conductances are set to zero. We develop a heuristic approach to solve this non-convex optimization problem using: (1) a continuation method to interpolate from the smooth, convex problem to the (non-smooth, non-convex) combinatorial problem, (2) the majorization-minimization algorithm to perform the necessary intermediate smooth but non-convex optimization steps. Ultimately, this involves solving a sequence of convex optimization problems in which we iteratively reweight a linear cost on line conductances to fit the actual non-convex cost. Several examples are presented which suggest that the overall method is a good heuristic for network design. We also consider how to obtain sparse networks that are still robust against failures of lines and/or generators. Comment: 8 pages, 3 figures. To appear in Proc. 49th IEEE Conference on Decision and Control (CDC '10)
04/2010;
-
[show abstract]
[hide abstract]
ABSTRACT: We present a new view of Gaussian belief propagation (GaBP) based on a representation of the determinant as a product over orbits of a graph. We show that the GaBP determinant estimate captures totally backtracking orbits of the graph and consider how to correct this estimate. We show that the missing orbits may be grouped into equivalence classes corresponding to backtrackless orbits and the contribution of each equivalence class is easily determined from the GaBP solution. Furthermore, we demonstrate that this multiplicative correction factor can be interpreted as the determinant of a backtrackless adjacency matrix of the graph with edge weights based on GaBP. Finally, an efficient method is proposed to compute a truncated correction factor including all backtrackless orbits up to a specified length. Comment: 8 pages, 3 figures. To appear, ICML '09
04/2009;
-
[show abstract]
[hide abstract]
ABSTRACT: Gaussian belief propagation (GaBP) is an iterative message-passing algorithm for inference in Gaussian graphical models. It is known that when GaBP converges it converges to the correct MAP estimate of the Gaussian random vector and simple sufficient conditions for its convergence have been established. In this paper we develop a double-loop algorithm for forcing convergence of GaBP. Our method computes the correct MAP estimate even in cases where standard GaBP would not have converged. We further extend this construction to compute least-squares solutions of over-constrained linear systems. We believe that our construction has numerous applications, since the GaBP algorithm is linked to solution of linear systems of equations, which is a fundamental problem in computer science and engineering. As a case study, we discuss the linear detection problem. We show that using our new construction, we are able to force convergence of Montanari's linear detection algorithm, in cases where it would originally fail. As a consequence, we are able to increase significantly the number of users that can transmit concurrently. Comment: In the IEEE International Symposium on Information Theory (ISIT) 2009, Seoul, South Korea, July 2009
01/2009;
-
[show abstract]
[hide abstract]
ABSTRACT: This paper presents recursive cavity modeling--a principled, tractable approach to approximate, near-optimal inference for large Gauss-Markov random fields. The main idea is to subdivide the random field into smaller subfields, constructing cavity models which approximate these subfields. Each cavity model is a concise, yet faithful, model for the surface of one subfield sufficient for near-optimal inference in adjacent subfields. This basic idea leads to a tree-structured algorithm which recursively builds a hierarchy of cavity models during an "upward pass" and then builds a complementary set of blanket models during a reverse "downward pass." The marginal statistics of individual variables can then be approximated using their blanket models. Model thinning plays an important role, allowing us to develop thinned cavity and blanket models thereby providing tractable approximate inference. We develop a maximum-entropy approach that exploits certain tractable representations of Fisher information on thin chordal graphs. Given the resulting set of thinned cavity models, we also develop a fast preconditioner, which provides a simple iterative method to compute optimal estimates. Thus, our overall approach combines recursive inference, variational learning and iterative estimation. We demonstrate the accuracy and scalability of this approach in several challenging, large-scale remote sensing problems.
IEEE Transactions on Image Processing 02/2008; 17(1):70-83. · 3.04 Impact Factor
-
IEEE Transactions on Signal Processing. 01/2008; 56:1916-1930.
-
IEEE Transactions on Signal Processing. 01/2008; 56:4621-4634.
-
[show abstract]
[hide abstract]
ABSTRACT: We develop a general framework for MAP estimation in discrete and Gaussian graphical models using Lagrangian relaxation techniques. The key idea is to reformulate an intractable estimation problem as one defined on a more tractable graph, but subject to additional constraints. Relaxing these constraints gives a tractable dual problem, one defined by a thin graph, which is then optimized by an iterative procedure. When this iterative optimization leads to a consistent estimate, one which also satisfies the constraints, then it corresponds to an optimal MAP estimate of the original model. Otherwise there is a ``duality gap'', and we obtain a bound on the optimal solution. Thus, our approach combines convex optimization with dynamic programming techniques applicable for thin graphs. The popular tree-reweighted max-product (TRMP) method may be seen as solving a particular class of such relaxations, where the intractable graph is relaxed to a set of spanning trees. We also consider relaxations to a set of small induced subgraphs, thin subgraphs (e.g. loops), and a connected tree obtained by ``unwinding'' cycles. In addition, we propose a new class of multiscale relaxations that introduce ``summary'' variables. The potential benefits of such generalizations include: reducing or eliminating the ``duality gap'' in hard problems, reducing the number or Lagrange multipliers in the dual problem, and accelerating convergence of the iterative optimization procedure.
10/2007;
-
[show abstract]
[hide abstract]
ABSTRACT: We develop a novel approach to approximate a specified collection of marginal distributions on subsets of variables by a globally consistent distribution on the entire collection of variables. In general, the specified marginal distributions may be inconsistent on overlapping subsets of variables. Our method is based on maximizing entropy over an exponential family of graphical models, subject to divergence constraints on small subsets of variables that enforce closeness to the specified marginals. The resulting optimization problem is convex, and can be solved efficiently using a primal-dual interior-point algorithm. Moreover, this framework leads naturally to a solution that is a sparse graphical model.
Statistical Signal Processing, 2007. SSP '07. IEEE/SP 14th Workshop on; 09/2007
-
Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007; 01/2007
-
Journal of Machine Learning Research - Proceedings Track. 01/2007; 2:203-210.
-
Journal of Machine Learning Research. 01/2006; 7:2031-2064.
-
Advances in Neural Information Processing Systems 18 [Neural Information Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Canada]; 01/2005
-
[show abstract]
[hide abstract]
ABSTRACT: We consider a class of multiscale Gaussian models on pyramidally structured graphs. While such models have been considered in the past, very recent advances in inference methods for graphical models not only yield additional motivation for this class of models but also bring techniques that lead to new and powerful algorithms. We provide a brief summary of these recent advances – including so-called walk-sum analysis, methods based on Lagrangian relaxation, and a new method for “low-rank,” wavelet-based, unbiased estimation of error variances – and then adapt and apply them to problems of estimation for pyramidal models. We demonstrate that our models not only capture long-range dependencies but that they also have the property that conditioned on neighboring scales, the correlation behavior within a scale is dramatically compressed. This leads to algorithms resembling multipole methods for solving partial differential equations in which we alternate computations across-scale (using an embedded tree in the pyramidal graph) with local updates within each scale. Not only are these algorithms guaranteed to converge to the correct answers but they also lead to new, adaptive methods for choosing embedded trees and subgraphs to achieve rapid convergence. This approach also leads to a solution to the so-called re-estimation problem in which we seek to update an estimate rapidly after local changes are made to the prior model or to the available data. In addition, by using a consistent probabilistic model across as well as within scales, we are able both to exploit low-rank variance estimation methods and to develop efficient iterative algorithms for parameter estimation.
Computer Methods in Applied Mechanics and Engineering.
