Fast Variational Sparse Bayesian Learning With Automatic Relevance Determination for Superimposed Signals
ABSTRACT In this work, a new fast variational sparse Bayesian learning (SBL) approach with automatic relevance determination (ARD) is proposed. The sparse Bayesian modeling, exemplified by the relevance vector machine (RVM), allows a sparse regression or classification function to be constructed as a linear combination of a few basis functions. It is demonstrated that, by computing the stationary points of the variational update expressions with noninformative (ARD) hyperpriors, a fast version of variational SBL can be constructed. Analysis of the computed stationary points indicates that SBL with Gaussian sparsity priors and noninformative hyperpriors corresponds to removing components with signal-to-noise ratio below a 0 dB threshold; this threshold can also be adjusted to significantly improve the convergence rate and sparsity of SBL. It is demonstrated that the pruning conditions derived for fast variational SBL coincide with those obtained for fast marginal likelihood maximization; moreover, the parameters that maximize the variational lower bound also maximize the marginal likelihood function. The effectiveness of fast variational SBL is demonstrated with synthetic as well as with real data.
- SourceAvailable from: Anita Faul[show abstract] [hide abstract]
ABSTRACT: The `sparse Bayesian' modelling approach, as exempli ed by the `relevance vector machine ', enables sparse classi cation and regression functions to be obtained by linearlyweighting a small number of xed basis functions from a large dictionary of potential candidates. Such a model conveys a number of advantages over the related and very popular `support vector machine', but the necessary `training' procedure | optimisation of the marginal likelihood function | is typically much slower. We describe a new and highly accelerated algorithm which exploits recently-elucidated properties of the marginal likelihood function to enable maximisation via a principled and ecient sequential addition and deletion of candidate basis functions.09/2002;
- Artificial Intelligence. 10/2002;
Conference Proceeding: Space-alternating attribute-distributed sparse learning[show abstract] [hide abstract]
ABSTRACT: The paper proposes a new variational Bayesian algorithm for multivariate regression with attribute-distributed or dimensionally distributed data. Compared to the existing approaches the proposed algorithm exploits the variational version of the Space-Alternating Generalized Expectation-Maximization (SAGE) algorithm that by means of admissible hidden data - an analog of the complete data in the EM framework - allows parameters of a single agent to be updated assuming that parameters of the other agents are fixed. This allows learning to be implemented in a distributed fashion by sequentially updating the agents one after another. Inspired by Bayesian sparsity techniques, the algorithm also introduces constraints on the agent parameters via parametric priors. This adds a mechanism for pruning irrelevant agents, as well as for minimizing the effect of overfitting. Using synthetic data, as well as measurement data from the UCI Machine Learning Repository it is demonstrated that the proposed algorithm outperforms existing solutions both in the achieved mean-square error (MSE), as well as in convergence speed due to the ability to sparsify noninformative agents, while at the same time allowing distributed implementation and flexible agent update protocols.Cognitive Information Processing (CIP), 2010 2nd International Workshop on; 07/2010