Explaining Variational Approximations

Abstract

Variational approximations facilitate approximate inference for the parameters in complex statistical models and provide fast, deterministic alternatives to Monte Carlo methods. However, much of the contemporary literature on variational approximations is in Computer Science rather than Statistics, and uses terminology, notation and examples from the former field. In this article we explain variational approximation in statistical terms. In particular, we illustrate the ideas of variational approximation using examples that are familiar to statisticians.

Full text

Page 1

University of Wollongong
Research Online
Centre for Statistical & Survey Methodology
Working Paper Series
Faculty of Informatics
2009
Explaining Variational Approximations
J. T. Ormerod
University of Wollongong, johno@uow.edu.au
M. P. Wand
University of Wollongong, mwand@uow.edu.au
Research Online is the open access institutional repository for the
University of Wollongong. For further information contact Manager
Repository Services: morgan@uow.edu.au.
Recommended Citation
Ormerod, J. T. and Wand, M. P., Explaining Variational Approximations, Centre for Statistical and Survey Methodology, University
of Wollongong, Working Paper 07-09, 2009, 23p.
http://ro.uow.edu.au/cssmwp/27

Page 2

Copyright © 2008 by the Centre for Statistical & Survey Methodology, UOW. Work in progress,
no part of this paper may be reproduced without permission from the Centre.

Centre for Statistical & Survey Methodology, University of Wollongong, Wollongong NSW
2522. Phone +61 2 4221 5435, Fax +61 2 4221 4845. Email: anica@uow.edu.au




Centre for Statistical and Survey Methodology

The University of Wollongong



Working Paper


07-09


Explaining Variational Approximations.


Ormerod, J.T. and Wand, M.P.

Page 3

Explaining Variational Approximations
BY J.T. ORMEROD & M.P. WAND1
1st October, 2009
SUMMARY
Variationalapproximationsfacilitateapproximateinferencefortheparametersincom-
plex statistical models and provide fast, deterministic alternatives to Monte Carlo meth-
ods. However, much of the contemporary literature on variational approximations is in
Computer Science rather than Statistics, and uses terminology, notation and examples
from the former field. In this article we explain variational approximation in statistical
terms. In particular, we illustrate the ideas of variational approximation using examples
that are familiar to statisticians.
Keywords: Bayesian inference; Bayesian networks; Directed acyclic graphs; Generalized
linear mixed models; Kullback-Leibler divergence; Linear mixed models.
1 Introduction
Variational approximations is a body of deterministic techniques for making approximate
inferenceforparametersincomplexstatisticalmodels. ItisnowpartofmainstreamCom-
puter Science methodology, where it enjoys use in elaborate problems such as speech
recognition, document retrieval and genetic linkage analysis (Jordan, 2004). Summaries
ofcontemporaryvariationalapproximationscanbefoundinJordan, Ghahramani, Jaakkola
& Saul (1999), Jordan (2004), Titterington (2004) and Bishop (2006, Chapter 10). In 2008,
a variational approximation-based software package named Infer.NET (Minka, Winn,
Guiver & Kannan, 2008) emerged with claims of being able to handle a wide variety of
statistical problems.
The name ‘variational approximations’ has its roots in the mathematical topic known
as variational calculus. Variational calculus is concerned with the problem of optimizing
a functional over a class of functions on which that functional depends. Approximate
solutions arise when the class of functions is restricted in some way – usually to enhance
tractability.
Despite their statistical overtones, variational approximations are not well-known
within the statistical community. In particular, they are overshadowed by Monte Carlo
methods, especially Markov chain Monte Carlo (MCMC), for performing approximate
inference, as well as Laplace approximation methods. Variational approximations are a
much faster alternative to MCMC, especially for large models, and are a richer class of
methods than the Laplace variety. They are, however, limited in their approximation ac-
curacy – as opposed to MCMC which can be made arbitrarily accurate through increases
in the Monte Carlo sample sizes. In the interests of brevity, we will not discuss the quality
of variational approximations in any detail. Jordan (2004) and Titterington (2004) point
to some relevant literature on variational approximation accuracy.
In the statistics literature, variational approximations are beginning to have a pres-
ence. Examples include Teschendorff et al. (2005), McGrory & Titterington (2007) and
1J.T. Ormerod is a Post-doctoral Research Fellow and M.P. Wand is a Research Professor in Statistics at the
Centre for Statistical and Survey Methodology, School of Mathematics and Applied Statistics, University of
Wollongong, Wollongong 2522, Australia. We are grateful to an associate editor, two referees for their sug-
gestions for improvement. We also thank Christel Faes, Doug Simpson, Mike Titterington and Shen Wang
for helpful comments. This research was partially supported by Australian Research Council Discovery
Project DP0877055.
1

Page 4

McGrory, Titterington, Reeves&Pettitt(2009)onnewvariationalapproximationmethod-
ology for particular applications, and Hall, Humphreys & Titterington (2002) and Wang
& Titterington (2006) on the statistical properties of estimators obtained via variational
approximation.
In this article we explain variational approximation in terms that are familiar to a sta-
tistical readership. Most of our exposition involves working through several illustrative
examples, starting with what is perhaps the most basic: inference from a Normal random
sample. Other contexts that are seen to benefit from variational approximation include
Bayesian generalized linear models, Bayesian linear mixed models and non-Bayesian
generalized linear mixed models. It is anticipated that a statistically literate reader who
works through all of the examples will have gained a good understanding of variational
approximations.
Variational approximations can be useful for both likelihood-based and Bayesian in-
ference. However, their utility is much greater for Bayesian inference where intractable
calculus problems abound. Hence, most of our description of variational approxima-
tions is for Bayesian inference. It is also worth noting that situations in which variational
approximations are useful closely correspond to situations where MCMC is useful.
It is helpful, although not necessary, to work with directed acyclic graph (DAG) de-
pictions of Bayesian statistical models. The nodes of the DAG correspond to random
variables or random vectors in the Bayesian model, and the directed edges convey con-
ditional independence. Because of this connection with Bayesian (hierarchical) models,
DAGs with random nodes are known as Bayesian networks in the Computer Science lit-
erature. Figure 1 provides DAGs corresponding to the Bayesian Poisson mixed model
(with notation as defined in Section 1.1):
Yij|Ui
ind.
∼ Poisson(eβ0+Ui), i = 1,2,3; j = 1,2,Ui|σ2
U
ind.
∼ N(0,σ2
U),
β0∼ N(0,σ2
β0),σ2
U∼ IG(A,B)
for constants σ2
β0,A,B > 0.
(1)
The DAG on the left side of Figure 1 has a separate node for each scalar random variable
and each constant. The arrows convey the conditional dependence structure. On the
right side the constant nodes are suppressed and two of the nodes correspond to the
random vectors u ≡ (U1,U2,U3) and y = (Y11,...,Y32).
Section 2 explains the most common variant of variational approximation, which we
call the density transform approach. A different type, the tangent transform approach, is
explained in Section 3. Sections 2 and 3 focus exclusively on Bayesian inference. In Sec-
tion 4 we point out that the same ideas transfer to frequentist contexts. Some concluding
remarks are made in Section 5.
1.1Notation
Integrals without limits or subscripts are assumed to be over the entire space of the in-
tegrand argument. If P is a logical condition then I(P) = 1 if P is true and I(P) = 0
if P is false. We use Φ and φ to denote the standard normal distribution function and
density function, respectively. The Gamma function, denoted by Γ, is given by Γ(x) =
?∞
bold-faced version of the letter for that variable. Round brackets will be used to denote
the entries of column vectors. For example x = (x1,...,xn) denotes a n × 1 vector with
entries x1,...,xn. Scalar functions applied to vectors are evaluated element-wise. For
example,
exp(a1, a2, a3) ≡ (exp(a1), exp(a2), exp(a3)).
Similarly, (a1, a2, a3)(b1,b2,b3)≡ (ab1
A and B is denoted by A?B. We use 1dto denote the d×1 column vector with all entries
0ux−1e−udu and the digamma function, denoted by ψ, is given by ψ(x) =
Column vectors with entries consisting of sub-scripted variables are denoted by a
d
dxlogΓ(x).
1, ab2
2, ab3
3). The element-wise product of two matrices
2

Page 5

σ σ2
U1
U2
U3
Y11
Y12
Y21
Y22
Y31
Y32
β β0
G GA
G GB
G Gσ σβ β0
2
σ σ2
u
y
β β0
Figure 1: DAGs corresponding to the Bayesian Poisson regression model (1). Left: the large nodes
correspond to scalar random variables in the model. The smaller nodes correspond to constants
and the observed data are shaded. Right: abbreviated DAG for the same model. The constants
are suppressed and the nodes u and y correspond to random vectors containing the Uiand yij
respectively.
equal to 1. The norm of a column vector v, defined to be
d × 1 vector a we let diag(a) denote the d × d diagonal matrix containing the the entries
of a along the main diagonal. For a d×d square matrix A, we let diagonal(A) denote the
d × 1 vector containing the diagonal entries of A. For square matrices A1,...,Arwe let
blockdiag(A1,...,Ar) denote the block diagonal matrix, with ith block equal to Ai.
Thedensityfunctionofarandomvectoruisdenotedbyp(u). Theconditionaldensity
of u given v is denoted by p(u|v). The covariance matrix of u is denoted by Cov(u). A
d×1 random vector x has a Multivariate Normal distribution with parameters µ and Σ,
denoted by x ∼ N(µ,Σ), if its density function is
p(x) = (2π)−d/2|Σ|−1/2exp{−1
A random variable x has an Inverse Gamma distribution with parameters A,B > 0,
denoted by x ∼ IG(A,B) if its density function is p(x) = BAΓ(A)−1x−A−1e−B/x, x > 0.
A random vector x = (x1,...,xK) has a Dirichlet distribution with parameter vector
α = (α1,...,αK), where each αk> 0, if its density function is
? ?
0,
√vTv is denoted by ?v?. For a
2(x − µ)TΣ−1(x − µ)}.
p(x) =
Γ(?K
k=1αk)/?K
k=1Γ(αk)
??K
k=1xαk−1
k
,
?K
k=1xk= 1,
otherwise.
We write x ∼ Dirichlet(α). If yihas distribution Difor each 1 ≤ i ≤ n, and the yiare
independent, then we write yi
∼ Di.
ind.
2 Density Transform Approach
The density transform approach to variational approximation involves approximation of
posterior densities by other densities for which inference is more tractable. The approxi-
mations are guided by the notion of Kullback-Leibler divergence, which we now explain.
3