Recovering 3D human pose from monocular images

Abstract

We describe a learning-based method for recovering 3D human body pose from single images and monocular image sequences. Our approach requires neither an explicit body model nor prior labeling of body parts in the image. Instead, it recovers pose by direct nonlinear regression against shape descriptor vectors extracted automatically from image silhouettes. For robustness against local silhouette segmentation errors, silhouette shape is encoded by histogram-of-shape-contexts descriptors. We evaluate several different regression methods: ridge regression, relevance vector machine (RVM) regression, and support vector machine (SVM) regression over both linear and kernel bases. The RVMs provide much sparser regressors without compromising performance, and kernel bases give a small but worthwhile improvement in performance. The loss of depth and limb labeling information often makes the recovery of 3D pose from single silhouettes ambiguous. To handle this, the method is embedded in a novel regressive tracking framework, using dynamics from the previous state estimate together with a learned regression value to disambiguate the pose. We show that the resulting system tracks long sequences stably. For realism and good generalization over a wide range of viewpoints, we train the regressors on images resynthesized from real human motion capture data. The method is demonstrated for several representations of full body pose, both quantitatively on independent but similar test data and qualitatively on real image sequences. Mean angular errors of 4-6° are obtained for a variety of walking motions.

Full text

Page 1

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. SUBMITTED FOR REVIEW.1
Recovering 3D Human Pose from
Monocular Images
Ankur Agarwal and Bill Triggs
Abstract—We describe a learning based method for recovering
3D human body pose from single images and monocular image
sequences. Our approach requires neither an explicit body model
nor prior labelling of body parts in the image. Instead, it recovers
pose by direct nonlinear regression against shape descriptor
vectors extracted automatically from image silhouettes. For
robustness against local silhouette segmentation errors, silhouette
shape is encoded by histogram-of-shape-contexts descriptors. We
evaluate several different regression methods: ridge regression,
Relevance Vector Machine (RVM) regression and Support Vector
Machine (SVM) regression over both linear and kernel bases. The
RVMs provide much sparser regressors without compromising
performance, and kernel bases give a small but worthwhile
improvement in performance. Loss of depth and limb labelling
information often makes the recovery of 3D pose from single
silhouettes ambiguous. We propose two solutions to this: the first
embeds the method in a tracking framework, using dynamics
from the previous state estimate to disambiguate the pose;
the second uses a mixture of regressors framework to return
multiple solutions for each silhouette. We show that the resulting
system tracks long sequences stably, and is also capable of
accurately reconstructing 3D human pose from single images,
giving multiple possible solutions in ambiguous cases. For realism
and good generalization over a wide range of viewpoints, we train
the regressors on images resynthesized from real human motion
capture data. The method is demonstrated on a 54-parameter
full body pose model, both quantitatively on independent but
similar test data, and qualitatively on real image sequences. Mean
angular errors of 4–5 degrees are obtained — a factor of 3 better
than the current state of the art for the much simpler upper body
problem.
Index Terms—Computer vision, human motion estimation,
machine learning, multivariate regression, Relevance Vector Ma-
chine
I. INTRODUCTION
We consider the problem of estimating and tracking 3D
configurations of complex articulated objects from monocular
images, e.g. for applications requiring 3D human body pose
and hand gesture analysis. There are two main schools of
thought on this. Model-based approaches presuppose an ex-
plicitly known parametric body model, and estimate the pose
either by directly inverting the kinematics (which has many
possible solutions and requires known image positions for each
body part) [28], or by numerically optimizing some form of
model-image correspondence metric over the pose variables,
using a forward rendering model to predict the images (which
is expensive and requires a good initialization, and the problem
always has many local minima [25]). An important sub-
case is model-based tracking, which focuses on tracking the
pose estimate from one time step to the next starting from
a known initialization, based on an approximate dynamical
model [9,23]. In contrast, learning based approaches try to
avoid the need for explicit initialization and accurate 3D
modelling and rendering, and to capitalize on the fact that
the set of typical human poses is far smaller than the set of
kinematically possible ones, by estimating (learning) a model
that directly recovers pose estimates from observable image
quantities. In particular, example based methods explicitly
store a set of training examples whose 3D poses are known,
and estimate pose by searching for training image(s) similar
to the given input image, and interpolating from their poses
[5,18,22,27].
In this paper we take a learning based approach, but
instead of explicitly storing and searching for similar training
examples, we use sparse Bayesian nonlinear regression to
distill a large training database into a single compact model
that has good generalization to unseen examples. Given the
high dimensionality and intrinsic ambiguity of the monocular
pose estimation problem, the selection of appropriate image
features and good control of overfitting is critical for success.
We are not aware of previous work on pose estimation that
directly addresses these issues. Our strategy is based on the
sparsification and generalization properties of our nonlinear
regression algorithm, which is a form of the Relevance Vector
Machine (RVM) [29]. RVMs have been used earlier, e.g.
to build kernel regressors for 2D displacement updates in
correlation-based patch tracking [33]. Human pose recovery is
significantly harder — more ill-conditioned and nonlinear and
much higher dimensional — but by selecting a sufficiently rich
set of image descriptors, it turns out that we can still obtain
enough information for successful regression. Loss of depth
and limb labelling information often makes the recovery of
3D pose from single silhouettes ambiguous. We propose two
solutions to this. The first embeds the method in a tracking
framework, using dynamics from the previous state estimate
to disambiguate the pose. The second uses a mixture of
regressors framework to return multiple possible solutions for
each silhouette, allowing accurate pose reconstructions from
single images. When working with a sequence of images, these
solutions are fed as input to a multiple hypothesis tracker to
give the most likely estimate for each time step.
Previous work: There is a good deal of prior work on
human pose analysis, but relatively little on directly learning
3D pose from image measurements. Brand [8] models a
dynamical manifold of human body configurations with a
Hidden Markov Model and learns using entropy minimization,
Athitsos and Sclaroff [4] learn a perceptron mapping between
the appearance and parameter spaces, and Shakhnarovich et al

Page 2

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. SUBMITTED FOR REVIEW.2
[22] use an interpolated-k-nearest-neighbor learning method.
Human pose is hard to ground truth, so most papers in this
area [4,8,18] use only heuristic visual inspection to judge
their results. However Shakhnarovich et al [22] used a human
model rendering package (POSER from Curious Labs) to
synthesize ground-truthed training and test images of 13 d.o.f.
upper body poses with a limited (±40◦) set of random torso
movements and view points. In comparison, our regression
algorithm estimates full body pose and orientation (54 d.o.f.)
— a problem whose high dimensionality would really stretch
the capacity of an example based method such as [22]. Like
[11,22], we used POSER to synthesize a large set of training
and test images from different viewpoints, but rather than using
random synthetic poses, we used poses taken from real human
motion capture sequences. Our results thus relate to real data.
Several publications have used the image locations of the
centre of each body joint as an intermediate representation,
first estimating these centre locations in the image, then
recovering the 3D pose from them. Howe et al [12] develop
a Bayesian learning framework to recover 3D pose from
known centres, based on a training set of pose-centre pairs
obtained from resynthesized motion capture data. Mori &
Malik [18] estimate the centres using shape context image
matching against a set of training images with pre-labelled
centres, then reconstruct 3D pose using the algorithm of
[28]. These approaches show that using 2D joint centres as
an intermediate representation can be an effective strategy,
but we have preferred to estimate pose directly from the
underlying local image descriptors as we feel that this is likely
to prove both more accurate and more robust, also providing
a generic framework for directly estimating and tracking any
prespecified set of parameters from image observations.
As regards tracking, some approaches have learned dynami-
cal models for specific human motions [19,20]. Particle filters
and MCMC methods have been widely used in probabilistic
tracking frameworks e.g. [23,31]. Most of these methods use
an explicit generative model to compute observation likeli-
hoods. We propose a discriminatively motivated framework
in which dynamical state predictions are directly fused with
descriptors computed from the observed image. Our algorithm
is related to Bayesian tracking, but we eliminate the need for
both an explicit body model that is projected to predict image
observations, and a corresponding error model that is used
to evaluate the likelihood of the observed image given this
projection. A brief description of our regression based scheme
is given is [1] and its first extension to resolve ambiguities
using dynamics within the regression is described in [2].
Overview of the approach: We represent 3D body pose by
55-D vectors x including 3 joint angles for each of the 18 ma-
jor body joints. This choice corresponds to the motion capture
data that we use to train the system (details in section II-B).
The input images are reduced to 100-D observation vectors z
that robustly encode the shape of a human image silhouette.
Given a set of labelled training examples {(zi,xi)|i =
1...n}, the RVM learns a smooth reconstruction function
x = r(z), valid over the region spanned by the training
points. The function is a weighted linear combination r(z) ≡
?
disambiguate pose in cases where there are several possible
reconstructions, the functional form is extended to include an
approximate preliminary pose estimate ˇ x, x = r(ˇ x,z). (See
section V.) At each time step, a state estimate ˇ xtis obtained
from the previous two pose vectors using an autoregressive dy-
namical model, and this is used to compute the basis functions,
which now take the form {φk(ˇ x,z)|k = 1...p}. Section
VI gives an alternative method for handling ambiguities by
returning multiple possible 3D configurations corresponding to
a silhouette. The functional form is extended to a probabilistic
mixture p(x) ∼?
Our solutions are well-regularized in the sense that the
weight vectors ak are damped to control over-fitting, and
sparse in the sense that many of them are zero. Sparsity occurs
because the RVM actively selects only the ‘most relevant’
basis functions — the ones that really need to have nonzero
coefficients to complete the regression successfully. A sparse
solution obtained by the RVM allows the system to select
relevant input features (components) in case of a linear basis
(φk(z) = kthcomponent of z). For a kernel basis — φk(z) ≡
K(z,zk) for some kernel function K(zi,zj) and centres zk—
relevant training examples are selected, allowing us to prune
a large training dataset and retain only a minimal subset.
kakφk(z) of a prespecifed set of scalar basis functions
{φk(z)|k = 1...p}. In our tracking framework, to help to
kπkδ(x,rk) allowing each reconstruction
rkto output a different solution.
Organization: §II describes our image descriptors and body
pose representation. §III gives an outline of our regression
methods. §IV details the recovery of 3D pose from single
images using this regression, discussing the RVM’s feature
selection properties but showing that ambiguities in estimating
3D pose from single images cause occasional ‘glitches’ in
the results. §V describes our first solution to this problem:
a tracking based regression framework capable of resolving
these ambiguities, with results from our novel tracker in §V-B.
§VI describes an alternative solution: a mixture of regressors
based approach incorporated in a multiple hypothesis tracker.
Finally, §VII concludes with some discussions and directions
for future work.
II. REPRESENTING IMAGES AND BODY POSES
Directly regressing pose on input images requires a robust,
compact and well-behaved representation of the observed
image information and a suitable parametrization of the body
poses that we wish to recover. To encode the observed images
we use robust descriptors of the shape of the subject’s image
silhouette, and to describe our body pose, we use vectors of
joint angles.
A. Images as Shape Descriptors
Silhouettes: Of the many different image descriptors that
could be used for human pose estimation, and in line with
[4,8], we have chosen to base our system on image silhouettes.
Silhouettes have three main advantages: (i) They can be ex-
tracted moderately reliably from images, at least when robust
background- or motion-based segmentation is available and

Page 3

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. SUBMITTED FOR REVIEW.3
Fig. 1. Different 3D poses can have very similar image observations, causing
the regression from image silhouettes to 3D pose to be inherently multi-valued.
The legs are the arms are reversed in the first two images, for example.
problems with shadows are avoided; (ii) they are insensitive
to irrelevant surface attributes like clothing colour and texture;
(iii) they encode a great deal of useful information about 3D
pose without the need of any labelling information1.
Two factors limit the performance attainable from sil-
houettes: (i) Artifacts such as shadow attachment and poor
background segmentation tend to distort their local form.
This often causes problems when global descriptors such as
shape moments are used (as in [4,8]), as every local error
pollutes each component of the descriptor: to be robust, shape
descriptors need to have good locality. (ii) Silhouettes make
several discrete and continuous degrees of freedom invisible
or poorly visible (see fig. 1). It is difficult to tell frontal
views from back ones, whether a person seen from the side is
stepping with the left leg or the right one, and what are the
exact poses of arms or hands that fall within (are “occluded”
by) the torso’s silhouette. Including interior edge information
within the silhouette [22] is likely to provide a useful degree of
disambiguation in such cases, but is difficult to disambiguate
from, e.g. markings on clothing.
Shape Context Distributions: To improve resistance to seg-
mentation errors and occlusions, we need a robust silhouette
representation. The first requirement for robustness is locality.
Histogramming edge information is a good way to encode
local shape robustly [17,6], so we begin by computing local
descriptors at regularly spaced points on the edge of the
silhouette. We use shape contexts (histograms of local edge
pixels into log-polar bins [6]) to encode silhouette shape quasi-
locally over a range of scales, computing the contexts in local
regions defined by diameter roughly equal to the size of a limb.
In our application we assume that the vertical is preserved,
so to improve discrimination, we do not normalize contexts
with respect to their dominant local orientations as originally
proposed in [6]. The silhouette shape is thus encoded as a
1We do not believe that any representation (Fourier coefficients, etc.)
based on treating the silhouette shape as a continuous parametrized curve is
appropriate for this application: silhouettes frequently change topology (e.g.
when a hand’s silhouette touches the torso’s one), so parametric curve-based
encodings necessarily have discontinuities w.r.t. shape.
−0.2 −0.100.10.2
−0.25
−0.2
−0.15
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
−0.2−0.100.10.2
−0.25
−0.2
−0.15
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
Fig. 2.
all shape context vectors from a training data sequence, with the k-means
centres superimposed. The average-over-human-silhouettes like form arises
because (besides finer distinctions) the context vectors encode approximate
spatial position on the silhouette: a context at the bottom left of the silhouette
receives votes only in its upper right bins, etc. (Centre) The same projection
for the edge-points of a single silhouette (shown on the right).
(Left) The first two principal components of the distribution of
distribution (in fact, as a noisy multibranched curve, but we
treat it as a distribution) in the 60-D shape context space. (In
our implementation, shape contexts contain 12 angular × 5 ra-
dial bins, giving rise to 60 dimensional histograms.) Matching
silhouettes is therefore reduced to matching these distributions
in shape context space. To implement this, a second level of
histogramming is performed: we reduce the distributions of
all points on each silhouette to 100-D histograms by vector
quantizing the shape context space. Silhouette comparison is
thus finally reduced to a comparison of 100-D histograms.
The 100 centre codebook is learned once and for all by
running k-means on the combined set of context vectors
of all of the training silhouettes. See fig. 2. (Other centre
selection methods give similar results.) For a given silhouette,
a 100-D histogram z is built by allowing each of its context
vectors to vote softly into the few centre-classes nearest to
it, and accumulating scores of all context vectors. This soft
voting reduces the effects of spatial quantization, allowing
us to compare histograms using simple Euclidean distance,
rather than, say, Earth Movers Distance [21]. (We have also
tested the normalized cellwise distance ?√p1−√p2?2, with
very similar results.) The histogram-of-shape-contexts scheme
gives us a reasonable degree of robustness to occlusions and
local silhouette segmentation failures, and indeed captures a
significant amount of pose information (see fig. 3).
B. Body Pose as Joint Angles
We recover 3D body pose (including orientation w.r.t. the
camera) as a real 55-D vector x, including 3 joint angles for
each of the 18 major body joints. The subject’s overall azimuth
(compass heading angle) θ can wrap around through 360◦. To
maintain continuity, we actually regress (a,b) = (cosθ,sinθ)
rather than θ, using atan2(b,a) to recover θ from the not-
necessarily-normalized vector returned by regression. So we
have 3×18+1 = 55 parameters.
We stress that our framework is inherently ‘model-free’ and
is independent of the choice of this pose representation. The
system itself has no explicit body model or rendering model,

Page 4

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. SUBMITTED FOR REVIEW.4
50 100150200 250300350400450
50
100
150
200
250
300
350
400
450
50 100150200 250300 350400 450
50
100
150
200
250
300
350
400
450
Fig. 3.
and (right) true 3D poses, for a 483-frame sequence of a person walking
in a decreasing spiral. The light off-diagonal bands that are visible in both
matrices denote regions of comparative similarity linking corresponding poses
on different cycles of the spiral. This indicates that our silhouette descriptors
do indeed capture a significant amount of pose information. (The light SW-
NE ripples in the 3D pose matrix just indicate that the standing-like poses at
the middle of each stride have mid-range joint values, and hence are closer
on average to other poses than the ‘stepping’ ones at the end of strides).
Pairwise similarity matrices for (left) image silhouette descriptors
and no knowledge of the ‘meaning’ of the motion capture
parameters that it is regressing — it simply learns to predict
these from silhouette data. Similarly, we have not sought to
learn a minimal representation of the true human pose degrees
of freedom, but simply to regress the original motion capture
based training format, and our regression methods handle such
redundant output representations without problems.
The motion capture data was taken from the public website
www.ict.usc.edu/graphics/animWeb/ humanoid. Although we
use real motion capture data for joint angles, we do not
have access to the corresponding image silhouettes, so we
currently use a graphics package, POSER from Curious Labs,
to synthesize suitable training images, and also to visualize the
final reconstruction. This does unfortunately involve the use
of a synthetic body model, but we stress that this model is not
part of our system and would not be needed if real motion
capture data with silhouettes were available.
III. REGRESSION METHODS
This section describes the regression methods that we have
evaluated for recovering 3D human body pose from the
above image descriptors. Here we follow standard regression
notation, representing the output pose by real vectors y ∈ Rm
and the input shape as vectors x ∈ Rd.2
For most of the paper, we assume that the relationship
between x and y — which a priori, given the ambiguities
of pose recovery, might be multi-valued and hence relational
rather than functional — can be approximated functionally as
a linear combination as a prespecified set of basis functions:
y =
p
?
k=1
akφk(x) + ? ≡ Af(x) + ?
(1)
2However note that in subsequent sections, outputs (3D-pose vectors) will
be denoted by x ∈ R55and inputs will be instances from either the
observation space, z ∈ R100, or the joint (predicted) state + observation
space, (x?,z?)?∈ R155.
Here, {φk(x)|k = 1...p} are the basis functions, ak
are Rm-valued weight vectors, and ? is a residual er-
ror vector. For compactness, we gather the weight vec-
tors into an m×p weight matrix A ≡ (a1 a2 ··· ap)
and the basis functions into a Rp-valued function f(x) =
(φ1(x)
φ2(x) ··· φp(x))
Af+b, we can include φ(x) ≡ 1 in f.
To train the model (estimate A), we are given a set of
training pairs {(yi,xi)|i = 1...n}. In this paper we will
usually use the Euclidean norm to measure y-space prediction
errors, so the estimation problem is of the form:
?
i=1
where R(−) is a regularizer on A. Gathering the training
points into an m×n output matrix Y ≡ (y1 y2 ··· yn) and
a p×n feature matrix F ≡ (f(x1)
estimation problem takes the form:
??AF − Y?2+ R(A)?
Note that the dependence on {φk(−)} and {xi} is encoded
entirely in the numerical matrix F.
?. To allow for a constant offset
A := argmin
A
n
?
?Af(xi) − yi?2+ R(A)
?
(2)
f(x2) ··· f(xn)), the
A := argmin
A
(3)
A. Ridge Regression
Pose estimation is a high dimensional and intrinsically ill-
conditioned problem, so simple least squares estimation —
setting R(A) ≡ 0 and solving for A in least squares —
typically produces severe overfitting and hence poor general-
ization. To reduce this, we need to add a smoothness constraint
on the learned mapping, for example by including a damping
or regularization term R(A) that penalizes large values in the
coefficient matrix A. Consider the simplest choice, R(A) ≡
λ?A?2, where λ is a regularization parameter. This gives
the ridge regressor, or damped least squares regressor, which
minimizes
?A˜F −˜Y?2:= ?AF − Y?2+ λ?A?2
where˜F ≡ (F
obtained by solving the linear system A˜F =˜Y (i.e.˜F?A?=
˜Y?) for A in least squares3, using QR decomposition or the
normal equations. Ridge solutions are not equivariant under
scaling of inputs, so we usually standardize the inputs (i.e.
scale them to have unit variance) before solving.
λ must be set large enough to control ill-conditioning
and overfitting, but not so large as to cause overdamping
(forcing A towards 0 so that the regressor systematically
underestimates the solution).
(4)
λI) and˜Y ≡ (Y 0). The solution can be
B. Relevance Vector Regression
Relevance Vector Machines (RVMs) [29,30] are a sparse
Bayesian approach to classification and regression. They in-
troduce Gaussian priors on each parameter or group of pa-
rameters, each prior being controlled by its own individual
3In case a constant offset y = Ax + b is included, this vector b must
not be damped and hence the system takes the form (A b)˜F =˜Y where
˜F ≡
10
„FλI
«
and˜Y ≡`Y0´.

Page 5

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. SUBMITTED FOR REVIEW.5
RVM Training Algorithm
1) Initialize A with ridge regression. Initialize the run-
ning scale estimates ascale= ?a? for the components
or vectors a.
2) Approximatethe
ν log?a?
“quadratic bridges”, the gradients of which match
at ascale. I.e. the penalty terms take the form
ν
2(a/ascale)2+ const.
(One can set const = ν(log?ascale?−1
function values at ascale, but this value is irrelevant for
the least squares minimization.)
3) Solve the resulting linear least squares problem in A.
4) Remove any components a that have become zero,
update the scale estimates ascale= ?a?, and continue
from 2 until convergence.
penaltyterms with
2) to match the
Fig. 4.An outline of our RVM training algorithm.
scale hyperparameter. Integrating out the hyperpriors (which
can be done analytically) gives singular, highly nonconvex
total priors of the form p(a) ∼ ?a?−νfor each parameter or
parameter group a, where ν is a hyperprior parameter. Taking
log likelihoods gives an equivalent regularization penalty of
the form R(a) = ν log?a?. Note the effect of this penalty.
If ?a? is large, the ‘regularizing force’ dR/da ∼ ν/?a?
is small so the prior has little effect on a. But the smaller
?a? becomes, the greater the regularizing force becomes.
At a certain point, the data term no longer suffices to hold
the parameter at a nonzero value against this force, and the
parameter rapidly converges to zero. Hence, the fitted model is
sparse — the RVM automatically selects a subset of ‘relevant’
basis functions that suffices to describe the problem. The
regularizing effect is invariant to rescalings of f() or Y. (E.g.
scaling f → αf forces a rescaling A → A/α with no change
in residual error, so the regularization forces 1/?a? ∝ α
track the data-term gradient AFF?∝ α correctly). ν serves
both as a sparsity parameter and as a scale-free regularization
parameter. The complete RVM model is highly nonconvex
with many local minima and optimizing it can be problematic
because relevant parameters can easily become accidentally
‘trapped’ in the singularity at zero. However, in practice this
does not prevent RVMs from giving useful results. Setting ν to
optimize the estimation error on a validation set, one typically
finds that RVMs give sparse regressors with performance very
similar to the much denser ones from analogous methods with
milder priors.
To train our RVMs, we do not use Tipping’s algorithm
[29], but rather a continuation method based on succes-
sively approximating the ν log?a? regularizers with quadratic
“bridges” ν (?a?/ascale)2chosen to match the prior gradient at
ascale, a running scale estimate for a (see fig. 5). The bridging
changes the apparent curvature if the cost surfaces, allowing
parameters to pass through zero if they need to, with less risk
of premature trapping. The algorithm is sketched in figure 4.
We have tested both componentwise priors, R(A) =
ν?
jklog|Ajk|, which effectively allow a different set of
relevant basis functions to be selected for each dimension
–4
–3
–2
–1
0
a
Fig. 5.
These are introduced to prevent parameters from prematurely becoming
trapped at zero. (See text.)
“Quadratic bridge” approximations to the ν log?a? regularizers.
of y, and columnwise ones, R(A) = ν?
relevant basis functions for all components of y. Both priors
give similar results, but one of the main advantages of sparsity
is in reducing the number of basis functions (support features
or examples) that need to be evaluated, so in the experiments
shown we use columnwise priors. Hence, we minimize
klog?ak? where
ak is the kthcolumn of A, which select a common set of
?AF − Y?2+ ν
?
k
log?ak?
(5)
C. Choice of Basis
We tested two kinds of regression bases f(x). (i) Lin-
ear bases, f(x) ≡ x, simply return the input vector, so
the regressor is linear in x and the RVM selects rele-
vant features (components of x). (ii) Kernel bases, f(x) =
(K(x,x1) ··· K(x,xn))
K(x,xi) instantiated at training examples xi, so the RVM
effectively selects relevant examples. Our experiments with
various kernels and combinations of kernels and linear func-
tions show that kernelization (of our already highly non linear
features) gives a small but useful improvement in performance
— about 0.8◦per body angle, out of a total mean error
of around 7◦. The form and parameters of the kernel have
remarkably little influence. The experiments shown use a
Gaussian kernel K(x,xi) = e−β?x−xi?2with β estimated
from the scatter matrix of the training data, but other β values
within a factor of 2 from this value give very similar results.
?, are based on a kernel function
IV. POSE FROM STATIC IMAGES
We conducted experiments using a database of motion
capture data for a 54 d.o.f. body model (3 angles for each
of 18 joints, including body orientation w.r.t the camera). We
report mean (over all 54 angles) RMS absolute difference
errors between the true and estimated joint angle vectors, in
degrees:
D(x,x?) =1
m
m
?
i=1
|(xi− x?
i)mod ± 180◦|
(6)