Particle Filter with Mode Tracker(PF-MT) for Visual Tracking Across Illumination Change

Abstract

In recent work, the authors introduced a multiplicative, low dimensional model of illumination that is computed as a linear combination of a set of simple-to-compute Legendre basis functions. The basis coefficients describing illumination change, are can be combined with the "shape" vector to define a joint "shape"-illumination space for tracking. The increased dimensionality of the state vector necessitates an increase in the number of particles required to maintain tracking accuracy. In this paper, we utilize the recently proposed PF-MT algorithm to estimate the illumination vector. This is motivated by the fact that, except in case of occlusions, multimodality of the state posterior is usually due to multimodality in the "shape" vector (e.g. there may be multiple objects in the scene that roughly match the template). In other words, given the "shape" vector at time t, the posterior of the illumination (probability distribution of illumination conditioned on the "shape" and illumination at previous time) is unimodal. In addition, it is also true that this posterior is usually quite narrow since illumination changes over time are slow. The choice of the illumination model permits the illumination coefficients to be solved in closed form as a solution of a regularized least squares problem. We demonstrate the use of our method for the problem of face tracking under variable lighting conditions existing in the scene

Full text

Page 1

PARTICLE FILTER WITH MODE TRACKER(PF-MT) FOR VISUAL
TRACKING ACROSS ILLUMINATION CHANGE
Amit Kale*, Namrata Vaswani** and Christopher Jaynes*
* Ctr. for Visualization and Virtual Environments and Dept. of Computer Science
University of Kentucky, {amit,jaynes}@cs.uky.edu
**Dept. of ECE, Iowa State University, Ames, IA 50011, namrata@iastate.edu
Index Terms— Lighting,Tracking,Monte Carlo methods
ABSTRACT
In recent work, the authors introduced a multiplicative, low dimen-
sional model of illumination that is computed as a linear combina-
tion of a set of simple-to-compute Legendre basis functions. The
basis coefficients describing illumination change, are can be com-
bined with the “shape” vector to define a joint ‘shape”-illumination
space for tracking. The increased dimensionality of the state vector
necessitates an increase in the number of particles required to main-
taintracking accuracy. In this paper, we utilize the recently proposed
PF-MT algorithm to estimate the illumination vector. This is moti-
vated by the fact that, except in case of occlusions, multimodality
of the state posterior is usually due to multimodality in the “shape”
vector (e.g. there may be multiple objects in the scene that roughly
match the template). In other words, given the “shape” vector at
time t, the posterior of the illumination (probability distribution of
illumination conditioned on the “shape” and illumination at previ-
ous time) is unimodal. In addition, it is also true that this posterior is
usually quite narrow since illumination changes over time are slow.
The choice of the illumination model permits the illumination co-
efficients to be solved in closed form as a solution of a regularized
least squares problem. We demonstrate the use of our method for the
problem of face tracking under variable lighting conditions existing
in the scene.
1. INTRODUCTION
Visual tracking involves generating an inference about the motion of
an object from measured image locations in a video sequence. Un-
fortunately, this goal is confounded by sources of image appearance
change that are only partly related to the position of the object in
the scene. For example, changes in pose of the object or illumina-
tion can cause a template to change appearance over time and lead
to tracking failure.
For situations where the weak perspective assumptions hold,
shape change for rigid objects can be captured by a low dimensional
“shape” vector(here “shape” refers to location and scale change, in
general can also be affine). Tracking is the problem of causally esti-
mating a hidden state sequence corresponding to this “shape” vector,
{Xt} (that is Markovian with state transition pdf p(Xt|Xt−1), from
a sequence of observations, {Yt}, that satisfy the Hidden Markov
Model (HMM) assumption (Xt → Ytis a Markov chain for each t,
with observation likelihood denoted p(Yt|Xt)). This interpretation
This work was funded by NSF CAREER Award IIS-0092874
and by Department of Homeland Security
forms the basis of several tracking algorithms including the well-
known Condensation algorithm [6] and its variants. A similarly con-
cise model is required if we are to robustly estimate illumination
changes in a statistical tracking framework while avoiding undue in-
crease in the dimensionality of the problem. The study of appear-
ance change as a function of illumination is a widely studied area in
computer vision [2, 1]. These methods focus on accurate models
of appearance under varying illumination and their utility for object
recognition. However they typically require an explicit 3-D model
of the object. This limits their use in surveillance where a 3-D model
or a large number of images of every object to be tracked under dif-
ferent illumination conditions is unavailable [5]. Examples of such
tasksthat involvetrackingobjects through simultaneous illumination
and “shape” change are shown in Figure 2. Note that features that
are considered to be invariant to illumination could be unreliable [2]
in such situations.
In [7], the authors introduced a multiplicative, low dimensional
model of illumination that is computed as a linear combination of
a set of simple-to-compute Legendre basis functions. Such a multi-
plicative model can be interpreted as an approximation of the illumi-
nation image as discussed in Weiss [12]. The basis coefficients de-
scribing illumination change can be combined with the “shape” vec-
tor(i.e. affineor similaritygroup) todefineajoint “shape-illumination”
space for tracking. Assuming that a “shape space” of dimension
Nu = 3 corresponding to x,y translation and scale and the number
of illumination coefficients, NΛ = 7 is sufficient to capture a sig-
nificant variability from the initial template to its repositioned and
re-lit counterparts in successive frames, we need to sample a 10 di-
mensional space. It is a well known fact that as state dimension in-
creases, the effective particle size reduces and hence more particles
are needed for a certain accuracy. The question is can we do better
than brute force PF on a 10 dim space? We can utilize the fact that,
except in case of occlusions, multimodality of the state posterior is
usually due to multimodality in the “shape” vector (e.g. there may
be multiple objects in the scene that roughly match the template). Or
in other words, given the “shape” vector at time t, the posterior of the
illumination (probability distribution of illumination conditioned on
the “shape”, the image and illumination at the previous time instant
is unimodal. In addition, it is also true that this posterior is usually
quite narrow since illumination changes over time are slow.
Under these two assumptions, we can utilize the PF-MT algo-
rithm proposed in [11, 10]. The main idea is to split the entire state
vector into “effective basis” and “residual space”. We run the SIR
PF (sample from state transition pdf) [3] on the effective basis, but
approximate importance sampling from the residual posterior by its

Page 2

mode. For our problem, we run SIR PF on1“shape” and we compute
themode of the posterior of illumination conditioned on the “shape”,
previous illumination vector and the current image. The mode com-
putation turns out to be a regularized least squares problem in our
case and hence can actually be done in closed form. This idea can
also be understood as an approximation of the Rao Blackwellized
PF (RB-PF) [9], but is more general since it only requires the sub-
system to have a unimodal posterior (need not be linear Gaussian).
We would like to point out though that for the specific observation
model considered in this paper, RB-PF can also be used. However,
if the observation noise is non-Gaussian or the illumination model is
nonlinear, RB-PF will not be applicable. In order to further reduce
the number of particles required, we also effectively use the Aux
PF [8] to improve resampling efficiency.
2. STATE SPACE MODEL
2.1. Illumination model
The image template throughout the tracking sequence can be ex-
pressed as:
Tt(x,y) = Lt(x,y)R(x,y)
whereLt(x,y)denotestheilluminationimageinframetandR(x,y)
denotes a fixed reflectance image [12]. Thus if the R is known,
tracking becomes the problem of estimating the illumination image
and a “shape”-vector. Of course, R is typically unavailable and the
illumination image can only be computed modulo the illumination
contained in the image template T0,
(1)
Lt(x,y) =˜ Lt(x,y)L0(x,y)R(x,y) =˜ Lt(x,y)T0(x,y)
(2)
where L0 is the initial illumination image and˜Lt is the unknown
illumination image for frame t.
Our proposed model of appearance change [7], then, is the prod-
uct of T0 with an approximation of Ltwhich is constructed using a
linear combination of a set of NΛ Legendre basis functions defined
over the template of size,M. Let pk(x) denote the k th Legendre
basis function. Then, for NΛ = 2k + 1, Λ = [λ0,··· ,λNΛ]T, the
scaled intensity value at a pixel of the template Ttis computed as:
ˆTt(x,y) = (1
NΛ(λ0+ λ1p1(x) + ··· + λkpk(x) +
λk+1p1(y) + ··· + λNΛpk(y)) + 1)T0(x,y)
(3)
so that when Λ ≡ 0ˆTt = T0. For purposes of notation, we will
denote the effect of Λ on T0as
∆ΛT0 ≡ T0⊗ PΛ + T0
(4)
where
P =
2
6
4
1
2n+1p0
...
1
2n+1p0
···
...
···
1
2n+1pn(y1)
...
1
2n+1pn(yM)
3
7
5.
(5)
We define ⊗ as an operator that scales the rows of P with the corre-
sponding element of T0written as a vector. Given a proposal image
region G and T0, the Legendre coefficients that relight T0to resem-
ble G can be computed by solving the least squares problem:
T0⊗ PΛ = AT0Λ ≈ T0− G
(6)
where AT0? T0⊗ P.
1in case of occlusions one can also treat the mean intensity (0th order
Legendre coefficient) as part of the effective basis
2.2. System and Observation Model
The new illumination model can be combined with “shape” to define
a joint “shape-illumination” vector
Xt =
»
ut
Λt
–
(7)
where ut = [s tx ty]?corresponds to a three dimensional “shape”
space encompassing scale (s) and translations tx and ty and Λ ∈
RNΛcorresponds to coefficients of the Legendre polynomials of or-
der k.
The system dynamics is assumed to be a random walk model on
object “shape”, utand on illumination coefficients, Λti.e.
ut+1 = ut+ νut, νut∼ h(.)
Λt+1 = Λt+ νΛt, νΛt∼ N(0,Π)
(8)
(9)
where ΠNΛ×NΛis a diagonal covariance matrix (variance of indi-
vidual components of Λ) and h(.) denotes the pdf of νutwhich is
described in Section (4.1).
Let T0 denote the original template and let M denote the num-
ber of pixels in it. The observation at time t, Ytis the image at time
t. We assume the following image formation process: the image
intensities of the image region that contains the object are illumina-
tion scaled versions of the intensities of the original template, T0,
plus Gaussian noise. Also, the proposals of the image region that
contains the object are obtained by applying the dynamics of the ob-
ject’s “shape” to each point of the template. Also, the rest of the
image (which does not contain the object) is independent of the ob-
ject intensity or “shape”(and hence can be thrown away). Thus we
have the following observation model:
Yt
„
Jut+
»
X0
Y0
– «
= ∆ΛtT0+ ψt
(10)
where ψt ∼ N(0,V ) where VM×M is a diagonal covariance matrix
(variance of individual pixel noise) and J is
J =
»
X0− ¯ x01
Y0− ¯ y01
1
0
0
1
–
.
(11)
where X0 and Y0 denote the x and y coordinates of each point on
the template and ¯ x0and ¯ y0 denote the corresponding means. 1 and
0 denote a vector of ones and zeros of size M respectively. J can
be easily modified for the affine case as described in [6]. For brevity
we will denote the image region in Ytindicated by a vector u as:
Gu
t= Yt
„
Ju +
»
X0
Y0
–
.
«
(12)
Thus the observation likelihood can be written as:
p(Yt|Xt) = p(Yt|ut,Λt) = exp[−||Gut
t − ∆ΛtT0||2
v
]
(13)
where (V )i,i = v.
3. PARTICLE FILTER WITH MODE TRACKER(PF-MT)
ALGORITHM
A naive approach would be to simply apply the SIR PF [4] to the
system (8),(9) and observation model (10). In (9) we use a k = 3 or-
der Legendre basis and hence NΛ = 2k+1 = 7. Thus the total state

Page 3

Algorithm 1 PF-MT. Going from πN
PN
1. Auxiliary Resampling: Compute gi
Xi
(wi
2. Importance Sample (IS) on effective basis: ∀i, sample νut∼
h(u) and compute ui
3. Mode Tracking (MT) in residual space:
using (15) and set Λi
t
4. Weighting: Compute wi
t−1 to πN
t(Xt)=
i=1w(i)
n δ(Xt− Xi
t), Xi
t= [ui,Λi]
tusing (17) and resample
t−1according to it. Reset the weights of the resampled to
t−1)newdefined in (18).
t= ui
t−1+ νut
∀i, compute mi
t
t= mi
tusing (16).
space dimension is10. It is a well known fact that the number of par-
ticles required for a certain accuracy increases with state dimension
[4], making the PF very expensive to run. But notice that condi-
tioned on ut, the posterior of Λt is unimodal. Also, we observed in
expts that covariance of change of Λt was small. Thus we can use
the recently proposed PF-MT idea [10] for this problem. The main
idea is to importance sample (IS) “shape”, ui
sition model (8), but replace IS by posterior Mode Tracking (MT)
for illumination, Λt, i.e. we compute the mode (denote it as mi
p(Λt|ui
pute mi
by MT isa valid approximation when the the covariance is small[10]
which is true in our case.
Now, it is easy to see that
tfrom its state tran-
t) of
t,Λi
tand use a Gaussian about mi
t−1,Yt) and set Λi
t= mi
t. In exact PF, one would com-
tas the IS density. Replacing IS
p(Λt|ui
t,Λi
t−1,Yt) ∝ p(Yt|ui
t,Λt)p(Λt|Λi
t−1)
(14)
where the first term is defined in (13) and the second term is given
by (10). Thus mi
of (14) and this turns out to be a nice regularized least squares prob-
lem (regularization term is the weighted distance from Λt−1) with a
closed form solution given by
tcan be computed as the minimizer of the −log[.]
mi
t= Λi
t−1+ (Π−1+ AT
T0V−1AT0)−1AT
T0V−1(Gui
t
t − ∆ΛtT0) (15)
Note all the multipliers can be pre-computed, making this a very
fast computation. With the above importance sampling strategy, the
weighting term will be [10]
wi
t∝ wi
t−1p(Yt|ui
t,Λi
t)p(Λi
t|Λi
t−1), Λi
t= mi
t
(16)
Usingthismethod greatlyreduces theweight variance, thus reducing
the number of particles required for a certain accuracy (or improving
tracking accuracy when number of particles available is small). We
have shown the comparison of PF-MT with other existing methods
- SIR PF (called FULLPF), Auxiliary PF(called FULLPFWAP) and
PF without tracking illumination (called NOILLUM) in Figure 2.
To improve resampling efficiency, we used the look-ahead re-
sampling idea of Auxiliary PF [8]. This performs resampling of the
past particles when the current observation, Yt comes in, and uses
the likelihood of Xi
according to
t−1generating Yt to resample, i.e. it resamples
gi
t= wi
t−1p(Yt|Xi
t−1)
(17)
After resampling, the weights of the resampled particles are set to
(wi
t−1)new=wi
t−1
Ngi
t
=p(Yt|Xi
t−1)
N
(18)
0 10203040 50
0
20
40
60
80
100
120
Frames
Location Tracking Error


PFMT
NOILLUM
FULLPFWAP
FULLPF
010 20304050
0
20
40
60
80
100
120
Frames
Location Tracking Error


PFMT
NOILLUM
FULLPFWAP
FULLPF
Fig. 1. Comparison of errors from ground truth of location us-
ing different particle filters using 300 particles (top) and 100 par-
ticles(bottom).
The complete algorithm is summarized in Algorithm 1.
Note, for the particular form of the observation model that we
use (additive Gaussian noise), one can also use Rao-Blackwellized
PF [9]. But if the observation noise were non-Gaussian (e.g. in
order to model occlusions or outliers), RB-PF cannot be used. Also,
in case of occlusions, one may want to use the mean intensity (λ0)
also as part of the effective basis (importance sample for it) while
mode tracking for the rest.
4. MODEL PARAMETER ESTIMATION AND RESULTS
4.1. Learning the Model Parameters
We need to learn the noise models for ”shape”(h(.) ), of illumi-
nation (Π) and the observation noise covariance (V ). We assume
that we have a static camera acquiring images and that the illumina-
tion conditions, although variable within the scene, do not change
significantly over time. Ground truth video sequences consisting
of a starting template T0 and its location and shape in subsequent
frames, Gt,t = 1,··· ,Nf are used to compute state-vectors Xt =
[ut Λt]T,t = 1,··· ,Nfusing (12) and (6)for this motion with the
corresponding approximations {ˆ Gt = ∆ΛtT0,t = 1,··· ,Nf}
We consider the “shape” difference vectors dut = ut − ut−1
for t = 1,··· ,Nf. Assuming that the individual components of
dutare independent, we build a “shape” sampling distribution h(u)
as follows. Given dut,t = 1...N the dynamic model was esti-
mated for the shape vector. The horizontal displacement dtx and
scale change ds were modeled as Gaussian random vectors whose
parameters were computed using standard MLE techniques. In or-
der to take into account the nature of human gait the vertical dis-
placement dty was modeled as a mixture of two Gaussians whose
parameters were estimated using EM. The “shape” sampling distri-

Page 4

bution is given by
h(u) = [N(µs,σ2
s) N(µtx,σ2
tx)
2
X
i=1
αiN(µi,σi)]
A third order Legendre polynomial (NΛ = 2 ∗ 3 + 1 = 7) was
used to represent the illumination effects. Given dΛt = Λt− Λt−1
for t = 1,··· ,Nf we estimate Π as
Π =
1
Nf− 1
Nf
X
t=2
(Λt− Λt−1)(Λt− Λt−1)T
(19)
The per-pixel observation noise V is estimated by averaging the
SSD between the corresponding pixels ofˆGtand Gtas
V =
1
Nf
Nf
X
t=1
(ˆGt− Gt) ⊗ (ˆGt− Gt)]
(20)
4.2. Results
Our test dataset contained several different subjects moving through
challenging illumination conditions in an indoor environment(see
Figure 2)including overhead, side-lit and partially shaded regions
as they approach a surveillance camera. Ground truth was gener-
ated from one sequence. Figure 2 shows the face tracking results
using the PF-MT algorithm using 100 particles. These sequences
are typical for this setup and only three frames are shown in the in-
terest of space. The box corresponds to the MMSE estimate of the
state vector u computed as ˜ u =
location error from the ground truth for different particle filters. The
same dynamic model was used for all the PFs. Full PF represents
the case where instead of importance sampling ut alone Λt is im-
portance sampled from N(Λi
represents FULLPF with SIR replaced by the Auxiliary PF. NOIL-
LUM represents the case where no illumination model is used while
PF-MT represents the the case using our algorithm. As can be seen,
the estimated “shape” using PF-MT has much lower error than all
the other algorithms. Also it remains in track even with just 100
particles (bottom row of Figure 1).
1
N
PN
i=1ui. Figure 1 shows the
t−1,Π) and SIR is used. FullPFWAP
5. CONCLUSIONS
In this paper we studied the problem of visual tracking as an in-
ference problem in a joint “shape-illumination” space introduced in
[7]. We used the PF-MT idea to exploit the fact that, except in case
of occlusions, multimodality of the state posterior is usually due to
multimodality in the “motion” vector and that given the “motion”
vector at time t, the posterior of the illumination (probability distri-
bution of illumination conditioned on the “motion”, the image and
illumination at time t − 1) is unimodal. We demonstrated the use
of our method for tracking faces under variable lighting conditions
existing in the scene without requiring an increase in the number of
particles despite the higher dimensionality of the state vector.
6. REFERENCES
[1] R. Basri and D. Jacobs. Lambertian reflectance and linear sub-
spaces. IEEE Trans. PAMI, 25(2):218–233, 2003.
[2] P. Belhumeur and D.J.Kriegman. What is the set of images
of an object under all possible illumination conditions. IJCV,
28(3):1–16, 1998.
Fig. 2. Face tracking using PF-MT as the individuals walk through
different lighting conditions. The white box shows the image region
corresponding to MMSE estimate of the “shape vector”.
[3] A. Doucet, N. deFreitas, and N. Gordon, editors. Sequential
Monte Carlo Methods in Practice. Springer, 2001.
[4] N. Gordon, D. Salmond, and A. Smith.
proach to nonlinear/nongaussian bayesian state estimation.
IEE Proceedings-F (Radar and Signal Processing), pages
140(2):107–113, 1993.
[5] G. Hager and P. Belhumeur. Efficient region tracking with
parametric models of geometry and illumination. IEEE Trans.
PAMI, 20(10):1025–1039, 1998.
[6] M. Isard and A. Blake. Condensation - conditional density
propogation for visual tracking. IJCV, 21(1):695–709, 1998.
[7] A. Kale and C. Jaynes. A joint illumination and shape model
for visual tracking. Proceedings of IEEE CVPR, pages 602–
609, 2006.
[8] M. K. Pittand N. Shephard. Filteringvia simulation: Auxiliary
particle filters. Journal of the American Statistical Association,
94(446), 1999.
[9] T. Schn, F. Gustafsson, and P. Nordlund. Marginalized particle
filtersfor nonlinear state-spacemodels. IEEETrans.Sig. Proc.,
2005.
[10] N. Vaswani, A. Yezzi, Y. Rathi, and A. Tannenbaum. Parti-
cle filters for infinite (or large) dimensional state spaces. In
IEEE Intl. Conf. on Acoustics, Speech and Signal Processing
(ICASSP), April 2006.
[11] N. Vaswani, A. Yezzi, Y. Rathi, and A. Tannenbaum. Time-
varying finite dimensional basis for tracking contour deforma-
tions. In IEEE Conf. Decision and Control (CDC), 2006.
[12] Y. Weiss. Deriving intrinsic images from image sequences.
Proc of ICCV, 2001.
Novel ap-