Generalized ELL for detecting and tracking through illumination model changes

Abstract

In previous work, we developed the Illum-PF-MT, which is the PF-MT idea applied to the problem of tracking temporally and spatially varying illumination change. In many practical problems, the rate at which illumination changes varies over time. For e.g. when a car transitions from shadow to sunlight or vice-versa the rate of illumination change is much higher than when it is in shadow or in sunlight. One way to model illumination change in such problems is using a Gaussian random walk model with two values of the change covariance - a large covariance when a "transition" is detected and a much smaller one when "no transition" is detected. But to use such a model, one needs to first detect the transition. The transition is a natural one and so it happens gradually (unlike a sudden manual dimming of the light in the room) and thus existing change detection statistics which are designed only for sudden changes are unable to detect the transition. In this paper, we propose to use the recently proposed generalized ELL (gELL) idea which uses the tracked part of the change to detect it and hence detects such partially trackable changes very quickly. Since gELL detects much before loss of track occurs, one is able to transition to the "transition" model and back without ever losing track. Also, for the first time, we demonstrate the use of gELL in combination with the PF-MT algorithm which is more stable to model change than the original PF.

Full text

Page 1

GENERALIZED ELL FOR DETECTING AND TRACKING THROUGH ILLUMINATION
MODEL CHANGES
Amit Kale∗and Namrata Vaswani∗∗
∗Siemens Corporate Technology India, Bangalore , kale.amit@siemens.com
∗∗Dept. of ECE, Iowa State University, Ames, IA 50011, namrata@iastate.edu
ABSTRACT
In previous work, we developed the Illum-PF-MT, which is the PF-
MT idea applied to the problem of tracking temporally and spatially
varying illumination change. In many practical problems, the rate
at which illumination changes varies over time. For e.g. when a
car transitions from shadow to sunlight or vice-versa the rate of il-
lumination change is much higher than when it is in shadow or in
sunlight. One way to model illumination change in such problems is
using a Gaussian random walk model with two values of the change
covariance - a large covariance when a “transition” is detected and
a much smaller one when “no transition” is detected. But to use
such a model, one needs to first detect the transition. The transition
isa natural one and so ithappens gradually (unlike asudden manual
dimming of the light in the room) and thus existing change detection
statistics which are designed only for sudden changes are unable to
detect the transition. In this paper, we propose to use the recently
proposed generalized ELL (gELL) idea which uses the tracked part
of the change to detect it and hence detects such partially trackable
changes very quickly. Since gELL detects much before loss of track
occurs, one is able to transition to the “transition” model and back
without ever losing track. Also, for the first time, we demonstrate
the use of gELL in combination with the PF-MT algorithm which is
more stable to model change than the original PF.
1. INTRODUCTION
Tracking illumination changes of moving objects is a challenging
problem. In absense of illumination changes, motion of a rigid ob-
jectmoving infront of acamera canbetracked using a3dimensional
vector consisting of x-y translation and uniform scale or more gen-
erally using a 6 dimensional affine model as in Condensation [1]. If
illumination changes over time, but is constant in space, then one
extra dimension gets added. But if different regions of an object ex-
perience different lighting conditions (e.g. a face with light falling at
different angles on different parts of the face, usually happens when
light source is near the object, invalidating assumptions about point
light sources at infinity), the maximum dimension of illumination
change is equal to the number of image pixels. Of course, the vari-
abilityisnever that large, and as been demonstrated inprevious work
[2], usually a 3 to 7 dimensional basis suffices for modeling illumi-
nation, but even that will increase the total state space dimension to
somewhere between 9-16. It is well known that as state space di-
mension increases, number of particles required to track using a PF
increases [3]. This makes PF impractical for dimensions larger than
7 or 8. But, as shown in [4], the conditional posterior of illumination
This research was partially supported by funds from NSF under grant
ECCS-0725849
change (conditioned on motion and previous state) is usually uni-
modal and narrow so that the conditional posterior of illumination
can be replaced by a Dirac delta function at its posterior mode, with
little error. Furthermore, this mode computation is very efficient,
since it turns out to be the solution of a regularized least squares
problem. This one step, reduces the importance sampling dimen-
sion to 3 instead of 10, drastically reducing the number of particles
required. The idea, called Illum PF-MT, was demonstrated in our
recent paper [4].Now, in certain problems, the rate at which illu-
mination changes varies over time. For e.g. when a car transitions
fromshadow to sunlight or vice-versa therateof illumination change
is much higher than when it is in shadow (see the first row of Fig.
1). One good way to model illumination change in such problems is
using a Gaussian random walk model with two values of the change
covariance - a large covariance (or in effect a weak prior) when a
“transition” is detected and a much smaller covariance (learnt from
training data) when “no transition” is detected. Note that, since the
transition itself from small to large covariance happens gradually
(since it is a natural one), even though to keep our modeling simple,
we use a single change point to model it. Since the transition is not
a sudden one (e.g. as would happen if the light in a room was man-
ually suddenly dimmed to a third of its original value), even without
any correction step, the PF-MT algorithm is able to partially track
it. Such changes which get partially tracked (are not sudden enough)
are usually missed by loss-of-track based statistics such as tracking
error [5] or averaged likelihood [6] or score function (see [7] for
a survey of sudden change detection methods using particle filters).
Note that the tracking error plots in the last columns of Figure 1 miss
the change.
But, the ELL statistic [8, 9] was designed for detecting exactly
such gradual changes. It uses the partially tracked part of the change
to detect it and hence isable to detect gradual transitions much better
than existing statistics [5, 6, 7]. In fact, it detects much before loss of
track. Now, in problems, such as ours, where the nominal model is
nonstationary and has continuously increasing prior state variance,
the sensitivity of ELL reduces with time. In this paper, we demon-
strate the use of a recently proposed generalization of ELL, called
gELL [9] (which was developed for detecting changes in nonstation-
ary nominal models), to detect the changes in the rate of illumination
change. Also, unlike ELL, the gELL is able to detect a sequence of
changes, for e.g., in our case, the increase the decrease of the change
covariance as shown in Fig. 1, last column.
Note that this is the first application where gELL (and not ELL)
is used for change detection (so far only one proof-of-concept sim-
ulation was shown in [9]). In addition, we successfully demonstrate
the use of gELL not only to detect illumination model change, but
alsotoincrease illuminationchange covariance toa largevaluewhen
thetransitionisdetected, and then reduce itto itsoriginal value when
gELL again goes below a threshold. Since gELL detects much be-

Page 2

fore loss of track occurs, one is able to transition to the next model
and back without ever losing track (see Figure 1). Also, for the first
time, we demonstrate the use of gELL in combination with the PF-
MT algorithm - in past work [9, 10], ELL was used in combination
with only the original PF [11]. This is important because PF-MT
(and also some other PFs such as [12]) importance sample using a
density that depends on the current observation. For this reason, PF-
MT is much more stable to model changes than original PF, i.e. is
able to partially track them better than original PF. This fact reduces
thedelay in the detection using gELL compared tousing it withorig-
inal PF.
2. STATE SPACE MODEL AND THE PROBLEM
We briefly describe below the state space model for illumination and
motion change over time. This is taken from our previous work [4]
where we introduced the PF-MT algorithm for illumination and mo-
tion tracking.
System Model: The state, Xt, consists of a 3-dimensional mo-
tion vector ut which contains x-y translation and scale, and a 7 di-
mensional illumination coefficients vector (illumination is parame-
terized using a Legendre basis) as in [2], i.e. Xt = [u′
system model is a random walk model on object motion, ut and on
illumination coefficients, Λti.e.
t Λ′
t]′The
ut+1
Λt+1
=
=
ut+ νut, νut∼ h(.)
Λt+ νΛt, νΛt∼ N(0,Π)
(1)
(2)
where ΠNΛ×NΛis a diagonal covariance matrix (variance of indi-
vidual components of Λ) and h(.) denotes the pdf of νutdescribed
in [4].
Observation Model: Let T0denote the original template and let
M denote the number of pixels in it. The observation at time t, Yt,
is the image at t. It assumes the following image formation process:
the image intensities of the region that contains the object, are illu-
mination scaled versions of the intensities of the original template,
T0, plus Gaussian noise. The region containing the object isthe orig-
inal template region scaled and translated using the current elements
of ut. The rest of the image (which does not contain the object) is
independent of the object intensity or “shape”, and hence is not used
in defining the observation likelihood. Thus we have the following
observation model:
Yt
?
Jut+
?X0
Y0
??
= [fT0(PΛt)]vec+ ψt, ψt ∼ N(0,V ) (3)
where the notation [.]vec denotes arranging a two dimensional ma-
trix as a column vector; (V )M×M is a diagonal covariance matrix
(variance of individual pixel noise); P contains the Legendre basis
directions as its column vectors (defined in (5) of [4]) and
fT0(PΛt) ? T0+ T0. ∗ PΛt,
J ?
?X0− ¯ x0 1 0
Y0− ¯ y0 0 1
?
(4)
where .∗ is the MATLAB notation, X0 and Y0 denote the x and y
coordinates of each point on the template and ¯ x0 and ¯ y0 denote the
corresponding means. 1 and 0 denote a vector of ones and zeros of
size M respectively. Thus the observation likelihood (OL) is:
p(Yt|Xt) = p(Yt|ut,Λt) = e−||[Gut
t
−fT0(PΛt)]vec||2
V
(5)
where ||a||V ? aTV−1a for a vector a and
Gut
t
? Yt
?
Jut+
?X0
Y0
??
(6)
The PF-MT algorithm for tracking using the above state space
model, when the illumination change covariance, Π, is a constant,
was proposed in [4]. It importance sampled on motion (since it had a
large variance and multimodal state transition prior and since it often
resulting in a multimodal observation likelihood), while mode track-
ing on illumination (whose change covariance was much smaller and
the observation likelihood was mostly unimodal conditioned on mo-
tion). It is summarized in the first few steps of Algorithm 1.
In the current work, we consider the problem where the model
of (2) can change with time. In particular, Π can take two possible
values (small and large) and the time when the transition between
them happens is unknown. The goal is to detect when to change Π
fromsmalltolarge(fortheshadow-light transitionframes) andwhen
to change it back. The value Πsmallis known (learnt from training
data), but the value Πlargefor the transition frames is not known.
3. DETECTING AND CHANGING THE SYSTEM MODEL
In many tracking applications, the system model parameters are not
time-invariant. For our problem, consider the random walkmodel on
illumination coefficients given in (2). As explained in the introduc-
tion, the rate of change of illumination over time (quantified by the
illumination change covariance) is much larger when the car transi-
tions from a shadowy region to a bright/sunlit region or vice versa
than when it is in a shadowy or bright region. One good model for
this situation is (2) with a small noise covariance value when the car
is in the shadowy or the sunlit region, but a large noise covariance
value when it transitions from shadow to sunlight or vice versa.
3.1. Computing Generalized-ELL (gELL) and gELL-max
To use the above model for tracking, one first needs to be able to
detect the change time (the time when the illumination change co-
varianceneeds tobeincreased orreduced), asquickly aspossible. To
avoid having to re-initialize the tracker, one would like to detect this
change before significant loss-of-track occurs. In [9], the Expected
(negative) Log-Likelihood of state (ELL) statistic was introduced to
detect changes before they resulted in significant loss-of-track. The
key idea was to use the “tracked part of the change” to detect it.
Thus ELL requires the change to be partially tracked in order to de-
tect it and it often does not detect very sudden changes that result in
immediate loss of track. Such sudden statistics do not occur in our
problem, but if they do, tracking error [5] or averaged likelihood [6]
can be used to also detect them in combination with ELL.
ELL can be interpreted as the Kerridge Inaccuracy (proportional
to Kullback-Leibler divergence) between the posterior at the current
time, πt|t(Xt) = p(Xt|Y1:t), and the prior state distribution at t,
whichisequal tothetstepahead prediction distribution, πt|0(Xt) =
p(Xt). As explained in [9], ELL cannot detect multiple changes in
a sequence and its sensitivity reduces with time in many problems
such as ours where the nominal model is nonstationary (because the
variance of p(Xt) increases with t). To handle this, a generalization
of ELL was defined in [9]. Generalized ELL (gELL) is the Kerridge
Inaccuracy between πt|tand the ∆ < t step ahead prediction distri-
bution, πt|t−∆(Xt) = p(Xt|Y1:t−∆), i.e.
gELL(t,∆) ? −Eπt|t[−logπt|t−∆(Xt)]
(7)

Page 3

Note that gELL and ELL may be computed for the entire state
Xt or for a part of it. In our problem, we need to detect changes
in illumination and hence we compute gELL, defined in (7), only
for the posterior of Λt. Note, the same idea can also be used to de-
tect changes in x or y direction velocity of the object (in that case
we would define gELL for only for posterior of ut). To compute
the gELL, we need a closed form expression for πt|t−∆. To get
that, we propose to approximate the PF estimate of the posterior at
t−∆, πt−∆|t−∆(Xt), by a Gaussian density, i.e. πN
N(µN
the empirical mean and covariance of the weighted particle set com-
prising of πN
distribution, πt|t−∆(Xt), which is obtained by applying the system
model of Λt given in (2) ∆ times to πt−∆|t−∆(Xt), is also Gaus-
sian, i.e. πt|t−∆(Xt) ≈ N(µN
fined below. Thus, in summary, gELL is computed as:
t−∆|t−∆(Xt) ≈
t−∆|t−∆,ΣN
t−∆|t−∆) where the parameters are estimated as
t−∆|t−∆(Xt). With this approximation, the prediction
t|t−∆,ΣN
t|t−∆) with parameters de-
gELL(t,∆)?
N
?
i=1
wi
t(Λi
t− µN
t|t−∆)TΣN
t|t−∆
−1(Λi
t− µN
t|t−∆),
µN
t|t−∆=µN
t−∆|t−∆?
N
?
i=1
wi
t−∆Λi
t−∆
ΣN
t|t−∆?ΣN
t−∆|t−∆+ ∆Π,
ΣN
t−∆|t−∆?
N
?
i=1
wi
t−∆(Λi
t−∆− µN
t|t−∆)(Λi
t−∆− µN
t|t−∆)T
(8)
The choice of ∆ in the above expression is not clear. If it is too
small, the change between timet and t−∆ may not be largeenough,
i.e. the numerator, (Λi
large, the prediction covariance, ΣN
larger than needed, thus reducing its sensitivity to smaller changes (a
problem similar to that of ELL which uses ∆ = t). Thus, a statistic
that is always more sensitive than gELL(t,∆) (i.e. its detection
delay is smaller than or equal to that of gELL(t,∆)) is
t− µN
t|t−∆), may be too small. If it is too
t|t−∆= ΣN
t−∆|t−∆+∆Πmay be
gELL-max(t) ?
max
∆=1,2,...tgELL(t,∆)
(9)
Of course it may also generate a few extra false alarms. In Sec. 4,
we show experiments with both gELL(t,∆), for the car sequence
which is faster moving and thus has faster rate of change of illumi-
nation covariance, and with the more sensitive, gELL-max(t), for
the face sequence in which the changes are slower.
3.2. Using gELL or gELL-max to Change System Model
We begin by tracking using PF-MT that uses (2) with a small co-
variance Π = Πsmall(learnt only from the shadow sequence) and
we keep computing the ∆ = 5 step ahead gELL (or gELL-max)
at every t. gELL (or gELL-max) exceeding its detection threshold
is used as a cue to increase the value of Π to Πlarge. Πlarge can
heuristically set to a large value (or even to ∞ to allow PF-MT to
only use the observations) or if enough training data for the transi-
tion frames is available, it can be learnt from it. A large value of
Π models a weak prior, i.e. the tracker mostly follows the changing
observations. It uses these observations to latch on to the sunlight
illumination. When the car has fully transitioned to the sunlight re-
gion, the value of gELL falls below its threshold, and this is used
as a cue to again reduce Π to Πsmall. The complete algorithm in
summarized in Algorithm 1.
Algorithm 1 Change Compensated Aux PF-MT.
At each t, do
1. Auxiliary Resampling:
wi
Reset the weights of the resampled particle to (wi
wi
t−1
Ngi
t
N
2. Importance Sample (IS) on effective basis: ∀i, sample νui
h(u) and compute ui
3. Mode Tracking (MT) in residual space:
using mi
D ? [Gui
∀i, compute gi
t using gi
t−1according to it.
t−1)new=
t
=
t−1p(Yt|Xt = Xi
t−1) and resample Xi
=
p(Yt|Xt=Xi
t−1)
.
t∼
t= ui
t−1+ νui
t.
∀i, compute mi
T0V−1D
t
t= Λi
t−1+ (Π−1+ AT
T0V−1AT0)−1AT
t
t]vec− fT0(PΛi
t−1) and set Λi
t= mi
t.
4. Weighting:
Compute wi
tusing wi
t =
˜ wi
j=1˜ wj
t
?N
t, ˜ wi
t =
p(Yt|ui
5. gELL Computation:
gELL-max(t) using (9).
6. Change Π:
when it goes below threshold set Π = Πlearnt.
t,Λi
t)p(Λi
t|Λi
t−1) .
Compute gELL using (8) or
If gELL exceeds threshold, set Π = Πlarge,
4. EXPERIMENTAL RESULTS
We now demonstrate the utility of the proposed approach for two
different datasets. The car dataset was generated from a camera ob-
serving a road from above as cars approach an intersection and move
in and out of shadow. The second dataset contained several subjects
moving through different illuminations in an outdoor environment.
In Figure 1 we show the results of using gELL and PFMT using
100 particles. Around frame 40, when the car starts to move from
shadow tosunlight (asindicatedby thedouble arrow inFigure??(d))
the gELL value starts to increase from its shadow value. When it
does we set Π = Πlarge in Algorithm1. We use Πlarge = ∞.
When gELL decreases again, we reset Π to Πlearnt. The tracking
is shown in first row of Figure 1. If we do not use gELL to detect
the transitions and increase Π, the tracker fails( Figure 1e-h). We
also show the use of normalized tracking error (normalized by ra-
tio of peaks of actual tracking error to gELL) for change detection
in Figure 1(d). As can be seen, tracking error does not show any
sharp change around frame 40 unlike gELL whose change is clearly
detectable. This makes the task of switching Π to Πlarge difficult
leading to loss of track. For the face tracking case Figure 1 i-p,
we tried the use of both gELL( 8) and gELL-max (9). gELL max
works better for this case since the rate of change of illumination
are slower than in the face case (See Section 3.1). The changeover
from changeover from sunlight to shadow (arrows indicate frame of
changeover) are again detected accurately. However in this case us-
ing PF-MT succeeds even across illumination changes.
5. CONCLUSION
In this paper, we proposed to use the recently proposed generalized
ELL (gELL) idea which uses the tracked part of the change to detect
it and hence detects such partially trackable changes very quickly.
Since gELL detects the illumination change before loss of track oc-
curs, one isabletotransitiontothe“transition”model andback with-
out ever losing track. Furthermore we demonstrate the use of gELL
in combination with the Illumination PF-MT algorithm [4] which is
more stable to model change than the original PF. We demonstrated

Page 4

020406080100
0
1
2
3
4
5
6
7
Frames
Tracking Error/gELL


Tracking Error
(normalized)
gELL
(a)t = 6
(b)t = 44
(c)t = 80
(d) using gELL from equation(8)
(e)t = 6
(f)t = 44
(g)t = 80
(h) t = 95
010203040
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
Frames
Tracking error/gELL


Tracking Error (normalized)
gELL
12
(i)t = 2
(j) t = 12
(k)t = 35
(l) Using gELL -max from equation(9)
010 2030 40
0
1
2
3
4
5
6
7
8
9x 10
−3
Frames
gELL
14
Transition from sunlight to
shadow occurs at Frame 14
(m)t = 2
(n)t = 14
(o)t = 40
(p) Using gELL -max from equation(9)
Fig. 1. Tracking using PF-MT and gELL as objects move through different lighting conditions. The white box corresponds to the MAP
estimate of the “shape vector”. (a) (b) (c) show tracking of a car (d) shows the comparison of the tracking error with gELL. The second
row shows the tracking of the car as it moves from shadow to sunlight when gELL is not used to detect change and PF-MT is left unaltered.
Bottom two rows show the face tracking cases where gELL max detects illum changes correctly
the algorithm for tracking faces and cars across drastic illumination
changes.
6. REFERENCES
[1] M. Isard and A. Blake, “Condensation: Conditional Density
Propagation for Visual Tracking,” Intl. Journal of Comp. Vision,
pp. 5–28, 1998.
[2] A. Kale and C. Jaynes, “A joint illumination and shape model
for visual tracking,” Proceedings of IEEE CVPR, pp. 602–609,
2006.
[3] A. Doucet, N. deFreitas, and N. Gordon, Eds., Sequential Monte
Carlo Methods in Practice, Springer, 2001.
[4] A. Kale, N. Vaswani, and C. Jaynes, “Particle filter with mode
tracker (pf-mt) for visual tracking across illumination change,”
in ICASSP, 2007.
[5] Y. Bar-Shalom and T. E. Fortmann, Tracking and Data Associ-
ation, Academic Press, 1988.
[6] Q. Zhang, F. Campillo, F. Cerou, and F. LeGland, “Nonlinear
system fault detection and isolation based on bootstrap particle
filters,” December.
[7] C. Andrieu, A. Doucet, S.S. Singh, and V.B. Tadic, “Particle
methods for change detection, system identification, and con-
trol,” Proceedings of the IEEE, vol. 93, pp. 423– 438, March
2004.
[8] N. Vaswani, “Change detection in partially observed nonlinear
dynamic systems with unknown change parameters,” in Ameri-
can Control Conference (ACC), 2004.
[9] N. Vaswani, “Additive change detection in nonlinear systems
with unknown change parameters,” pp. 859–872, March 2007.
[10] N. Vaswani, A. RoyChowdhury, and R. Chellappa, ““Shape
Activity”: A Continuous State HMM for Moving/Deforming
Shapes with Application to Abnormal Activity Detection,”
IEEE Trans. Image Proc., pp. 1603–1616, October 2005.
[11] N.J. Gordon, D.J. Salmond, and A.F.M. Smith, “Novel ap-
proach tononlinear/nongaussian bayesian stateestimation,” IEE
Proceedings-F (Radar and Signal Processing), pp. 140(2):107–
113, 1993.
[12] A. Doucet, “On sequential monte carlo sampling methods for
bayesian filtering,” in Technical Report CUED/F-INFENG/TR.
310, Cambridge University Department of Engineering, 1998.