Multiple pedestrian detection and tracking based on weighted temporal texture features

Abstract

This work presents a novel method for detecting and tracking pedestrians from video images taken by a fixed camera. A pedestrian may be totally or partially occluded in a scene for some period of time. The proposed approach uses the appearance model for the identification of pedestrians and the weighted temporal texture features. We compared the proposed method with other related methods using color and shape features, and analyzed the features' stability. Experimental results with various real video data revealed that real time pedestrian detection and tracking is possible with increased stability over 5-15% even under occasional occlusions in video surveillance applications.

Full text

Page 1

Multiple Pedestrian Detection and Tracking
based on Weighted Temporal Texture Features
Hee-Deok Yang, Seong-Whan Lee
Department of Computer Science and Engineering, Korea University
Anam-dong, Seongbuk-ku, Seoul 136-701, Korea
{hdyang, swlee}@image.korea.ac.kr
Abstract
This paper presents a novel method for detecting and
tracking pedestrians from video images taken by a fixed
camera. A pedestrian may be totally or partially occluded
in a scene for some period of time. The proposed approach
uses the appearance model for the identification of pedes-
trians and the weighted temporal texture features. We com-
pared the proposed method with other related methods us-
ing color and shape features, and analyzed the features’
stability. Experimental results with various real video data
revealed that real time pedestrian detection and tracking
is possible with increased stability over 5–15% even under
occasional occlusions in video surveillance applications.
1. Introduction
Tracking humans in video is an important task in many
applications such as video surveillance and virtual reality
interface as they are the primary actors in those task do-
mains. Its goal is to detect and track people, and monitor
their activities in a given environment. The required capa-
bilities include detecting moving people, segmenting them
into individual persons, and understanding their behavior.
The success of such a surveillance system owes largely to
the tracking method which performance is independent of
the number of people in sight, the variation of illumination,
the distance from the camera and the camera setup [6].
In the past few years, a number of real time systems have
been developed to detect and track people. Haritaoglu et
al. [3] introduced the system W4 for detecting and track-
ing multiple people or part of their bodies. VSAM [1] is
a CMU development system for tracking people and mov-
ing objects. The system’s infrastructure involves fourteen
cameras. Kang and Lee [2] introduced a system for detect-
ing and tracking multiple people who are totally or partially
occluded occasionally.
More recently there has been an increasing interest in in-
tegrating the temporal information in tracking system [4].
Among the systems listed above, Roh and Lee’s [2] and W4
[3] system already employed temporal information to im-
prove the performance of tracking and identification. With
temporal information, they could narrow down the search
range significantly. Pedestrian tracking generally consists
of two subtasks of target detection and verification [4].
In this paper, we propose a pedestrian tracking method
based on the appearance model using the weighted temporal
texture features of the image sequence. Weighted temporal
texture is a set of pairs of a texture value and an associated
weight. The weight summarizes the size, position, velocity,
duration, and frequency of appearance of the texture region,
as well as the number of the pedestrians adjacent to the tar-
get pedestrian. Finally, we employed a simple method of
recognition for the verification task. The system consists
of three parts: pedestrian detection, pedestrian tracking and
face recognition.
2. Pedestrian Detection
The goal of pedestrian detection is to locate all and only
thepedestriansinascene. Thetaskisdividedintotwoparts:
detection of candidate regions and detection of individual
pedestrians.
2.1Candidate region detection
Candidate region detection is a simple process of locat-
ing any moving objects in a scene. Here we took the meth-
ods of adaptive background subtraction and frame differ-
encing.
Let Bn(x) denote the current background intensity value
at pixel x at time n learned by observation over time, and
Tn(x) the difference threshold. B0(x) is initially set to the
value of the first frame, i.e., B0(x) = I0(x), and T0(x) is
suppliedexternally. Bn(x)andTn(x)areupdatedovertime
as follows [1]:
0-7695-2128-2/04 $20.00 (C) 2004 IEEE

Page 2

Bn+1(x) =
?
αBn(x) + (1 − α)In(x),
Bn(x),
x is non-moving
x is moving
(1)
Tn+1(x) =


where α is a time constant denoting the relative magnitude
of the influence from the new input.

αTn(x) + (1 − α)
· (5 × |In(x) − Bn(x)|),
Tn(x),
x is non-moving
x is moving
(2)
2.2Pedestrian detection
Usually a foreground region contains more than one
pedestrian. In this case we need to partition such a region
into individual pedestrians. In the first step of pedestrian
detection we apply morphological operators to eliminate
noise. The resulting image is referred to as a silhouette im-
age.
The following segmentation step mainly concerns locat-
ing heads in the silhouette image, by shape analysis and ver-
tical projection. By observing the curvature of the boundary
of a silhouette, the part of a pedestrian that looks like a head
may be located. The point of maximal convex value is a
good indicator of head, while the point of minimal concave
value is a good indicator of separation between two pedes-
trians. In this step we consider five types of concave points:
• case 1 : space between pedestrians
• case 2 : space between shoulder and background
• case 3 : space between pedestrian and background
• case 4 : space between pedestrian and obstacle
• case 5 : space between background and obstacle
Next, the system divides the region into sub-regions,
each corresponding to an individual pedestrian with a head
detected inthepreceding stage. Finallythetorsoaxisisesti-
mated from the head center down, and each pedestrian seg-
ment is estimated according to the distance from the torso
axis. The distance designates, for each pixel, which axis is
nearer to the pixel than other axes. The distance measure
used here has first been described in [3].
3. Pedestrian Tracking
The temporal information along the video frame se-
quence is conjectured to have a potential for greater accu-
racy and speed-up. This point will be discussed here to-
gether with a solution for brief occlusions.
Figure 1. Pedestrian detection using the cur-
vature of the boundary of a silhouette
3.1Weighted temporal texture features
For faster and accurate pedestrian tracking, we selected
the weighted temporal texture features as a supplement to
the appearance model. This should capture the inter-frame
correlation and continuation for successful tracking. Let us
start with the definition; The temporal texture(T) is defined
as a set of pairs of texture values(t) and its temporal weights
(w) as
T = {(t1,w1),(t2,w2),...,(ti,wi)}
where i is the number of intensity clusters.
The texture value, measured in intensity unit, represents
the mean or center of a texture cluster in the intensity space.
The first step of calculating the temporal texture is cluster-
ing textures. For each pedestrian, an intensity histogram is
created and the mean intensity value of pedestrian is calcu-
lated. In this way, two or three intensity clusters for each
pedestrian can be obtained.
The temporal weight is the coherency of the intensity
cluster. It is a function of the size of intensity clusters and
the variance of the intensity values. The relationship be-
tween each parameter is
(3)
wi∝
1
∆sti· ∆mti
(4)
where ∆stiis the difference of number of pixels at texture
value tiand ∆mtiis the difference of mean at texture value
ti.
The velocity of the pedestrian is temporal in that it
changes at every frame while the size of pedestrian is as-
sumedly not temporal, and hence not explicitly used in the
model. This has a nontrivial influence on the amount of
computation. For this reason we include it as a temporal
texture feature.
The weight W is the coherency of the pedestrian. It is a
function of the size of pedestrian, the frequency of the asso-
ciated texture, the set of temporal weight and the existence
of adjacent objects. The relation between each parameter is
0-7695-2128-2/04 $20.00 (C) 2004 IEEE

Page 3

Wn∝
Σwn· TAn
Γ(n) · ∆Sn· ∆pn· ∆vn
(5)
where n is frame number in an image sequence, Snis the
number of pixels in the region of a pedestrian, ∆Sn= Sn−
Sn−1, pisthepositionofapedestrianand∆pn=pn−pn−1,
v = ∆pn/∆t, ∆vn= vn− vn−1, TAnis the duration of
the associated texture, and finally ∆t is the time interval
between frames.
Theadjacency functionΓmeasuresthe2Dgeometricad-
jacency or separation between two pedestrians. It is related
to the shape of target pedestrian and the distance from the
other pedestrian, as:
?
Γp(t) =
Ap
(xtp− xtq)2+ (yt
p− yt
q)2
(6)
where xt, ytis the center coordinate of a target pedestrian
at time t, respectively, and A is the length of the target in
the direction that the adjacency occurs.
3.2 Occlusion detection
Let us consider the situation where two pedestrians are
being tracked through occasional occlusions. We can deter-
mine an approximate depth for each target with a separate
model for occlusion such as:
?
where δ(a,b) =
1,
different pedestrians, R represents the overlapping region
between the two pedestrians, DAB(x) denotes the nearer
pedestrian
OAB= max
x∈R
a = b
otherwise, A and B represent two
δ(IAB(x),DAB(x))
(7)
?
0,
arg min
{A,B}{|x − Ac|,|x − Bc|}
where Ac and Bc are pedestrian centers, IAB(x) denotes
the pedestrian whose intensity at x, IA(x) or IB(x), more
similar to the current observation I(x)
arg min
{A,B}{|I(x) − IA(x)|,|I(x) − IB(x)|}.
4. Experimental Results and Analysis
4.1Experimental environment
Our tracking system was implemented and evaluated
on a Pentium IV–1.7 GHz PC, which Microsoft Windows
2000. The video images were acquired at the rate of 30 fps
using a Meteor II frame grabber and Jai 3CCD camera with
the resolution set to 320 x 240 pixels. The video captured
two or three persons entering an office room together. The
data set includes 600 frames in total.
4.2Experimental results
Fig. 2. shows some results for tracking three people
in different occlusion situations. You can see two persons
are continuously tracked throughout the occlusion. The last
frame of Fig. 2(a) shows the case in which one pedestrian is
missing when two persons overlap. In this case the heads of
the two pedestrians overlap, and one head is lost. Fig. 2(b)
shows a different result obtained by incorporating temporal
weights.
(a) result of detected pedestrian without temporal weight
(b) result of detected pedestrian with temporal weight
Figure2.Trackingexamples, withoutandwith
temporal weights
Now let us assume that we have a complete and high-
performance tracking system. As a probe into the feasibility
of an intelligent integrated system, we combined the track-
ingsystemwithourexistingfacerecognitionsystemforreal
time surveillance task. The face recognizer is based on the
support vector machine as reported in [5]. For detailed de-
scription, please refer to the paper.
Inthecurrentsetupofthesystem, wehavenotyetcarried
out a systematic evaluation. Thus, for now, the best result
we can provide is a few snapshots of the working system.
Fig. 3 shows some frames containing the result of person
tracking and, if sufficiently near, identification.
Note that the pedestrian #1 in frame #68 is changed to
”Kim” in frame #99. The information about the name and
the face is stored in the database.
4.3Performance analysis
In the final set of tests, we compared the performance of
the texture feature used in our system with the shape fea-
ture of the temporal templates used in [3]. The quantitative
comparison of the performance is based on the following
measure [2]
?
|P|
x,y∈P(MP− I(x,y))
(8)
0-7695-2128-2/04 $20.00 (C) 2004 IEEE

Page 4

(a) #68(b) #99
Figure 3. Example of pedestrian tracking and
face recognition
where P is a set of foreground pixels of the target pedes-
?
I(x,y) is the pixel intensity for the texture feature at (x,y),
and |P| is the number of pixels of the target pedestrian in
P.
Fig. 4(c) implies that the stability can improve if we
introduce an occlusion model.
trian, MP =
x,y∈PI(x,y)
|P|
is the mean intensity in P,
5. Conclusion and Further Research
In this paper, we proposed an appearance model using
temporal textures for robust multiple pedestrian tracking in
videostreams. Thetexturefeaturesincludetemporaltexture
value, size, position, velocity, and frequency of the texture
value region and they are temporal in that they vary over
time with associated weights and an occlusion model.
The experimental result shows that the stability of the
proposed method in the tracking. The temporal features are
not complete. Rather they can be further generalized with
any kind of features in the appearance model for the pedes-
trian tracking process. One problem with temporal texture
arises when the pedestrians are in a uniform or clothes with
the same color and intensity. To solve this problem, we
introduced other features such as position and face recogni-
tion result. Although face recognition is the most discrimi-
nating feature, the computational cost of face analysis is too
expensive and the accuracy is often not sufficiently high.
However, we apply face recognition to our system when the
region of pedestrian is sufficiently large for recognition. As
the result of applying face recognition, we could increase
the discrimination and the security level.
Future research will be focused on the tracking system
using multiple cameras. By using multiple cameras, we
believe that we can reduce the tracking error and expand
tracking area.
References
[1] T. Collins et al., ”A System for Video Surveillance and Mon-
itoring:VSAM Finial Report,” Technical report CMU-RI-TR-
(a) shape
(b) texture
(c) texture + occlusion model
Figure 4. Comparison of the stability of each
feature
00-12, Robotics Institute, Carnegie Mellon University, May
2000.
[2] S. Kang, B.-W. Hwang and S.-W. Lee, ”Multiple People
Tracking Based on Temporal Color Feature,” International
Journal of Pattern Recognition and Artificial Intelligence,
Vol. 17, No. 6, 2003, pp. 931-949.
[3] I. Haritaoglu, D. Harwood, and L. S. Davis, ”W4: Who?
When? Where? What? A Real Time System for Detecting
and Tracking People,” Proc. of 3rd Int’l Conf. on Face and
Gesture Recognition, Nara, Japan, April 1998, pp. 222-227.
[4] L. Baoxin and R. Chellappa, ”A Generic Approach to Simul-
taneous Tracking and Verification in video,” IEEE Trans. on
Image Processing, Vol. 11, No. 5, 2002, pp. 530-544.
[5] D. Xi and S.-W. Lee, ”Face Detection and Facial Feature Ex-
traction Using Support Vector Machines,” Proc. of 16th Int’l
Conf. on Pattern Recognition, Quebec City, Canada, August
2002, pp. 209-212.
[6] I. Haritaoglu, D. Harwood, and L. S. Davis, “Hydra: Multi-
ple People Detection and Tracking Using Silhouettes,” Proc.
of 2nd IEEE Workshop on Visual Surveillance , Fort Collins,
Colorado, June 1999, pp. 6-13.
0-7695-2128-2/04 $20.00 (C) 2004 IEEE