Making Action Recognition Robust to Occlusions and Viewpoint Changes

Abstract

Most state-of-the-art approaches to action recognition rely on global representations either by concatenating local information in a long descriptor vector or by computing a single location independent histogram. This limits their performance in presence of occlusions and when running on multiple viewpoints. We propose a novel approach to providing robustness to both occlusions and viewpoint changes that yields significant improvements over existing techniques. At its heart is a local partitioning and hierarchical classification of the 3D Histogram of Oriented Gradients (HOG) descriptor to represent sequences of images that have been concatenated into a data volume. We achieve robustness to occlusions and viewpoint changes by combining training data from all viewpoints to train classifiers that estimate action labels independently over sets of HOG blocks. A top level classifier combines these local labels into a global action class decision.

Full text

Page 1

Making Action Recognition Robust to
Occlusions and Viewpoint Changes
Daniel Weinland1, Mustafa¨Ozuysal2�, and Pascal Fua2
1Deutsche Telekom Laboratories, TU Berlin, Germany
daniel.weinland@tu-berlin.de
2Computer Vision Laboratory, EPFL, Switzerland
{mustafa.oezuysal,pascal.fua}@epfl.ch
Abstract. Most state-of-the-art approaches to action recognition rely
on global representations either by concatenating local information in a
long descriptor vector or by computing a single location independent his-
togram. This limits their performance in presence of occlusions and when
running on multiple viewpoints. We propose a novel approach to pro-
viding robustness to both occlusions and viewpoint changes that yields
significant improvements over existing techniques. At its heart is a local
partitioning and hierarchical classification of the 3D Histogram of Ori-
ented Gradients (HOG) descriptor to represent sequences of images that
have been concatenated into a data volume. We achieve robustness to
occlusions and viewpoint changes by combining training data from all
viewpoints to train classifiers that estimate action labels independently
over sets of HOG blocks. A top level classifier combines these local labels
into a global action class decision.
1Introduction
Action recognition has applications in video surveillance, human computer inter-
action, and multimedia retrieval, among others. It is also very challenging both
because the range of possible human motions is so large and because variations
in scene, viewpoint, and clothing add an additional layer of complexity.
Most state-of-the-art approaches compute image-sequence descriptors based
on variants of either sparse interest points [3,11,17,20,24] or dense holistic fea-
tures [9,13,19,22,23]. They integrate information over space and time into a
global representation, bag of words or a space-time volume, and use a classifier,
such as an SVM, to label the resulting representation.
These approaches achieve nearly perfect results on the well-known KTH and
Weizmann datasets [20,1]. These, however, are relatively easy because subjects
are seen from similar viewpoints and against uniform backgrounds. Furthermore,
the motions in the test and training set look very similar, so that test-motions are
well explained as small variations of training ones. Most of the above-mentioned
publications do not report results on difficult multiview datasets, such as the
�This work was funded in part by Swiss National Science Foundation project.

Page 2

2Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
Fig.1. We evaluate our approach on several datasets. (Top) Weizmann, KTH, and
UCF datasets. (Middle) IXMAS dataset, which contains strong viewpoint changes.
(Bottom) Finally, to measure robustness to occlusions, we evaluate the models learned
from the IXMAS dataset on a new dataset that contains substantial occlusions, clut-
tered backgrounds, and viewpoint variations.
IXMAS [27] one, which includes subjects seen from arbitrary viewpoints. Nor do
they discuss what happens when the subjects are partially occluded so that none
of the training samples resembles the observation for the whole body. One must
note that some of these approaches have been tested on the even more challenging
Hollywood [12] dataset. However, the recognition rates on the Hollywood dataset
are much lower, and the action classes contain scene context cues that can be
exploited by scene classification techniques. Such discriminative scene context is
not always present depending on the set of actions and also for tasks that require
action classification in the same scene, such as surveillance or HCI.
To handle occlusions, an alternative to global models is to use part based
ones to make independent decisions for individual body parts and to fuse them
into a global interpretation [7]. However, robustly tracking body parts remains
an open problem, especially in the presence of occlusions. As a result, these
methods have not been tested on sequences containing substantial occlusion.
In this paper, we propose a hybrid approach that uses a local partitioning of
a dense 3DHOG representation in a hierarchical classifier, which first performs
local classification followed by global, to provide robustness to both viewpoint
changes and occlusions. Not only can it handle sequences with substantial oc-
clusions such as in Fig. 1, it also yields significant improvements on the IXMAS
dataset [27] against recent methods explicitly designed with view-invariance in
mind [5,10,27,28]. This is achieved without any performance loss on the Weiz-
mann, KTH, and UCF datasets [1,20,18].
2 Related Work
Early attempts at view independent action recognition [15,16] required indi-
vidual body parts being detected or feature points being tracked over long se-
quences. However, in a typical single-camera setup, it is difficult both to track

Page 3

Making Action Recognition Robust to Occlusions and Viewpoint Changes3
individual limbs and to find feature points in images of people wearing normal
clothes. Current approaches proceed differently and can be partitioned into two
classes depending on whether they represent the spatio-temporal information
densely or sparsely.
Sparse Representations Many approaches rely on 3D interest points, also
known as space-time corners and represent them using SIFT-like 2D descrip-
tors [3,11,17,20,24]. These descriptors are often incorporated into a single his-
togram to be used for classification purposes using a Bag-of-Words (BoW) ap-
proach.
These approaches depend neither on background subtraction nor on exact
localization of the person. They perform particularly well with periodic actions,
such as walking or running that produce many space-time corners. A major
limitation, however, is that all geometric information is lost during the BoW
step and we will show that this results in a drop in performance. Furthermore,
they are not suitable for action sequences that do not contain enough repeatable
space time corners such as aperiodic motions.
Dense Representations The requirement for space time corners can be elimi-
nated by replacing sparse representations with dense ones, such as those provided
by HMAX [21] or HOG [2]. These descriptors can represent 2D gradients, op-
tical flow, or a combination thereof. For instance [22] encodes video sequences
into histograms of 2D HOG descriptor and the biologically inspired approaches
of [9,19] use 2D Gabor-filter responses combined with optical flow. Such dense
representations avoid some of the problems discussed above but require a region
of interest (ROI) around the human body, which is usually obtained by using
either a separate human body detector or background subtraction followed by
blob detection. Nevertheless, they have shown much better performance on the
Weizmann [1] and KTH [20] datasets than sparse representations. Interestingly,
improved performance on some datasets was obtained using BoW-based repre-
sentations when the interest point detection was replaced by dense sampling [24].
View Independence The above described methods have not been designed
with view-independence in mind. To achieve it, several avenues have been ex-
plored. In [5], the change of silhouettes and optical flow with viewpoint is learned
and used to transfer action models from a single source-view into novel target-
views. This requires source and target views that record the same action, which
severely limits the applicability of this technique. By contrast, in [10], actions
are learned from arbitrary number of views by extracting view-invariant features
based on frame-to-frame similarities within a sequence, which yields very stable
features under difficult viewing conditions. However, discarding all absolute view
information results in a loss of discriminative power. For instance, a moving arm
or leg might produce exactly the same self-similarity measures.
Another class of techniques relies on recovering the 3D body orientation from
silhouettes. For example, in [27], 3D models are projected onto 2D silhouettes
with respect to different viewpoints and, in [28], 2D features are detected and

Page 4

4 Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
back-projected onto action features based on a 3D visual hull. Such approaches
require a search over model parameters to find the best match between the 3D
model and the 2D observation, which is both computationally expensive and
known to be relatively fragile. As a result, these techniques are usually only
deployed in very constrained environments.
Occlusion Handling The above mentioned approaches for action recognition
have not been demonstrated on partially occluded action sequences. Recently,
[25] tried to infer occlusion maps from a global HOG-SVM classifier for pedes-
trian detection by analyzing the individual contribution of each HOG block to
the classifier response. However, this approach requires estimation of determin-
istic local occlusion labels based on a globally trained classifier. By contrast, we
directly learn local SVM classifiers, each one tuned to a specific region of the
HOG feature and combine the results without the need of hard decisions.
3 Recognition of Action Classes
Our approach is depicted by Fig. 2. It relies on the 3D extension of the HOG
descriptor [2] to represent image sequences that have been concatenated into a
data volume. The volume is subdivided into equally spaced overlapping blocks
and information within each block is represented by a histogram of oriented 3D
spatio-temporal gradients [11]. The resulting block descriptors are embedded
temporally [26] at each spatial location, providing a discriminative representa-
tion that has fixed dimension independent of the duration of a sequence and
hence can be easily fed to a classifier. By contrast to HOG and BoW, the fea-
ture descriptors are not spatially integrated into a global representation, i.e. by
concatenating the blocks into a single vector (HOG) or by computing a location
independent histogram of the blocks (BoW). Instead each location is individually
encoded using a set of location dependent classifiers. Preserving location depen-
dent information introduces additional discriminative power. Moreover, the local
classifiers let us also estimate probabilities for occlusion, which we use to filter
out contributions from cluttered and occluded regions when finally combining
the local action assignments into a global decision.
In our experiments, we will demonstrate this additional robustness to oc-
clusion over using the standard HOG and BoW. Moreover, and even though
our representation is not view-independent, if trained using samples from differ-
ent viewpoints such as those in the IXMAS dataset [27], our experiments also
demonstrate strong robustness to realistic viewpoint variations. Surprisingly, our
approach not only outperforms similar learning based approaches, but also those
specially designed with view-independence in mind. While our approach can not
generalize to view orientations that are significantly far away from all training
samples, the performance of our approach does not degrade much trained on
the IXMAS data and tested on new recordings acquired in a different setup and
with a wide range of different viewpoints depicted by Fig. 1.

Page 5

Making Action Recognition Robust to Occlusions and Viewpoint Changes5
Fig.2. We use a 3D HOG descriptor to represent a video sequence. Temporal infor-
mation at each grid location is integrated over time using temporal embedding, and
classified using location dependent classifiers. Finally the local results are combined
into a global decision.
3.1 3D Histograms of Oriented Gradients
We use the 3DHOG descriptor introduced in [11]. In difference to [11] we com-
pute the descriptor not at previously detected interest locations, but at densely
distributed locations within a ROI centered around the person. Computing
the descriptor involves the following steps: First, the region to be character-
ized is partitioned into regular cells and a histogram h of 3D gradient orien-
tations is computed in each one. This compactly represents temporal and spa-
tial texture information and is invariant to local deformations. Histograms for
all cells in a small neighborhood are then concatenated into a block descriptor
B = L2–clip([h1,...,hNC]), to which SIFT-like L2normalization with clipping
is applied to increase robustness. Since the blocks overlap with each other, this
yields a redundant representation, which increases discriminative power because
normalization emphasizes different bins in different blocks.
Finally, let
Bp= [B1,...,BNT] , p = 1,...,NP
be the sequence of blocks computed at spatial location p along the time axis,
where NT is the number of overlapping blocks that fit within the duration of
the sequence, and NP is the number of blocks that fit within the ROI centered
around the subject.
As will be discussed in the following Sections, these blocks are the primitives
that we will feed first to the embedding and then to the local classifiers for
recognition purposes. Such individual treatment of HOG blocks is what sets us
apart from the original HOG and BoW computation that combine all blocks
into a global representation, as discussed above. We will show it to be critical
for occlusion handling.
(1)
3.2Block Embedding and Classification
In this Section, we present an effective way to compute the probability that a
block represents a specific action using information from all subsequences along

Page 6

6Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
the temporal axis. To this end, we create a set V = {V1,...,VNV} of NV proto-
type descriptors by randomly sampling the HOG blocks computed for the train-
ing subsequences. Given an action sequence and the block descriptors of Eq.1, we
create an NV-dimensional vector made of the distances of each one of the Vito
the closest block within the sequence. In the case of a sequence belonging to the
training set, some of these distances will be exactly zero since some elements of
V are contained in its set of block descriptors but they may not be the only one
to be small. Prototypes that do not belong to the sequence but resemble one of
the blocks will also be assigned a small value. This Sequence Embedding, which
is inspired by max-pooling of action descriptors [9] and exemplar-based embed-
ding [26], makes the training and recognition much more effective. We discuss it
in more details below.
Let Bp be a sequence of blocks at spatial location p partitioned into NT
overlapping blocks, as defined in Eq.1. We represent Bpin terms of minimum-
distances to the set V of NV prototype descriptors introduced above. We take
the distance of the sequence to each Vito be
d∗
i(Bp) = min
t
d(Bt,Vi) , Bt∈ Bp ,(2)
where d represents the distance between orientation histograms. We compute it
as the χ2-distance
�
which we experimentally found to be more suited for our purposes than both the
squared-Euclidean-distance and Kullback-Leibler divergence. Fig 3 illustrates
the embedding for an action sequence.
d(B,V ) =1
2
k
(hk− vk)2
hk+ vk
,(3)
(a)(b)
Fig.3. Embedding of HOG block sequence. (a) Each exemplar Vi is compared against
all blocks extracted from the sequence using the χ2-distance and the minimum distance
d∗
iis stored in a feature vector that we use for classification. The blocks extracted from
the HOG descriptors overlap to minimize quantization error. (b) Same set of blocks
and exemplars visualized in the space of histograms.

Page 7

Making Action Recognition Robust to Occlusions and Viewpoint Changes7
We then take the resulting set of d∗distances
D∗
p= (d∗
1(Bp),...,d∗
NV(Bp))�∈ RNV, (4)
as input to a classifier trained for location p.
An alternative is to use a local BoW approach that performs the bagging
along the time axis. Each HOG descriptor in the sequence can vote for the closest
words in the vocabulary and a histogram over the vocabulary can be input to the
SVMs. Since NV is much larger than NT in a typical sequence, every descriptor
must vote for multiple words in the vocabulary to avoid quantization effects and
sparse histograms. This can be facilitated by votes that decay exponentially with
the distance between Btand Vi. Optimizing the rate of decay for each dataset
yields comparable performance to the embedding method. However we prefer
the embedding method because it is simpler and does not involve adjusting an
additional parameter to each dataset.
We pick the exemplars Vi from the training set by random sampling. We
experimented with a selection strategy as in [26]. This gave better results with a
small number of exemplars, however using a sufficiently large number (500) the
performance of both approaches was very close. We therefore report results for
random selection since it is simpler.
Finally, we use L2-regularized logistic regression [4] to produce probability
estimates p(c|D∗
of Eq. 4 and Θpis the learned logistic regression weights at position p.
p,Θp) for each class c = 1,...,NC, where D∗
pis the descriptor
3.3Occlusion Handling
The overall framework that we propose resembles that of a global HOG rep-
resentation that is well known for being sensitive to occlusions [25]. We have
introduced the local partitioning and embedding of the 3DHOG descriptor to
preserve the advantages of HOG, while at the same time making it robust to
occlusions. This is achieved by individually classifying each embedded block de-
scriptor and then combining the classification responses from all blocks in a final
stage as detailed in the next Section.
To further improve occlusion robustness, we learn at each location in addition
to the Ncactions a separate class. Thus p(c = NC+ 1|D∗
probability of region p being occluded. If a region is occluded with high probabil-
ity, and because the probability distribution normalizes to one, the probabilities
for all other classes will be reduced. Hence when fusing the results as discussed
in the next Section, such a region will carry reduced weight.
To generate a large variety of potential occlusions during training, we artifi-
cially hide parts of the training images, as shown in Fig. 4. These occluders are
placed so that approximately either the lower part of the body, the right or left
side is occluded. We then calculate for each region the amount of overlap with
the occluding object; if it is higher than a predefined threshold the correspond-
ing HOG block is labeled as belonging to the occluded class during training. In
practice, we found that setting the threshold to 90% yields the best results.
p,Θp) represents the

Page 8

8Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
Fig.4. Sample images from artificially occluded training data used to introduce addi-
tional robustness against occlusions in learned classifiers.
3.4Classifier Combination
The previously described local classifiers produce action probabilities at uni-
formly distributed locations of the HOG window. We have evaluated the follow-
ing strategies to fuse these results into a single decision.
Product Rule Our classifiers produce probabilities p(c|D∗
pendence can be assumed, the natural choice is to combine these by the product
rule p(c|D∗
rameters [4] that are used to convert the classifier outputs into probabilities so
that the resulting probability estimates are not overly confident.
p,Θp). Thus if inde-
1:NP,Θ1:NP) =�
pp(c|D∗
p,Θp). Note that we choose the sigmoid pa-
Sum Rule It is also possible to compute a score for each class by averaging the
probabilities of the individual classifiers, i.e. f(c) =�
accurately estimated.
pp(c|D∗
p,Θp), which can
produce better results than the product rules, when the probabilities are not
Weighted Sum Not every region of the HOG window carries equally dis-
criminative information for each action. Thus, when summing the individual
probabilities from each region they can be weighted accordingly. One way to
choose the weights is via conditional error probabilities p(˜ c|c,p), which repre-
sent the probability that the true class label is ˜ c conditioned on the actual
output c of a classifier. Following [8], a weighed sum can then be computed as
p(˜ c|D∗
fier that is easily confused between several actions will distribute its vote over all
those actions, while a classifier that is very confident in classifying an action will
account its vote only to this action. The conditional error probabilities p(˜ c|c,p)
are estimated from confusion matrices, i.e. by counting how often an observation
is classified as c if the true class label was actually ˜ c.
1:NP,Θ1:NP) =�
p
�
cp(c|D∗
p,Θp)p(˜ c|c,p). Thus, intuitively, a local classi-
Top-Level SVM Classifier Using a hierarchical classification scheme, we can
combine the outputs of all local classifiers into a single feature vector and learn
a global SVM classifier on top of this representation.
As shown in Table 2, when there are no occlusions, the product rule combination
and the SVM classifier perform best, closely followed by the rest. However, as
shown in Table 4, occlusions degrade the performance of the product rule even
below that of the sum rule. This was to be expected since we use artificial
occlusions in training and real ones in the testing sets and the product rule is

Page 9

Making Action Recognition Robust to Occlusions and Viewpoint Changes9
the most sensitive one to biases in the learned probabilities. Overall, we therefore
prefer the weighted sum and SVM based methods since they result in higher or
at least similar classification rates on all datasets.
4 Experiments
We experimented with the well-known Weizmann [1], KTH [20], UCF [18], and
IXMAS [27] datasets. Since none of these datasets includes occluded subjects,
we also acquired and processed our own video sequences involving the actions in
the IXMAS dataset, but with substantial occlusions and cluttered backgrounds.
We implemented two baseline methods to compare our results on this newly
acquired multiview dataset with occlusions. Both methods use the same 3DHOG
features and training data as the local method that we advocate in this paper.
However global SVM, the first baseline method, combines the HOG blocks into
a single feature vector followed by global embedding along the temporal axis
and a linear SVM classifier. This method hence resembles the original global
HOG approach [2] combined with the temporal embedding of [26]. BoW SVM,
the second baseline method, accumulates the HOG blocks into a histogram of
4000 visual words and classifies them using a non-linear SVM with χ2-kernel.
This approach hence resembles the approaches [11,12] and more specifically the
dense 3DHOG representations in [24], except that for comparison purposes we
sample features not at multiple scales, because the local approach and global
HOG also use only a single scale, and we use information only within the same
ROIs centered around the subjects as for the other methods.
To compute the ROIs around people that our approach requires, we proceed
as follows. For KTH, we use the bounding boxes provided by [13]. For UCF
we use the bounding boxes available in the dataset. For Weizmann and IXMAS
we use the background subtracted silhouettes and fit a bounding box around
them. For our new recordings we interactively determine the bounding box in
every first frame of an action, because simple background subtraction can not
accurately detect the partially occluded persons. For all datasets, the ROIs are
scaled and concatenated to produce 48× 64× t cubes, where t is the number of
frames in the sequence.
Unless stated otherwise, we use 16×16×16 pixel blocks subdivided in 2×2×2
cells for 3DHOG, which implies an overlap of 8 pixels in all dimensions. We
compute histograms using the dodecahedron based quantization [11] with 6 ori-
entation bins. For the embedding we use a set of approximately 500 prototypes.
Also, unless stated otherwise, recognition rates are computed by the leave-
one-out method: If K subjects appear in a dataset, we average over K runs,
leaving a different person out of the training set each time.
The recognition speed depends on the length of a sequence and on the HOG
and embedding dimensions used. For the experimental setting on the IXMAS
data, computing the HOG features takes on average 75.5ms per sequence, with
our Matlab implementation on a Core i7 CPU. Since the feature vector is par-
titioned into small blocks, computing each local embedding is fast and the total

Page 10

10Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
Table 1. Comparison of recognition rates (in %) on Weizmann (left), KTH (middle),
and UCF (right) datasets.
Method
Local SVM
Local Weighted
Local Product
Local Sum
Global SVM
BoW SVM
Lin [13]
Schindler [19]
Blank [1]
Jhuang [9]
Thurau [22]
Kl¨ aser [11]
Weizmann
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
99.6
98.8
94.4
84.3
Method
Local SVM
Local Weighted 92.4
Local Product
Local Sum
Global SVM
BoW SVM
Gilbert [6]
Lin [13]
Schindler [19]
Wang [12]
Laptev [12]
Jhuang [9]
Kl¨ aser [11]
Rodriguez [18]
Schuldt [20]
KTH
92.2
92.2
92.0
90.7
89.3
94.5
93.4
92.7
92.1
91.8
91.7
91.4
88.7
71.7
Method
Local SVM
Local Weighted 89.4
Local Product
Local Sum
Global SVM
BoW SVM
Wang [24]
Rodriguez [18]
UCF
90.1
87.7
87.7
85.6
81.2
85.6
69.2
cost of computing them is on average 34ms per sequence. The hierarchical clas-
sification is the fastest step and takes on average 1ms per sequence.
4.1 Weizmann, KTH, and UCF Datasets
The Weizmann dataset consists of videos of 9 actors performing 9 actions. Re-
cently, several approaches reported close to perfect recognition rates on this
relatively easy dataset. Note that existing approaches use slightly different eval-
uation methodologies on the data. Some evaluate on the whole sequences, others
split sequences into multiple subparts, which is possible because most actions are
periodic. We report here results for the full sequences, where our method yields
perfect recognition rates, that is 100%. In Table 1, we summarize our recognition
results and compare them against other approaches.
The KTH dataset consists of 6 actions performed by 25 actors in four different
scenarios. We follow the evaluation procedure of the original paper [20] and split
the data into training/validation (8+8 people) and testing (9 people) sets, and
report results for learning a single model from all scenarios. Note that some of the
approaches use slightly different evaluation schemes, e.g. a leave-one-out cross
validation, or do not require bounding boxes, etc. Optimizing our parameters
on the validation set, we found HOG blocks of size 16 × 16 × 2 subdivided into
2×2×1 cells, and an icosahedron based quantization to give best results. With
this setting, we achieve a recognition rate of 92.4% using the weighted sum
based combination, which is among the best results reported for this dataset. In
Table 1, we summarize our recognition results and compare them against other
approaches.
We also evaluate our approach on the UCF dataset that consists of 10 actions.
Since the publicly available part of the dataset does not contain the videos for
pole vaulting, we report results using the 9 available ones and achieve a mean

Page 11

Making Action Recognition Robust to Occlusions and Viewpoint Changes 11
recognition rate of 90.1% using the SVM which is the best reported result for
this dataset. Note that these are not directly comparable to the reported rate
of 69.2% [18], nevertheless, they demonstrate that our approach generalizes well
to broadcast action videos.
Table 2. Average recognition rates (in %) on IXMAS dataset for different combination
strategies for our local method compared against the global SVM and BoW SVM
baselines.
Method
Local
SVM Product Sum Weighted SVM SVM
Rec. Rate 83.483.5 82.8
Local Local LocalGlobal BoW
82.480.6 71.9
4.2IXMAS Dataset
The IXMAS dataset [27] is a multiview action recognition dataset. It consists of
videos of 10 actors performing each 3 times 11 actions. Each action was recorded
with 5 cameras observing the subjects from very different perspectives and as
shown in Fig. 1, the actors freely choose their orientation for each sequence.
We learn single action models from all camera views. Average recognition
rates for the different combination strategies are shown in Table 2 evaluated on
all cameras. In Table 3 we show individual rates per camera when learning from
all views or individual views, and also compare against other methods that used
the same evaluation methodology on the full IXMAS dataset. For each camera,
we improve upon previously published results.
On the original IXMAS data without occlusions, global SVM performs slightly
worse than any of the local methods. BoW SVM has an approximately 10% worse
71.9% recognition rate which is similar to results reported by other BoW based
approaches [17,14] on the IXMAS dataset.
In summary, we observed that combining training data from multiple view-
points and using a non-invariant dense representation yields comparable recogni-
tion rates than invariant representations. However, performance is adversely af-
fected by local changes in feature statistics. Our local classification step mitigates
this problem. As a result, our local approach performs better than competing
ones.
4.3IXMAS Actions with Occlusions
To demonstrate the generalization power of our approach, we recorded our own
dataset composed of the IXMAS actions, but performed by different actors, who
could be partially occluded. The actions were performed on average 3 times by
6 actors and recorded with 5 cameras. As shown in Fig. 1, actors chose their
orientation freely and the occluding objects were rearranged between each take.
We split the data into two subsets: 395 sequences were recorded without
occlusions, and 698 sequences contain objects partially occluding the actors. We
then evaluate on the two sets by learning from all sequences of the original
IXMAS dataset and by testing on either one of these subsets.

Page 12

12Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
Table 3. Recognition rates (in %) on IXMAS dataset for individual cameras. The left
half of the table shows the results when all cameras are used for training. The other
half shows the results for training using a single camera. Although the camera is fixed,
actors change orientation freely. Note that recognition rates for our approach are very
close for both cases, except for camera 5 that provides a top view of the actor, which
results in a lower rate.
Method
Training with All Cameras
cam1 cam2 cam3 cam4 cam5 cam1 cam2 cam3 cam4 cam5
83.4 86.7 89.9 86.4 87.6 66.4
Local Product 83.5 87.0 88.3 85.6 87.0 69.7 85.8 86.4 88.0 88.2 74.7
Tran [23]80.2———
Liu [14]—76.7 73.3 72.0 73.0
Junejo [10]72.7 74.8 74.5 74.8 70.6 61.2
Reddy [17]72.6 69.6 69.2 62.0 65.1
Yan [28]————
Farhadi [5]58.1———
Weinland [27]57.9 65.4 70.0 54.3 66.0 33.6
Training with Single Camera
all
Local SVM 84.7 85.8 87.9 88.5 72.6
——
—
—
—
76.4 77.6 73.6 68.8 66.1
———
72.0 53.0 68.0 63.0
———
55.2 63.5—
—
—
—
—
—
—
—
—
—
—
—
——
—
—
—
—
——
60.0
Results are shown in Fig. 5 and Table 4. Columns clean in Table 4 show
results on the occlusion free sequences. This is relevant because it still requires
that our approach generalizes to new viewpoints and actors not included in the
training data. We observe the best recognition rate of 83.5% for the SVM based
combination, which is similar to the results that the method achieved on the
original IXMAS data. Because the sequences contain no occlusions, also the per-
formance of the global HOG classifier generalizes well to this sequences (74.4%).
Interestingly, when introducing the additional occlusion classifier (columns oc),
performance on the dataset improves (86.3% for SVM combination), even though
it contains no occlusions. This is because the occlusion classifier also responds to
background clutter, reducing its effect on classification. Note, that for columns
oc the baseline classifiers were trained using all clean as well as all artificially
occluded sequences as a single training set.
When evaluating on the sequences with occlusions (columns occluded) the
effect of the additional occlusion classifier (columns oc) becomes even more evi-
dent. We observe the best performance with 76.7% recognition rate for the SVM
based combination and also for the weighted sum, which is around 15% better
than their results without using the occlusion classifier (columns no oc). Com-
bination via sum rule performs slightly worse, but all three strategies clearly
outperform the Global and BoW SVM baselines.
In all cases, our experiments demonstrates that using local classifiers as well
as explicitly introducing occlusions into the training set leads to strong perfor-
mance improvements for recognition of partially occluded actions.
5 Conclusion
In this paper, we proposed a new approach based on a local 3D HOG descriptor.
Our approach is simple, efficient, and combines the benefits of the HOG based

Page 13

Making Action Recognition Robust to Occlusions and Viewpoint Changes 13
Table 4. Average recognition rates (in %) when learning from IXMAS dataset and
testing on new clean and occluded recordings. Results are shown for learning models
with (oc) and without (no oc) the additional occlusion class. In all cases our local
combination strategy outperforms the baselines.
Method
cleanoccluded
no oc ocno oc oc
83.5 86.3 61.9 76.7Local SVM
Local Weighted 83.3 85.1
Local Sum
Local Product
Global SVM
BoW SVM
61.6 76.7
54.0 72.8
44.6 68.9
46.1 58.3
18.1 27.8
79.0 82.5
77.7 81.5
74.4 76.0
47.1 52.9
dense representation with that of local approaches to achieve occlusion robust
action recognition. We demonstrated that our descriptor, when trained from
multiple views, can perform action recognition from multiple viewpoints, with
highest recognition rates on the difficult IXMAS dataset. Moreover, we showed
that these results carry over to new situations, with different backgrounds, sub-
jects, viewpoints, and partial occlusions.
Fig.5. Confusion matrixes (in %) for the new IXMAS recording. (Left) Recordings
without occlusions with average recognition rate 86.3%. (Right) Recordings with oc-
clusions with average recognition rate 76.7%.
References
1. Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R.: Actions as space-time
shapes. In: ICCV. pp. 1395–1402 (2005)
2. Dalal, N., Triggs, B.: Histograms of Oriented Gradients for Human Detection. In:
CVPR (2005)
3. Dollar, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse
spatio-temporal features. In: VS-PETS. pp. 65–72. Beijing, China (October 2005)
4. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: A li-
brary for large linear classification. Journal of Machine Learning Research 9, 1871–
1874 (2008)

Page 14

14Daniel Weinland, Mustafa¨Ozuysal, and Pascal Fua
5. Farhadi, A., Tabrizi, M.K.: Learning to recognize activities from the wrong view
point. In: ECCV. pp. 154–166 (2008)
6. Gilbert, A., Illingworth, J., Bowden, R.: Fast realistic multi-action recognition
using mined dense spatio-temporal features. In: ICCV (2009)
7. Ikizler, N., , Forsyth, D.: Searching video for complex activities with finite state
models. In: Forsyth, D. (ed.) CVPR. pp. 1–8 (2007)
8. Ivanov, Y., Heisele, B., Serre, T.: Using component features for face recognition.
In: FG (2004)
9. Jhuang, H., Serre, T., Wolf, L., Poggio, T.: A biologically inspired system for action.
In: ICCV (2007)
10. Junejo, I., Dexter, E., Laptev, I., Prez, P.: Cross-view action recognition from
temporal self-similarities. In: ECCV. Marseille, France (October 2008)
11. Kl¨ aser, A., Marsza� lek, M., Schmid, C.: A spatio-temporal descriptor based on 3d-
gradients. In: BMVC (2008)
12. Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human
actions from movies. In: CVPR (2008)
13. Lin, Z., Jiang, Z., Davis, L.S.: Recognizing actions by shape-motion prototype
trees. In: ICCV (2009)
14. Liu, J., Shah, M.: Learning human actions via information maximization. In:
CVPR (2008)
15. Parameswaran, V., Chellappa, R.: View invariants for human action recognition.
In: Chellappa, R. (ed.) CVPR. vol. 2, pp. II–613–19 vol.2 (2003)
16. Rao, C., Yilmaz, A., Shah, M.: View-invariant representation and recognition of
actions. IJCV 50(2), 203–226 (2002)
17. Reddy, K.K., Liu, J., Shah, M.: Incremental action recognition using feature-tree.
In: ICCV (2009)
18. Rodriguez, M.D., Ahmed, J., Shah, M.: Action mach a spatio-temporal maximum
average correlation height filter for action recognition. In: CVPR. pp. 1–8 (23–28
June 2008)
19. Schindler, K., van Gool, L.: Action snippets: How many frames does human action
recognition require? In: CVPR. pp. 1–8 (23–28 June 2008)
20. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local svm ap-
proach. In: ICPR. pp. 32–36 (2004)
21. Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual
cortex. In: CVPR (2005)
22. Thurau, C., Hlavac, V.: Pose primitive based human action recognition in videos
or still images. In: CVPR. pp. 1–8 (23–28 June 2008)
23. Tran, D., Sorokin, A.: Human activity recognition with metric learning. In: ECCV
(2008)
24. Wang, H., Ullah, M.M., Kl¨ aser, A., Laptev, I., Schmid, C.: Evaluation of local
spatio-temporal features for action recognition. In: BMVC. p. 127 (September
2009)
25. Wang, X., Han, T.X., Yan, S.: An hog-lbp human detector with partial occlusion
handling. In: ICCV (2009)
26. Weinland, D., Boyer, E.: Action recognition using exemplar-based embedding. In:
CVPR (2008)
27. Weinland, D., Boyer, E., Ronfard, R.: Action recognition from arbitrary views
using 3d exemplars. In: ICCV (2007)
28. Yan, P., Khan, S.M., Shah, M.: Learning 4d action feature models for arbitrary
view action recognition. In: CVPR (2008)