Stacked Hierarchical Labeling.

Full text

Page 1

Stacked Hierarchical Labeling
Daniel Munoz, J. Andrew Bagnell, and Martial Hebert
The Robotics Institute
Carnegie Mellon University
{dmunoz, dbagnell, hebert}@ri.cmu.edu
Abstract. In this work we propose a hierarchical approach for labeling
semantic objects and regions in scenes. Our approach is reminiscent of
early vision literature in that we use a decomposition of the image in
order to encode relational and spatial information. In contrast to much
existing work on structured prediction for scene understanding, we by-
pass a global probabilistic model and instead directly train a hierarchical
inference procedure inspired by the message passing mechanics of some
approximate inference procedures in graphical models. This approach
mitigates both the theoretical and empirical difficulties of learning proba-
bilistic models when exact inference is intractable. In particular, we draw
from recent work in machine learning and break the complex inference
process into a hierarchical series of simple machine learning subproblems.
Each subproblem in the hierarchy is designed to capture the image and
contextual statistics in the scene. This hierarchy spans coarse-to-fine re-
gions and explicitly models the mixtures of semantic labels that may be
present due to imperfect segmentation. To avoid cascading of errors and
overfitting, we train the learning problems in sequence to ensure robust-
ness to likely errors earlier in the inference sequence and leverage the
stacking approach developed by Cohen et al.
1Introduction
The challenging problem of segmenting and labeling an image into semantically
coherent regions can be naturally modeled as a hierarchical process to interpret
the scene [23]. Typically, a graphical model is used where each node represents
the labels present in some region of the image with dependencies that tie to-
gether multiple regions [3,6]. The nodes at the bottom of the hierarchy provide
low-level discriminative information, while nodes higher up resolve ambiguities
using global information. While these representations seem intuitive, learning
the optimal components of the model is practically intractable due to complex
dependencies. Furthermore, even with simplified representations exact inference
remains intractable [13] and prohibits learning these models. Although training
with exact inference is infeasible, a natural alternative is to use approximate in-
ference. However, as we discuss in the next section, these approximations during
learning can lead to undesirable behavior [16]. Therefore, we move away from a
representation for which training is intractable and toward an approach which
relies on effective components that are simple to train.

Page 2

2Daniel Munoz, J. Andrew Bagnell, and Martial Hebert
Fig.1. A synthetic example of our hierarchical labeling process. Given an image and its
hierarchical decomposition of regions, we sequentially predict the proportion of labels
present (drawn in the dashed boxes) using image features and previous predictions.
In this work, we model low-level information combined with higher-order
reasoning using a hierarchical representation. Our approach is similar to previ-
ous structured models with the key difference that we no longer attempt the
intractable task of finding the mode of the joint posterior distribution using a
generic approximate inference algorithm. Instead we simplify the problem into
a series of subproblems that are specifically trained to perform well for our task.
That is, we train these subproblems to model the relations present in the image
so that the overall prediction is correct. One major advantage of this approach
is that test-time structured prediction is simply a sequence of predictions. Our
contribution is a novel hierarchical algorithm for labeling semantic regions.
An idealized example of our approach is depicted in Fig. 1. We represent the
inference process as a series of predictions along the hierarchy from coarse to
fine. Given an image, we first create a hierarchy of regions that range from very
large regions in the image (including the image itself as one region at the top)
down to small regions (e.g., superpixels) at the bottom. We do not rely on each
region to contain one label; instead we explicitly model the label proportions
in each region. Starting with the entire image, we train a classifier1to predict
the proportions of labels in the image. As we further discuss in Sect. 3, these
predictions are passed to the child level and are used to train another classifier
over the child subregions. The procedure is repeated until the leaves are reached.
1In this work, we refer to a classifier as an algorithm that predicts a distribution over
labels, instead of a single label.

Page 3

Stacked Hierarchical Labeling3
Fig.2. Hierarchy analysis for two images. From left to right: input image with ground
truth overlaid, the segmentation maps for L2 (second level), L4, L6, L8, and most likely
label for each region in L8.
Since we model label proportions over regions: we are robust to imperfect seg-
mentation, we can use features defined over large regions, and we do not make
hard commitments during inference.
Figure 2 illustrates four levels from the hierarchy on two images from the
Stanford Background Dataset (SBD) [8]. Ideally the leaves in the hierarchy are
regions that contain only one label, but as Fig. 2 also illustrates, this assumption
is not always true, especially for more complex scenes. With our hierarchical ap-
proach, we demonstrate state-of-the-art performance on SBD and MSRC-21 [26]
with the added benefit of drastically simpler computations over global methods.
2 Background
2.1Motivation
Random field models in vision have proven to be an effective tool and are also at-
tractive due to their convex nature (assuming no latent states) [19]. Furthermore,
although exact inference is NP-hard over these models, there has been much
recent progress towards efficient approximate inference techniques [12,14]. How-
ever, correctly optimizing these convex models requires exact inference during
learning. Unfortunately, when exact inference cannot be performed, converging
to the optimum is no longer guaranteed [16]. For example, Kulesza and Pereira
[16] demonstrate a case where learning with bounded approximate inference can
prevent the learning procedure from ever reaching a feasible zero empirical risk
solution. Similarly in another example, they show that learning with loopy belief
propagation can diverge.
As we are forced to use approximate inference during learning, the learned
model (e.g., parameters) is tightly tied to the chosen inference procedure in both
theory [29] and in practice [17]. However, learning the best model for the chosen
inference procedure is often still difficult. Due to this fundamental limitation
when training with approximate inference techniques, we move away from the
global probabilistic interpretation used in hierarchical random field formulations,

Page 4

4Daniel Munoz, J. Andrew Bagnell, and Martial Hebert
such as [18]. Instead, in a manner inspired by inference procedures over graphical
models, we propose a novel method using iterative classifiers that are trained to
encode interactions between levels but correspond to no explicit joint probability
distribution.
2.2Related work
Our hierarchical formulation resembles early directed graphical models from
Bouman and Shapiro [3] and Feng et al. [6] for scene analysis. Whereas these
approaches rely on tree-based interactions to enable tractable learning, we no
longer train a graphical model and are not restricted in the types of contextual
cues that we can use. Instead we focus on maximizing what we ultimately care
about: predicting correct labelings. This idea is analogous to the difficult and
non-convex problem of maximizing the marginals [11]. The notion of training
the inference algorithm to make correct predictions is also similar to Barbu [2]
for image denoising, in which a model is trained knowing that an inaccurate, but
fast, inference algorithm will be used. In our approach we break up the complex
structured prediction problem into a series of simpler classification problems, in-
spired by recent works in machine learning focused on sequence prediction [4,5].
In the vision setting, this notion of a series of classification problems is similar
to Auto-context [27], in which pixel classifiers are trained in series using the
previous classifier’s predictions with pairwise information to model contextual
cues. In our work, we go beyond typical site-wise representations that require
entities to contain one label. Because we model label proportions, we can use
features defined over large regions to better represent the context, rather than
an aggregation of site-wise labels. Furthermore, the hierarchy provides spatial
support context between levels and naturally propagates long-range interactions
that may be hard to capture with pairwise interactions. We build on the for-
ward sequential learning approach used and analyzed in [28,10,25] to prevent
cascading errors and leverage the sequential stacking idea to minimize cascaded
overfitting [30,4,15].
3 Stacked Hierarchical Labeling
3.1Overview
Given an image and its hierarchical region representation, we train a series of
classifiers, from coarse to fine, to predict the label proportions in each region in
the level. After a level has been trained, the predicted labels are passed to the
child regions to be used as features that model contextual relationships. Figure 3
illustrates (on test data) how the probabilities for the respective labels increase
and become more precise along three levels in the lower half of the hierarchy. Our
approach is robust to the quality of the segmentation at each level as we explicitly
model that regions may contain multiple labels. Therefore, depending on how
the hierarchy is constructed, our algorithm will learn how regions for different

Page 5

Stacked Hierarchical Labeling5
Fig.3. Refinement in label predictions down the hierarchy. Row 1: Test image (with
ground truth overlaid) and predictions for three levels in the hierarchy. Rows 2-4: The
respective level’s label probability maps, where white indicates high probability.
labels are split between levels. We create the hierarchy using the technique from
Arbelaez et al. [1,22].
The next subsections describe each component of the training procedure.
We first introduce the notations and describe the basic classifier used at each
level (Sect. 3.2). We then describe how predictions from a parent region are
incorporated as features (Sect. 3.3) and how classifiers are trained across levels
in the hierarchy to finalize the procedure (Sect. 3.4).
3.2Modeling Heterogeneous Regions
We denote by K the set of possible labels, L the number of levels in the hierarchy,
T the set of training images, IIthe image data for image I, RIits set of regions
in the hierarchy, and RI,?the set of regions at level ?. For each region r ∈ RI,
we define Yrto be the random variable that represents the label of the region.
For each level ?, we train a probabilistic classifier to match the empirical label
distribution of r ∈ RI,?across all training images. For its simplicity, we use a
generalized maximum entropy classifier qφ?, where φ? : Rd→ R is a function
that defines the distribution:
qφ?(Yr= a|II) =
exp(φ?(fI(r,a)))
k∈Kexp(φ?(fI(r,k))),
?
(1)