Ensemble-based discriminant learning with boosting for face recognition

Abstract

In this paper, we propose a novel ensemble-based approach to boost performance of traditional Linear Discriminant Analysis (LDA)-based methods used in face recognition. The ensemble-based approach is based on the recently emerged technique known as "boosting". However, it is generally believed that boosting-like learning rules are not suited to a strong and stable learner such as LDA. To break the limitation, a novel weakness analysis theory is developed here. The theory attempts to boost a strong learner by increasing the diversity between the classifiers created by the learner, at the expense of decreasing their margins, so as to achieve a tradeoff suggested by recent boosting studies for a low generalization error. In addition, a novel distribution accounting for the pairwise class discriminant information is introduced for effective interaction between the booster and the LDA-based learner. The integration of all these methodologies proposed here leads to the novel ensemble-based discriminant learning approach, capable of taking advantage of both the boosting and LDA techniques. Promising experimental results obtained on various difficult face recognition scenarios demonstrate the effectiveness of the proposed approach. We believe that this work is especially beneficial in extending the boosting framework to accommodate general (strong/weak) learners.

Full text

Page 1

166IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 1, JANUARY 2006
Ensemble-Based Discriminant Learning
With Boosting for Face Recognition
Juwei Lu, Member, IEEE, K. N. Plataniotis, Senior Member, IEEE, A. N. Venetsanopoulos, Fellow, IEEE, and
Stan Z. Li
Abstract—In this paper, we propose a novel ensemble-based
approach to boost performance of traditional Linear Discriminant
Analysis (LDA)-based methods used in face recognition. The
ensemble-based approach is based on the recently emerged tech-
nique known as “boosting.” However, it is generally believed that
boosting-like learning rules are not suited to a strong and stable
learner such as LDA. To break the limitation, a novel weakness
analysis theory is developed here. The theory attempts to boost a
strong learner by increasing the diversity between the classifiers
created by the learner, at the expense of decreasing their margins,
so as to achieve a tradeoff suggested by recent boosting studies
for a low generalization error. In addition, a novel distribution
accounting for the pairwise class discriminant information is
introduced for effective interaction between the booster and the
LDA-based learner. The integration of all these methodologies
proposed here leads to the novel ensemble-based discriminant
learning approach, capable of taking advantage of both the
boosting and LDA techniques. Promising experimental results ob-
tained on various difficult face recognition scenarios demonstrate
the effectiveness of the proposed approach. We believe that this
work is especially beneficial in extending the boosting framework
to accommodate general (strong/weak) learners.
Index Terms—Boosting, face recognition (FR), linear discrimi-
nant analysis, machine learning, mixture of linear models, small-
sample-size (SSS) problem, strong learner.
I. INTRODUCTION
A. Face Recognition
F
engines, biometric identity authentication, human-computer
interaction, and multimedia monitoring/surveillance. Within
the past two decades, numerous FR algorithms have been
proposed, and detailed surveys of the developments in the
area have appeared in the literature [1]–[6]. Among various
FR methodologies used, the most popular are the so-called
appearance-based approaches, which include the three most
well-known FR methods, namely Eigenfaces [7], Fisherfaces
[8], and Bayes Matching [9]. With focus on low-dimensional
statistical feature extraction, the appearance-based approaches
ACE RECOGNITION (FR) has a wide range of appli-
cations, such as face-based video indexing and browsing
Manuscript received March 23, 2004; revised December 24, 2004.
This work was supported in part by the Bell University Laboratories at the
University of Toronto.
J. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos are with The Edward S.
Rogers Sr. Department of Electrical and Computer Engineering, University of
Toronto, ON M5S 3G4 Canada (e-mail: kostas@dsp.toronto.edu).
Stan Z. Li is with the Center for Biometrics and Security Research, Institute
of Automation, Chinese Academy of Sciences, Beijing 100080, P.R. China.
Digital Object Identifier 10.1109/TNN.2005.860853
generally operate directly on appearance images of face object
and process them as two-dimensional (2-D) holistic patterns
to avoid difficulties associated with three-dimensional (3-D)
modeling, and shape or landmark detection [5]. Of the appear-
ance-based FR methods, those based on linear discriminant
analysis (LDA) have shown promising results as it is demon-
strated in [8], [10]–[15]. However, statistical learning methods
such as the LDA-based ones often suffer from the so-called
“small-sample-size” (SSS) problem [16], encountered in
high-dimensional pattern recognition tasks where the number
of training samples available for each subject is smaller than the
dimensionality of the samples. For example, in the experiments
reported here only
are available while the dimensionality of the sample space
is up to
. In addition, the performance of linear
appearance-based methods including LDA often deteriorates
rapidly when face patterns are subject to large variations in
viewpoints, illumination or facial expression. These variations
result in a highly nonconvex and complex distribution of face
images [17]. Thus, the limited success of these methods should
be attributed to their linear nature.
In general, a nonconvex distribution can be handled either
by globally nonlinear models or by a mixture of locally linear
models (or ensemble-based methods as they are known in the
machine learning literature [18]). Globally nonlinear methods
are not without problems. Approaches such as those based on
kernel machines [19]–[26] require the optimization of many de-
sign parameters, tend to overfit easily due to the increased al-
gorithmic complexity, and they are computationally expensive
compared to their linear counterparts. The last point is particu-
larlyimportantfortaskssuchasfacerecognition,whichareper-
formed in a high-dimensional input space. On the other hand,
ensemble-based approaches embody the principle of “divide
and conquer,” by which a complex recognition task is decom-
posed into a set of simpler ones, in each of which a locally
linear pattern distribution can be generalized and dealt with by
a relatively simple linear solution. As such, the ensemble-based
methods are simpler, easier to implement, and more cost effec-
tivecomparedtothenonlinearones.However,mostexistingen-
semble-based FR methods are developed based on traditional
cluster analysis [27]–[30]. As a consequence, a disadvantage to
classification tasks is that the submodels’ division/combination
criteria used in these clustering techniques are not directly re-
lated to the classification error rate (CER) of the resulting clas-
sifier, especially the true CER (often referred to as the general-
ization error rate).
training samples per subject
1045-9227/$20.00 © 2006 IEEE

Page 2

LU et al.: ENSEMBLE-BASED DISCRIMINANT LEARNING WITH BOOSTING FOR FACE RECOGNITION167
B. Ensemble-Based Learning With Boosting
Recently, a machine-learning technique known as “boosting”
has received considerable attention in the pattern recognition
community, due to its usefulness in designing ensemble-based
classifiers[31],[32].Theideabehindboostingistosequentially
employ a base classifier on a weighted version of the training
sample set to generalize a set of classifiers of its kind. Often
the base classifier is also called “learner.” These weights are
updated at each iteration through a classification-error-driven
mechanism. Although any individual classifier produced by the
learner may perform slightly better than random guessing, the
formed ensemble can provide a veryaccurate (strong) classifier.
It has been shown, both theoretically and experimentally, that
boosting is particularly robust in preventing overfitting and re-
ducing the generalization error by increasing the so-called mar-
gins of the training examples [32]–[35]. The margin is defined
as the minimal distance of an example to the decision surface of
classification [36]. For a classifier, a larger expected margin of
training data generally leads to a lower generalization error.
Since its introduction, AdaBoost became known as the most
accurategeneralpurposeclassificationalgorithmavailable[37].
However, the machine-learning community generally regards
ensemble-based learning rules, including boosting and bagging
[38], not suited to a strong and stable learner, such as LDA [35],
[39]. The reason behind this belief is that the effectiveness
of these rules depends, to a great extent, on the learner’s
“instability,” which means that small changes in the training
set could cause large changes in the resulting classifier [35].
On the other hand, it has been found in practical applications
that boosting may fail given a too weak learner [32]. In recent
boosting studies, Murua [40] introduced a useful notion of
weak dependence between classifiers constructed with thesame
training data, and proposed an interesting upper bound on the
generalization errorwithrespecttothemarginsoftheclassifiers
and their dependence. Murua’s bound reveals that to achieve
a low generalization error, the boosting procedure should not
only create the classifiers with large expected margins, but also
keep their dependence low or weak. This suggests in theory
that there exists a tradeoff between the large margins and the
weak dependence.
The requirement for an appropriately weak learner signifi-
cantly restricts the applicability of the boosting algorithms in
practicalapplications,giventhefactthatmostofstate-of-the-art
recognition methods involve the utilization of a strong learner.
Therefore, it is highly desirable to improve the traditional
boosting frameworks, so that they are capable of accommo-
dating more general learners in both the pattern recognition and
machine learning communities.
C. Overview of the Contributions
Inthispaper,anovelweaknessanalysistheoryisdevelopedto
overcome the limitation of the weak learners, which are neces-
sary in existing boosting algorithms. To this end, a new variable
called “learning difficulty degree” (LDD) is introduced along
with a cross-validation method. They are used to analyze and
appropriately regulate the weakness of the classifiers general-
ized by a strong learner via the training data. In addition, a new
loss function with respect to the LDD is proposed to quantita-
tively estimate the generalization power of these produced clas-
sifiers. This is achieved in the loss function by balancing the
averaged empirical error of the classifiers and their mutual de-
pendence. They are two key factors to the generalization error
of the formed ensemble classifier as shown in Murua’s theory
[40].
The proposed weakness analysis theory is applied to boost
the performance of the traditional LDA-based approaches
in complex FR tasks. Thus, the learners in this work are the
LDA-based ones,whichdifferfrom thetraditional learnersused
in boosting at two aspects: 1) They are rather strong and stable
and 2) they are feature extractors rather than pure classifiers.
The latter makes this work similar in spirit to those of Viola,
Tieu and Jones [41]–[43], where the boosting process is viewed
as a feature selection process. Particularly, to boost the specific
LDA-based learners, a new variable called “pairwise class
discriminant distribution” (PCDD) is also introduced to build
an effective interaction mechanism between the booster and
the learner. As a result, a novel ensemble-based discriminant
learning method is developed here under the boosting frame-
work through the utilization of the PCDD and the weakness
analysistheory. In the proposed method, each round of boosting
generalizes a new LDA subspace particularly targeting those
examples from the hard-to-separate pairs of classes indicated
by its preceding PCDD, so that the separability between these
classes is enhanced in the new LDA subspace. The final result
obtained by the process is an ensemble of multiple relatively
weak but very specific LDA solutions. The ensemble-based
solution is able to take advantage of both boosting and LDA.
It is shown by the FR experiments to outperform the single
solutions created by the LDA-based learners in various difficult
learning scenarios, which include the cases with different SSS
settings and the case with increased nonlinear variations.
Therestofthepaperis organized asfollows.InSectionII,we
briefly review the AdaBoost approach and its multiclass exten-
sions. Then, in Section III, the theory and algorithm of how to
boost a LDA-based strong learner are introduced and described
in detail. Section IV reports on a set of experiments conducted
on the FERET face database to demonstrate the effectiveness
of the proposed methodologies. Finally, conclusions are sum-
marized in Section V. In addition, a brief introduction to the
adopted LDA-based learners is given in Appendix I.
II. RELATED WORK
Since the boosting method proposed here is developed from
AdaBoost [31], we begin with a brief review of the algorithm
and its multiclass extensions.
In the case of pattern classification, the task of learning from
examples can be formulated in the following way: Given a
training set,
, containing
class
consisting of a number of exam-
ples
and their corresponding class labels
examples are available in the set. Let
sample space:
, and
. Taking as input such a set
classes with each
, a total of
be the
be the label set:
, the objective of

Page 3

168IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 1, JANUARY 2006
Fig. 1.
or? ? ????? can be used to replace each other during the boosting process.
Algorithm of boosting a LDA-style learner (simply replacing the learner with JD-LDA or EFM in step 3 to obtain B-JD-LDA or B-EFM). Either ? ?????
learning is to estimate a function or classifier
such that
will correctly classify unseen examples
To this end, AdaBoost works by repeatedly applying a given
weak learner to a weighted version of the training set in a se-
ries of rounds
, and then linearly combining these
weak classifiers
constructed in each round into a single
strongclassifier
. Themostinterestingfeatureof AdaBoostis
its surprising ability to reduce the amount of overfitting and the
generalization error of classification, even as
[31], [34]. To explain the property, quite a number of perspec-
tivesonAdaBoosthaveemergedsinceitsintroduction[44].The
dominantamongstthemisthemargintheory,whichregardsAd-
aBoost to be an efficient method for maximizing the margin
[34]. However, many researchers have shown that the margin
theory provides only partial answers to the puzzle [45], [46].
As a result, AdaBoost still remains as a mysterious algorithm,
which is considered one of the most important unsolved prob-
lems in machine learning [37]. On the other hand, the limita-
tion in the theoretical explanation does not seem to hamper the
success of AdaBoost-style approaches in practical applications.
For example, Viola and Jones [43] build the first real-time face
detection system by using AdaBoost, whichis considered a dra-
matic breakthrough in the face detection research.
,
.
becomes large
AdaBoost is originally developed to support binary classifi-
cation tasks. Its multiclass extensions include two variants, Ad-
aBoost.M1 and AdaBoost.M2 [31]. AdaBoost.M1 is the most
straightforward generalization. However, the algorithm halts if
the classification error rate (CER) of the weak classifier
duced in any iterative step is
limitation often terminates the procedure too early, resulting in
insufficient classification capabilities [32], [34]. To avoid the
problem, rather than the ordinary CER, AdaBoost.M2 attempts
to minimize a more sophisticated error measure called “pseu-
doloss,”
, which is expressed as
pro-
%. Research indicates that this
(1)
where
so-called “mislabel distribution” defined over the set of all mis-
labels:
. With the pseudoloss, the boosting process can continue
as long as the weak classifier produced has pseudoloss slightly
better than random guessing. In addition, the introduction of the
mislabel distribution enhances the communication between the
(see steps 7,8 of Fig. 1 for definition) is the

Page 4

LU et al.: ENSEMBLE-BASED DISCRIMINANT LEARNING WITH BOOSTING FOR FACE RECOGNITION 169
learner and the booster. In this way, AdaBoost.M2 can focus the
learner not only on hard-to-classify examples, but more specif-
ically, on the incorrect labels [31]. For all these reasons, we de-
velop the ensemble-based discriminant algorithm proposed in
the next section following the AdaBoost.M2 paradigm.
There are two LDA-based FR approaches (or learners) that
are boosted in this work. One is the so-called “Enhanced Fisher
LDA Model” (hereafter EFM) [13], and the other is called “Re-
vised Direct LDA” (hereafter JD-LDA) [47] proposed by the
authors recently. The EFM method is an improvement of the
Fisherfaces method [8], while the JD-LDA method is a LDA
variant introduced specifically for face recognition in high-di-
mensional, small-sample-size scenarios. For completeness, the
details of the two learners are described in Appendix I. Com-
pared to traditional learners used in the boosting algorithms,
the two LDA-based learners should be emphasized again at the
following two points. 1) They are strong and stable learners,
which can be successfully used as stand-alone procedures in
FR tasks [13], [47], [48]. That obviously contradicts the gen-
eral belief that boosting solutions should operate only on top of
weak learners. 2) The EFM or JD-LDA learner is composed of
a LDA-based featureextractorand anearest centerclassifier.As
it can be seen in Appendix I, the learning focus of such a learner
is on the feature extractor rather than the classifier. It is rather
different at this point from the original boosting design where
the weak learners are used only as pure classifiers without con-
cerning feature extraction. This makes the AdaBoost learning
tend to be an adaptively feature selection process, some of the
ideas seen in [43]. Therefore, accommodating a learner such as
JD-LDA or EFM requires a generalized boosting framework,
which is not restricted by the assumption of the weak learner
availability. To highlight these difference, we call “gClassifier”
the more general classifier produced by the LDA-based learners
in the rest of the paper.
III. BOOSTING A LDA-STYLE LEARNER
A. Interaction Between the LDA Learner and the Booster
To boost a learner, we first have to build a strong connection
between the learner and the boosting framework. In AdaBoost,
this is implemented by manipulating the so-called “sample dis-
tribution,” which is a measure of how hard to classify an ex-
ample. However, we need a more specific connecting variable
in this work, given the fact that the nature of LDA is a feature
extractor, which goal is to find a linear mapping to enhance the
between-class separability of the samples under learning. For
this purpose, a new distribution called “pairwise class discrimi-
nant distribution” (PCDD),
isdevelopedfromthemislabeldistribution
Defined on any one pair of classes
PCDD can be computed at the th iteration as (2), shown at the
bottomofthepage,where
and
, is introduced here. The PCDD
ofAdaBoost.M2.
, the
arethenumberofelements
in classes
aBoost.M2 developments, the mislabel distribution
indicates the extent of difficulty in distinguishing the example
fromtheincorrectlabelbasedonthefeedbackinformation
from the preceding
gClassifiers. Thus,
intuitively considered as a measure of how important it is to dis-
criminate between the classes
rent gClassifier
. Obviously, a larger value of
worse separability between the two classes. It is, therefore, suit-
able to drive a LDA-based learner through
focused specifically on the hard-to-separate pairs of classes. To
this end, rather than the ordinary definition of the between-class
scatter matrix
is the mean of the class
is the average of the ensemble
introduce a variant of
, which can be expressed as
and, respectively. As it is known from the Ad-
can be
andwhen designing the cur-
implies
, so that it is
where
and
), we
with
(3)
It should be noted at this point that the variant
by
embodies the design principle behind the so-called “frac-
tional-step” LDA presented in [49]. According to this principle,
objectclassesthatare difficulttobe separatedinthelow-dimen-
sional output spaces
rounds can potentially result in misclassification. Thus, they
should be paid more attention by being more heavily weighted
inthehigh-dimensionalinputspaceofthecurrent( th)round,so
that their separability is enhanced in the resulting feature space
. It can be easily seenthat the variant
is equal to a constant.
Similarly,theweightedversionofthewithin-classscatterma-
trix
can be given as follows:
weighted
generalized in previous
reduces towhen
(4)
where
sample distribution, similar to the one given in AdaBoost. Since
is derived indirectly from the pseudoloss
a “pseudo sample distribution” for the distinguishing
purpose. It can be seen that a larger value of
harder-to-classify example for those preceding gClassifiers.
Recently, it is shown that to achieve a low generalization
error, the boosting procedure should not only create classifiers
with large expected margins, but also keep their dependence
low or weak [40]. Obviously, classifiers trained with more
overlapping examples will result in stronger dependence
among them. A way to avoid building similar gClassifiers
is defined over as the
, we call
implies a
if
otherwise
(2)

Page 5

170IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 1, JANUARY 2006
repeatedly is to artificially introduce some randomness in the
construction of the training data. To this end, a modified PCDD
is introduced as (5), shown at the bottom of the page, where
. As a result of using
instead of , it can be seen that only those subject sets
that include the examples mislabeled by the last gClassi-
fier
are contributing to the construction of the current
gClassifier
(through). By manipulating
reduce the extent of overlapping between the training exam-
ples used to build different gClassifiers, and thus achieve the
goal of weakening the dependence among these gClassifiers.
Also, this has the effect of forcing every gClassifier to focus
only on the hard-to-separate pairs of classes suggested by its
preceding gClassifier, resulting in a more diverse committee of
gClassifiers to be generalized in the end. On the other hand,
the classification ability of the individual gClassifier
weakened to some extent due to less training examples involved
in its construction. This weakening results in decrease in the
examples’ margins. However, it should be noted at this point
that there appears to be a tradeoff between weak dependence
and large expected margins to achieve a low generalization
error [40]. Our experimentation indicates that in some cases,
the utilization of
may yield a better balance than that
obtained by
, improving the classification performance.
Based on the introduction of
and, we can now give a new boosting solution, depicted
in Fig. 1, from which it can be seen that the LDA-style learner
at every iteration is tuned to conquer a particular subproblem
generalized by the feedback
maticgaincontrol,”andthefinalsolutionisamixtureof
subspaces by weighted linear combination. Either JD-LDA or
EFM can be adopted as the LDA learner in the step 3 during
the boosting process. In the remainder of the paper, we call
“B-JD-LDA” the algorithm utilizing JD-LDA, while “B-EFM”
indicates the one employing EFM.
, we can
is
in a manner similar to “auto-
LDA
B. A Cross-Validation Mechanism to Weaken the Strong
Learner
As we mentioned earlier, JD-LDA or EFM itself has been
a rather strong and stable learner in terms of classification
ability. As a consequence, two problems are often encountered:
1) gClassifiers created exhibit a high similarity or mutual
dependence, given the same training data; 2) the pseudoloss
is often obtained halting the boosting process too
early. To solve the problems, we have to artificially weaken the
gClassifiers and increase their diversity accordingly. Generally
speaking, the learning capacity of any LDA-like algorithm
is directly proportional to the number of training examples
per subject,
, and reciprocally proportional to the number of
the subjects, . Combining the two factors, we can define a
variable called Learning Difficulty Degree (LDD):
to roughly estimate the degree of difficulty for the discriminant
,
Fig. 2.
proposed boosting framework to weaken the LDA-style learner. The flow chart
is based on one iteration, and the NCC denotes the nearest center classifier.
Flow chart of the cross-validation mechanism embedded in the
learning task on hand. It should be noted that the average
is considered as subjects are allowed to
have different number of training examples,
a smaller
value implies a more difficult learning task. In
other words, if a learner is trained with different sample sets,
the classification strength of the obtained gClassifiers will be
different: A sample set with a smaller
gClassifier. Thus, from the training data point of view, the
LDD provides a qualitative measure of the weakness of the
gClassifiers created by the same learner. For the purpose of
distinguishing the two meanings, we denote the LDD as
when it is used to express the degree of difficulty for a learning
task, while
denotes the weakness of a gClassifier.
Based on the above analysis, we can introduce into the pro-
posed boosting framework the cross-validation mechanism de-
picted in Fig. 2 With the mechanism in place, only a subset of
the entire training set
learner. The subset
is formed in each iteration by choosing
the
hardest-to-classify examples per class based on cur-
rent values of
. Please note that
denotes the size of. In the sequence, the obtained LDA
feature extractor
(see Appendix I for details of
sifier,
based on the nearest center rule.
The gClassifier is applied to the entire training set
those unseen, to the learner, examples
ables defined onsuch as
reported and used in the next iteration. The detailed implemen-
tation steps of the mechanism have been embedded in Fig. 1.
. Obviously,
value leads to a weaker
, is used to train the LDA-style
, where
) are used to build a gClas-
including
. All the vari-
(or) are then, and
if
otherwise
(5)