Characterization of the effectiveness of reporting lists of small feature sets relative to the accuracy of the prior biological knowledge.

Abstract

When confronted with a small sample, feature-selection algorithms often fail to find good feature sets, a problem exacerbated for high-dimensional data and large feature sets. The problem is compounded by the fact that, if one obtains a feature set with a low error estimate, the estimate is unreliable because training-data-based error estimators typically perform poorly on small samples, exhibiting optimistic bias or high variance. One way around the problem is limit the number of features being considered, restrict features sets to sizes such that all feature sets can be examined by exhaustive search, and report a list of the best performing feature sets. If the list is short, then it greatly restricts the possible feature sets to be considered as candidates; however, one can expect the lowest error estimates obtained to be optimistically biased so that there may not be a close-to-optimal feature set on the list. This paper provides a power analysis of this methodology; in particular, it examines the kind of results one should expect to obtain relative to the length of the list and the number of discriminating features among those considered. Two measures are employed. The first is the probability that there is at least one feature set on the list whose true classification error is within some given tolerance of the best feature set and the second is the expected number of feature sets on the list whose true errors are within the given tolerance of the best feature set. These values are plotted as functions of the list length to generate power curves. The results show that, if the number of discriminating features is not too small-that is, the prior biological knowledge is not too poor-then one should expect, with high probability, to find good feature sets.

Full text

Page 1

Cancer Informatics 2010:9 49–60
This article is available from http://www.la-press.com.
© the author(s), publisher and licensee Libertas Academica Ltd.
This is an open access article. Unrestricted non-commercial use is permitted provided the original work is properly cited.
Open Access
Full open access to this and
thousands of other papers at
http://www.la-press.com.
Cancer Informatics 2010:9
49
Cancer Informatics
M e T h o d o L o g y
characterization of the effectiveness of Reporting Lists
of small Feature sets Relative to the Accuracy of the prior
Biological Knowledge
Chen Zhao1, Michael L. Bittner2 Robert S. Chapkin3 and edward R. dougherty1,2,4
1department of electrical and Computer engineering, Texas. A&M University, College Station, TX, 77843, USA.
2Computational Biology division, Translational genomics Research Institute, Phoenix, AZ, 85004, USA. 3Center for
environmental and Rural health, Texas A&M University, College Station, TX, 77843, USA. 4department of Bioinformatics
and Computational Biology, University of Texas M.d. Anderson Cancer Center, houston, TX, 77030, USA.
email: edward@ece.tamu.edu
Abstract: When confronted with a small sample, feature-selection algorithms often fail to find good feature sets, a problem exacerbated
for high-dimensional data and large feature sets. The problem is compounded by the fact that, if one obtains a feature set with a low error
estimate, the estimate is unreliable because training-data-based error estimators typically perform poorly on small samples, exhibiting
optimistic bias or high variance. One way around the problem is limit the number of features being considered, restrict features sets to
sizes such that all feature sets can be examined by exhaustive search, and report a list of the best performing feature sets. If the list is
short, then it greatly restricts the possible feature sets to be considered as candidates; however, one can expect the lowest error estimates
obtained to be optimistically biased so that there may not be a close-to-optimal feature set on the list. This paper provides a power
analysis of this methodology; in particular, it examines the kind of results one should expect to obtain relative to the length of the list
and the number of discriminating features among those considered. Two measures are employed. The first is the probability that there is
at least one feature set on the list whose true classification error is within some given tolerance of the best feature set and the second is
the expected number of feature sets on the list whose true errors are within the given tolerance of the best feature set. These values are
plotted as functions of the list length to generate power curves. The results show that, if the number of discriminating features is not too
small—that is, the prior biological knowledge is not too poor—then one should expect, with high probability, to find good feature sets.
Availability: companion website at http://gsp.tamu.edu/Publications/supplementary/zhao09a/
Keywords: classification, feature ranking, ranking power

Page 2

Zhao et al
50
Cancer Informatics 2010:9
Introduction
One of the most challenging issues facing cancer
informatics, and bioinformatics in general, is fea-
ture selection for classification. A typical microarray
contains tens of thousands of genes from which a set
must be chosen to form the features for whatever kind
of discrimination is desired, be it for diagnosis or
prognosis. Not only is feature selection problematic
in high-dimensional settings; it is made more difficult
by small samples, which are commonplace in high-
throughput genomics and proteomics.
The supplementary table lists 20 cancer classifi-
cation studies, showing the sample size, method of
cross-validation error estimation, and classification
problem for each. As we will discuss, finding and val-
idating a single good feature set for these sample sizes
is virtually impossible; on the other hand, reporting a
list of small feature sets can, with high probability,
assure one of obtaining one or more well-performing
feature sets.
Numerous feature-selection algorithms have been
proposed during the last few decades1,2 and this num-
ber has increased dramatically since the advent of
high-throughput genomic technology.3,4 All feature-
selection methods suffer from two problems, one
inherent in the multivariate nature of classification
and the other a consequence of sampling. First, even
given the joint distribution of all the features and
labels, if one wishes to select the best feature set of
size k from among a family of n features, then all fea-
ture sets must be checked to be guaranteed that the
best one is selected.5 Nothing but an exhaustive search
can assure finding the best feature set. Second, even if
a feature-selection algorithm is capable of generally
finding good feature sets given full knowledge of the
joint distribution, in practice, feature selection must
proceed from sample data and here even an exhaus-
tive search can fail to produce a good feature set, even
if one exists, the situation becoming more problem-
atic as sample size decreases.
Perhaps the most well-known consequence
of sample-based feature selection is the peaking
phenomenon.6–9 With full knowledge of the feature-
label distribution, increasing the number of features
cannot produce poorer classification; however, when
using sample data, increasing the number of features
beyond a point can degrade classification. For certain
classification rules and feature-label distributions this
point may consist of very few features when samples
are small. Classically, peaking has been studied in
the absence of feature selection, meaning that the
features are added in a pre-determined manner. Peak-
ing becomes far more complicated in the presence
of feature selection.10 Moreover, classical studies do
not consider the extremely large numbers of features
found in genomics, so that feature-selection perfor-
mance for moderately large feature families must be
re-examined in the framework of high-throughput
biology.11 In sum, the peaking phenomenon argues
for limiting the number of features unless there is rea-
son to believe that peaking will not be a problem.
When applying a feature-selection algorithm, given
a classifier design rule, two basic related questions
arise12: (a) Can one expect feature selection to yield
a feature set whose error is close to that of an optimal
feature set? (b) If a good feature set is not found, should
one conclude that good feature sets do not exist? These
questions translate quantitatively: (a) Given the error
of an optimal feature set, what is the conditionally
expected error of the selected feature set? (b) Given the
error of the selected feature set, what is the condition-
ally expected error of an optimal feature set? Rather
than using the conditional expectation, one can take a
simpler route and look at the linear regression in both
cases. For small samples it is commonplace to have
very little regression in both cases and little correla-
tion between the two errors. Thus, one cannot expect
to find a close-to-optimal feature set or draw any con-
clusion regarding the existence of a good feature set
when a good one is not found.12
Perhaps one might be fortunate and select a good
feature set. But how would it be known that the fea-
ture set is good? Since the sample size is small, error
estimation will be done on the training data and, when
samples are small, error estimators such as cross-vali-
dation and bootstrap give generally poor results,13 and
the performance of these error estimators gets even
worse when used in conjunction with feature selec-
tion.14,15 The problem is that, with small samples and
a large number of features (using feature selection or
not), these error estimators have substantial variance,
especially cross-validation, and they possess little
correlation or regression with the true error.16
To address the dual problems of feature selec-
tion and error estimation, one can invoke two con-
straints. First, the number of potential features can be

Page 3

effectiveness of reporting lists of small feature sets
Cancer Informatics 2010:9
51
cut without using the data by restricting attention to
features known to have some relation with the labels
to be classified, say to a particular cancer of interest,
and by not considering features whose measurements
are suspect, say by throwing away features with miss-
ing values. Second, one can use small feature sets.
Not only does this avoid the peaking phenomenon
and enhance error estimation, it can also avoid fea-
ture selection altogether by facilitating an exhaustive
search of feature sets. By only considering feature sets
of size 1, 2, and 3, so long as the total number of fea-
tures is not too large, one can test every feature set.
Not only does such a restriction mitigate the statistical
and computational issues, it also facilitates biological
understanding. Studies have shown good classification
can be achieved with 2 or 3 genes when re-examining
data from studies that had originally used much larger
feature sets,17,18 with the advantage that the error esti-
mates for the small gene sets are more credible.
While small feature sets help reduce the uncertainty
introduced by feature selection and error estimation,
even an exhaustive search with small feature sets does
not fully overcome the problem.19 Rather than report
a single feature set when samples are small, report-
ing a list of the best performing feature sets increases
the likelihood of finding good features sets, the idea
being that some in the list of top-performing feature
sets will be close to optimal. This strategy has been
taken in a number of cancer classification studies that
give lists of the best performing 1, 2, and 3 gene fea-
ture sets.20–23 Given the list, one can either focus on
the feature sets in the list for further sampling or take
a classical wet-lab approach to determining which
ones are predictive.
For illustration purposes, we briefly describe some
results from a study that designed linear classifiers to
distinguish four types of glioma (leaving details to the
original paper): oligodendroglioma (OL), anaplas-
tic oligodendroglioma (AO), anaplastic astrocytoma
(AA), and glioblastoma multiforme (GM)—in partic-
ular, classification of OL from others, AO from oth-
ers, AA from others, and GM from others.20 The study
involved 25 patients and the gene list was reduced to
597 genes prior to utilizing the data. Table 1 gives
the best 5 single-gene sets and the best 10 two-gene
sets based on the estimated errors. It also gives the
three-gene sets among the top 50 three-gene sets for
which the estimated error of the three-gene set is at
least 0.03 less than the estimated error of its best two-
gene subset. The purpose for placing this requirement
on the marginal gain of a three-gene set over its two-
gene subsets is to avoid redundancy caused by adjoin-
ing features to already strong performing feature sets.
For each feature set, the table gives the error and the
marginal gain.
The risk with this strategy is that, just as selecting
a single feature set may be biased by a low error esti-
mate, and thereby in actuality provide a poor feature
set, an exhaustive list will be affected at the low end
by optimistic error estimates and at the high end by
pessimistic error estimates. The problem is illustrated
in Figure 1, in which the x-axis shows the ranks of the
selected feature sets and the y-axis shows their errors.
Table 1. errors and increments for discriminating Ao from
other gliomas.
Gene1
DNaseX
TNFSF5
RAD50
HBEGF
NF45
DNaseX
DNaseX
TNFSF5
DNaseX
TNFRSF5
TNFSF5
TNFSF5
erbB4
DNaseX
TNFSF5
DNaseX
DNaseX
TNFSF5
DNaseX
TNFSF5
TNFSF5
DNaseX
TNFSF5
TNFSF5
TNFSF5
Gene2 Gene3error
0.1556
0.1658
0.1659
0.1670
0.1731
0.0750
0.0784
0.0826
0.0892
0.0907
0.0909
0.0950
0.1012
0.1013
0.1020
0.0441
0.0454
0.0464
0.0476
0.0526
0.0529
0.0534
0.0591
0.0616
0.0625
Gain
TNFSF5
PTGER4
GNA13
HGF
PTGER4
RAB5A
SNF2L4
PTGER4
β–PPT
MERLIN
TNFSF5
TNFSRF5
RAB5A
PTGER4
GNA13
β–PPT
β–PPT
PKAC–α
LIG4
β–PPT
0.0806
0.0772
0.0832
0.0664
0.0947
0.0749
0.0708
0.0841
0.0544
0.0638
0.0309
0.0330
0.0362
0.0308
0.0300
0.0549
0.0479
0.0488
0.0474
0.0325
RAB5A
PTGER4
GNA13
SAP97
HGF
PKAC–α
RkB
LIG4
HBGF–1
SMARCA4

Page 4

Zhao et al
52
Cancer Informatics 2010:9
The red, green, and black curves show the estimated
error, true error and difference between the true and
estimated errors, respectively. The curves are select-
ing 2 features out of 50 for linear discriminant analy-
sis (LDA) classification in a Gaussian model. At the
top end of the ranking the estimated errors are opti-
mistic; at the low end, they are pessimistic. We only
care about the top end. If the list is too short, then,
there may not be any good features sets in the list; if
the list is too long; then there will likely be good fea-
ture sets but the list will be impractically long.
This paper uses a model-based approach to inves-
tigate the kind of results that can be expected from
generating feature-set lists. This is accomplished via a
characterization of the goodness of the list relative to
the number of potential features and the sample size.
Since our purpose is to quantify the effects of forming
a feature list and since quantification depends on the
number of potential features selected by the biologist
for consideration and the number of those features that
are significant contributors to discrimination, only a
model-based approach can provide meaningful results,
since the contributive and non-contributive features
are not known a priori for real data and could only be
determined with certainty if we knew the distribution
from which the real data arise, which we do not.
systems and Methods
Ranking power
We define a measure of goodness for the list based
on the closeness of the estimate-based feature
sets to optimality. It depends on the feature-label
distribution, the classification rule, the total number D
of features, the number d of features to be selected, the
sample size n, and the list length m. Let Abest be the best
feature set relative to the feature-label distribution,
A(1), A(2), …, A(m) be a list of feature sets, ε0 be the true
error of the classifier for Abest designed on the sample,
ε1, ε2, …, εm be the true errors, listed from lowest to
highest, for A(1), A(2), …, A(m) in the estimated-error list,
and r . 0. We define the ranking power of the list by

∆D d
n r,( )
,
().mPr
=ε10
−<ε
(1)
The ranking power gives the probability that at
least one feature set in the list has error within r of the
best feature set. We would like ∆D d
1 when m is relatively small. In practice, A(1), A(2), …,
A(m) are obtained by ranking the feature sets accord-
ing to their estimated errors,
all feature sets considered. Notice that the ith lowest
estimated error
( )
ˆi
ε corresponds to the feature set A(i)
however, the ith lowest true error εi may not neces-
sarily correspond to the feature set A(i). It could arise
anywhere from the top m list. Our interest is to see if
the top m list can produce at least a close to optimal
feature set.
We consider ranking power curves ∆D d
function of m. For fixed D, d, and r, we are inter-
ested in the minimum m for which ∆D d
(or some other threshold). For small r, the minimum
m gives the length of the list required to get within r
of the best feature set with probability 0.95.
In a sense ∆D d,
of goodness because it requires at least one feature set
satisfying the requirement. We can also consider the
expected number of feature sets in the estimated-error
list satisfying the requirement; to wit, we define
n rm
,
,( ) to be close to
(1)(2) ( )m
ˆ
ε
ˆ
ε
ˆ
ε
,,,
…
, among
n rm
,
,( ) as a
n rm
,
,( )  0:95
n rm
,( ) represents a minimal measure

∆D d
r
ii
m E card
[
r
,
,
):}
n({,
=−
]
ε ε ε0<

(2)
where card denotes cardinality (number of elements
in the set).
In the remainder of this paper we will investigate
properties of these power curves, in particular, how
they are affected by D, d, and the number of marker
features among the total number.
0.6
0.5
Estimated error
True error
Error difference
0.4
0.3
0.2
0.1
0
0 200400600
Ranking
Errors
800 10001200
−0.1
−0.2
Figure 1. error curves for a gaussian model.

Page 5

effectiveness of reporting lists of small feature sets
Cancer Informatics 2010:9
53
Model
We consider a hybrid Gaussian model M containing
marker and noise features. The marker features
come from a two-class Gaussian model Mµ with
equally likely classes and class-conditional den-
sities having common covariance matrix Σµ. One
class mean is located at the origin 
µ
µ
,,,aD
D1
)aa
ers, these are divided among B blocks, and a1, a2, …,
aDµ are evenly spaced between 1 and 0.8, with a1 = 1
and aDµ = 0.8. In this setting, every marker performs
well, but not exactly the same.
The covariance matrix, Σµ, for Mµ is blocked:
0 and the other at
= ( ,
T
2…
µ is the total number of mark-

∑ =
µ
∑
∑
∑













ρ
ρ
ρ

00
0

0

0

0
…
…

…

B blocks


where Σρ has variance σµ
correlation coefficient ρ off the diagonal. Features
from the same block have the same correlation, ρ,
while features from different blocks are uncorrelated,
thereby simulating the situation where genes in the
same pathway are strongly correlated but those in dif-
ferent pathways are uncorrelated.
The noise features are modeled as zero-mean ran-
dom Gaussian noise, with a total of Dn noise features,
with Dn .. Dµ. For the hybrid model M, there is a
total of D = D0 + D1 features. D0 reflects biologists’
input to the classification problem and the larger D0,
the better the classification. In this vein, we will com-
pare the ranking power with respect to different D
and D0. In the simulations, we will assume a given
D0, the size of D0 reflecting the extent of the prior
knowledge, and then D0 and D1 features will be ran-
domly chosen from the useful and noise features,
respectively.
Given the adopted model, for any feature set of
size d, the Bayes classifier and corresponding Bayes
error can be found. Assuming equal covariance matri-
ces and equal prior probabilities for the classes, the
Bayes error for feature set A is given by Φ(–∆/2),
where Φ is the standard normal cumulative distribution
function and ∆ is the Mahalanobis distance between
the class-conditional distributions corresponding to
2 along the diagonal and
A. Abest is the feature set possessing minimum Bayes
error, equivalently, the largest Mahalanobis distance.
Note that we do not have to consider noise features
when searching for the best feature set.
Two points regarding Abest should be recognized.
First, although Abest provides the minimum error rela-
tive to the feature-label distribution, we design clas-
sifiers from samples and the error of the designed
classifier for Abest may not be minimal for a given
sample; nevertheless, Abest represents the gold stan-
dard for the feature-label distribution and therefore
we use the error of its designed classifier as the
benchmark. Owing to the direction of the inequality
in Eqs. 1 and 2, no problem arises should the error
of the classifier designed for Abest not be minimal.
Second, Abest is designed relative to Dµ, not D0. Once
the classification problem is given, Abest should not
change because it indicates the best we can do for the
problem at hand. D0 depends on the extent of the prior
knowledge and the poorer that knowledge, the poorer
we should expect to do compared to Abest.
To find the true error of a designed classifier we
generate a very large test set of independent data from
the feature-label distribution and compute its error
rate on the test set. Estimated errors are computed
by bolstered resubstitution, which has been shown
to perform well in comparison with other training-
sample-based error estimators when it comes to rank-
ing feature sets.19
To illustrate the advantage of obtaining a list of
feature sets by exhaustive search, we compare the list
to ordinary feature selection by computing

ΩD d
n r,( )
FS
mPr
,
(),
=−<εε0

(3)
where εFS is the true error of the feature set found by
feature selection. We use a two-stage approach for
feature selection. The t-test is used in the first stage to
reduce the number of features and sequential forward
search (SFS) is use in the second stage to arrive at a
feature set.
Implementation
We focus on the LDA classification rule, the power
curve method being applicable to any classification
rule. We utilize the following simulation procedures:
Compute ∆D d,
m :
r,( )
n