GENE SELECTION USING LOGISTIC REGRESSIONS BASED ON AIC, BIC AND MDL CRITERIA

Abstract

In microarray-based cancer classification, gene selection is an important issue owing to the large number of variables (gene expressions) and the small number of experimental conditions. Many gene-selection and classification methods have been proposed; however most of these treat gene selection and classification separately, and not under the same model. We propose a Bayesian approach to gene selection using the logistic regression model. The Akaike information criterion (AIC), the Bayesian information criterion (BIC) and the minimum description length (MDL) principle are used in constructing the posterior distribution of the chosen genes. The same logistic regression model is then used for cancer classification. Fast implementation issues for these methods are discussed. The proposed methods are tested on several data sets including those arising from hereditary breast cancer, small round blue-cell tumors, lymphoma, and acute leukemia. The experimental results indicate that the proposed methods show high classification accuracies on these data sets. Some robustness and sensitivity properties of the proposed methods are also discussed. Finally, mixing logistic-regression based gene selection with other classification methods and mixing logistic-regression-based classification with other gene-selection methods are considered.

Full text

Page 1

New Mathematics and Natural Computation
Vol. 1, No. 1 (2005) 129–145
c ? World Scientific Publishing Company
GENE SELECTION USING LOGISTIC REGRESSIONS
BASED ON AIC, BIC AND MDL CRITERIA
XIAOBO ZHOU
Department of Electrical Engineering, Texas A&M University
College Station, TX 77843, USA
XIAODONG WANG
Department of Electrical Engineering, Columbia University
New York, NY 10027, USA
wangx@ee.columbia.edu
EDWARD R. DOUGHERTY
Department of Electrical Engineering, Texas A&M University
College Station, TX 77843, USA
Department of Pathology, University of Texas M.D. Anderson Cancer Center
Houston, TX 77030, USA
edward@ee.tamu.edu
In microarray-based cancer classification, gene selection is an important issue owing to
the large number of variables (gene expressions) and the small number of experimental
conditions. Many gene-selection and classification methods have been proposed; however
most of these treat gene selection and classification separately, and not under the same
model. We propose a Bayesian approach to gene selection using the logistic regression
model. The Akaike information criterion (AIC), the Bayesian information criterion (BIC)
and the minimum description length (MDL) principle are used in constructing the pos-
terior distribution of the chosen genes. The same logistic regression model is then used
for cancer classification. Fast implementation issues for these methods are discussed. The
proposed methods are tested on several data sets including those arising from heredi-
tary breast cancer, small round blue-cell tumors, lymphoma, and acute leukemia. The
experimental results indicate that the proposed methods show high classification accu-
racies on these data sets. Some robustness and sensitivity properties of the proposed
methods are also discussed. Finally, mixing logistic-regression based gene selection with
other classification methods and mixing logistic-regression-based classification with other
gene-selection methods are considered.
Keywords: Gene microarray; logistic regression; Bayesian gene selection; cancer
classification.
1. Introduction
Given the thousands of genes and the small number of data samples involved
in microarray-based classification, gene selection is a critical issue.15Methods
129

Page 2

130X. Zhou, X. Wang & E. R. Dougherty
based on various algorithms have been proposed in the context of gene classifi-
cation: support vector machines,8genetic algorithms,14perceptrons,12Bayesian
variable selection,13,23and the minimum description length principle for model
selection.10The logistic regression model is an important model for binary data
prediction, regression and classification, and it has been successfully applied to
cancer classification;3,16,17however, gene selection and classification based on
the same logistic regression model has not been addressed, most likely because
no closed form expression for the posterior distribution of the selected genes
exists for logistic regression. Note that such a closed form expression exists
for linear probit regression.13Some variable selection schemes for the logis-
tic regression model have been proposed in the statistics literature,4,18but
they are not suitable for problems with large numbers of variables and small
sample sizes.
In this paper, we propose to use the Akaike information criterion, the Bayesian
information criterion and the minimum description length principle to construct
the posterior distribution of the selected genes for the logistic regression model.
A Gibbs sampler is employed to find the strongest genes based on such a posterior
distribution. Since these methods have very high computational complexities, we
also discuss some numerical techniques to speed up the computation. Further-
more, a gene pre-selection procedure is adopted to reduce the huge number of
genes being considered for selection. After finding the strongest genes, we perform
classification based on the strongest genes using the estimated logistic regressions.
The proposed method is tested on several data sets including those from heredi-
tary breast cancer, small round blue-cell tumors, lymphoma, and acute leukemia.
The experimental results show that the proposed methods can effectively find
important genes consistent with the biological considerations, and the classifi-
cation accuracies are very high. Some robustness and sensitivity properties for
the proposed methods are also discussed. Since gene selection and classification
are separate (but related) tasks, we pair three Bayesian selection methods with
four classifier methods using the heritary breast cancer data. In particular, we
wish to see how using different models for gene selection and classification affects
the results.
2. Problem Formulation
Assume we are interested in classifying whether a particular cancer is present or
not. Let y = [y1,...,ym]Tdenote the class labels, where yi= 1 indicates sample
i has the cancer, and yi= 0 indicates sample i does not have the cancer. Denote
x1,...,xnas the expression levels of n genes. Let xi,j be the measurement of the
expression level of the jth gene for the ith sample. Let X = (xi,j)m,ndenote the

Page 3

Gene Selection Using Logistic Regressions131
expression levels of all genes, i.e.
X =



Gene 1
x1,1
x2,1
...
xm,1
Gene 2
x1,2
x2,2
...
xm,2
···
···
···
...
···
Gene n
x1,n
x2,n
...
xm,n



.
(1)
Let xi ? [xi,1,xi,2,...,xi,n] denote the ith row of the above matrix. We model
πi= P(yi= 1|X) by using a logistic regression model given by
log
1 − πi
where β ? [β1,...,βn]Tcontains the regression coefficients. Equivalently, (2) can
be rewritten as πi = (1 + exp(−xiβ))−1. The likelihood function of the logistic
regression is
?
The corresponding log-likelihood function is given by
πi
= xiβ ? xi,1β1+ ··· + xi,nβn,i = 1,...,m,
(2)
L(β) ?
m
?
i=1
1
1 + exp(−xiβ)
?yi?
1
1 + exp(xiβ)
?(1−yi)
.
(3)
logL(β) = yTXβ −
m
?
i=1
log?1 + exp(xiβ)?.
(4)
Then iterative methods such as the Newton-Raphson procedure can be adopted to
obtain the maximum likelihood estimate of β.6
Define γ as the n×1 indicator vector with the jth element γjsuch that γj= 0 if
βj= 0 (the variable is not selected) and γj= 1 if βj?= 0 (the variable is selected).
The Bayesian variable selection is to estimate γ from the posterior distribution
p(γ|y,X). Given γ, let βγconsists of all nonzero elements of β and let Xγbe the
columns of X corresponding to those γ that are equal to 1. Then (2) is rewritten as
π
1 − π= Xγβγ,
where log
π
estimate γ and the corresponding βγ. Note that no closed form expression exists for
the posterior distribution p(β|γ,y,X), and neither for p(γ|y,X). In what follows,
we propose to construct the posterior distribution p(γ|y,X) based on information
criteria such as the AIC, the BIC and the MDL.
log
(5)
1−π??log
π1
1−π1,log
π2
1−π2,...,log
πm
1−πm
?T. Now the problem is how to
3. Bayesian Gene Selection Based on AIC, BIC or MDL
The indicator vector γ can be modeled as a realization from any prior p(γ) on the
2npossible values of γ given by p(γ) =?n
i=1νγi
i(1−νi)(1−γi), where νi= P(γi= 1)

Page 4

132X. Zhou, X. Wang & E. R. Dougherty
is a prior probability to select the jth gene. This form is actually a Bernoulli dis-
tribution for selecting each gene. Now we define a posterior probability distribution
for p(γ|y,X) as
p(γ|y,X) ∝ exp{−S(γ|y,X)}
n
?
i=1
νγi
i(1 − νi)(1−γi).
(6)
We next specify the form of S(γ|y,X) based on the AIC, BIC and MDL citeria as
follows.
• The Akaike information criterion (AIC). The AIC was first introduced in Ref. 1
to measure a model fitting accuracy. It has been shown that an asymptotically
unbiased estimator of an essential part of the relative entropy can be obtained as
the negative maximum log-likelihood plus a penalty term equal to the dimension
of the parameters in the employed model. For the logistic regression model, it is
given by
S(γ|y,X) = −logL(βγ) + nγ,
where βγis the maximum likelihood estimate of the logistic regression, and
nγ??n
by a Bayesian approach using a uniform prior for the candidate models.20The
posterior distribution of the candidate models is proportional to the negative
maximum log-likelihood plus a penalty term. For the logistic regression model, it
is given by
S(γ|y,X) = −logL(βγ) +nγ
• The minimum description length (MDL) principle. The stochastic complexity
based model selection was introduced in Ref. 19. It becomes a well known model
selection criterion, i.e. the minimum description length principle. For the logistic
regression model, it is approximated by Ref. 18
(7)
j=1γj.
• The Bayesian information criterion (BIC). The model utility can also assessed
2
logm.
(8)
S(γ|y,X) ≈ −logL(βγ) +1
2log|In(βγ)| +
nγ
?
j=1
log|βγ|,
(9)
with In(βγ) = XT
γdiag{π1(1 − π1),...,πm(1 − πm)}Xγ.
Here In(β) is the Fisher information matrix for βγ.
Obviously, the above three criteria have a common form: S(γ|y,X) ? −logL(βγ)+
g(β,γ,m), where g(β,γ,m) is given in (7)–(9) respectively for the three different
criteria.
Based on the posterior distribution (6), a Gibbs sampler can be employed to
estimate all the parameters. We use the following Gibbs sampling algorithm to
estimate {γ,βγ}.
• Draw γ(t)from p(γ|y,X) in (6). Here we set as νj = 15/n based on the total
number of samples m that we have. If νjis chosen as a larger value, then we have

Page 5

Gene Selection Using Logistic Regressions133
found that often times (XγTXγ)−1(which will be used in estimating βγ) does
not exist. In particular, we sample each γ(t)
j
independently from
p(γj|y,X,γi?=j) ∝ exp{−S(γ|y,X)}νγi
• Given the sample of γ(t), obtain the maximum likelihood estimate of βγusing
the method given in Ref. 6.
i(1 − νi)(1−γi),j = 1,...,n.
(10)
In this study, 25,000 Gibbs iterations are implemented with the first 5,000 as burn-
in period. We obtain the Monte Carlo samples as {γ(t), t = 1,...,T}, where T =
25,000. Finally, we count the number of times that each gene appears in {γ(t), t =
5,001,...,25,000}. We define the appearance frequency of a gene as the number of
appearances of this gene divided by the total iteration (i.e. 20,000 here). The genes
with the highest appearance frequencies play the strongest role in predicting the
target gene. We will discuss some implementation issues in the next section.
Bayesian Estimation Using the Strongest Genes: Now assume the genes correspond-
ing to non-zero γ are the strongest genes obtained by the above Bayesian variable-
selection algorithm. We still use xγto denote the profiles of these strongest genes.
After estimating βγusing the maximum-likelihood estimation method, we predict
the tested sample by
P(y = 1|xγ,βγ) =
exp{xγβγ}
1 + exp{xγβγ}.
(11)
4. Fast Implementation
If there are 3,000 gene variables, then for each iteration we have to estimate βγ
3,000 times because we need to sample γj for each gene according to (10). The
computational complexity of the Bayesian gene selection algorithm in the previous
section is very high. Hence, some fast algorithms must be developed to speed up
the computation.
4.1. Pre-selection method
Suppose that the total number of genes is p, and we will only consider n < p
candidates in the Bayesian selection algorithm. We next discuss how to pre-select
the n genes using an F-test. In pattern recognition, we usually adopt the following
criterion: the smaller is the sum of squares within groups and the bigger is the sum
of squares between groups, the better is the classification accuracy. Therefore, we
can define a score using the above two statistics to pre-select genes, i.e the ratio of
the between-group to within-group sum of squares:
?m
i=1
where K the number of classes; p is the total number of original genes (note that
the number of genes n used in the Bayesian selection procedure is much smaller
R(j) ?
i=1
?K−1
k=01(yi=k)(¯ xk,j− ¯ xj)2
?K−1
?m
k=01(yi=k)(xi,j− ¯ xk,j)2,
1 ≤ j ≤ p,
(12)