A comparative study of discriminating human heart failure etiology using gene expression profiles.

Abstract

Human heart failure is a complex disease that manifests from multiple genetic and environmental factors. Although ischemic and non-ischemic heart disease present clinically with many similar decreases in ventricular function, emerging work suggests that they are distinct diseases with different responses to therapy. The ability to distinguish between ischemic and non-ischemic heart failure may be essential to guide appropriate therapy and determine prognosis for successful treatment. In this paper we consider discriminating the etiologies of heart failure using gene expression libraries from two separate institutions.
We apply five new statistical methods, including partial least squares, penalized partial least squares, LASSO, nearest shrunken centroids and random forest, to two real datasets and compare their performance for multiclass classification. It is found that the five statistical methods perform similarly on each of the two datasets: it is difficult to correctly distinguish the etiologies of heart failure in one dataset whereas it is easy for the other one. In a simulation study, it is confirmed that the five methods tend to have close performance, though the random forest seems to have a slight edge.
For some gene expression data, several recently developed discriminant methods may perform similarly. More importantly, one must remain cautious when assessing the discriminating performance using gene expression profiles based on a small dataset; our analysis suggests the importance of utilizing multiple or larger datasets.

Full text

Page 1

BioMed Central
Page 1 of 15
(page number not for citation purposes)
BMC Bioinformatics
Open Access
Research article
A comparative study of discriminating human heart failure etiology
using gene expression profiles
Xiaohong Huang1, Wei Pan*1, Suzanne Grindle2, Xinqiang Han2,
Yingjie Chen2, Soon J Park2, Leslie W Miller2 and Jennifer Hall2
Address: 1Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN 55455, USA and 2Cardiovascular Division,
Department of Medicine, Medical School, University of Minnesota, Minneapolis, MN 55455, USA
Email: Xiaohong Huang - xiaohong@biostat.umn.edu; Wei Pan* - weip@biostat.umn.edu; Suzanne Grindle - grind001@umn.edu;
Xinqiang Han - hanxx057@umn.edu; Yingjie Chen - chenx106@umn.edu; Soon J Park - jlhall@umn.edu; Leslie W Miller - mille278@umn.edu;
Jennifer Hall - jlhall@umn.edu
* Corresponding author
Abstract
Background: Human heart failure is a complex disease that manifests from multiple genetic and
environmental factors. Although ischemic and non-ischemic heart disease present clinically with
many similar decreases in ventricular function, emerging work suggests that they are distinct
diseases with different responses to therapy. The ability to distinguish between ischemic and non-
ischemic heart failure may be essential to guide appropriate therapy and determine prognosis for
successful treatment. In this paper we consider discriminating the etiologies of heart failure using
gene expression libraries from two separate institutions.
Results: We apply five new statistical methods, including partial least squares, penalized partial
least squares, LASSO, nearest shrunken centroids and random forest, to two real datasets and
compare their performance for multiclass classification. It is found that the five statistical methods
perform similarly on each of the two datasets: it is difficult to correctly distinguish the etiologies of
heart failure in one dataset whereas it is easy for the other one. In a simulation study, it is confirmed
that the five methods tend to have close performance, though the random forest seems to have a
slight edge.
Conclusions: For some gene expression data, several recently developed discriminant methods
may perform similarly. More importantly, one must remain cautious when assessing the
discriminating performance using gene expression profiles based on a small dataset; our analysis
suggests the importance of utilizing multiple or larger datasets.
Background
Human heart failure is a complex disease diagnosed in
over 500,000 American people every year, causing more
than 250,000 deaths annually. It may arise from coronary
atherosclerosis, exposure to toxins, infection, inflamma-
tion, valvular disease leading to volume/pressure over-
load, or an underlying genetic or idiopathic event [1-3].
Emerging work suggests the heterogeneity of heart failure.
For example, patients with ischemic heart failure have
decreased survival compared to the non-ischemic heart
failure [4,5] and respond differently to therapies [6-9].
Although benefits can be achieved using ischemic heart
Published: 24 August 2005
BMC Bioinformatics 2005, 6:205 doi:10.1186/1471-2105-6-205
Received: 13 April 2005
Accepted: 24 August 2005
This article is available from: http://www.biomedcentral.com/1471-2105/6/205
© 2005 Huang et al; licensee BioMed Central Ltd.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 2

BMC Bioinformatics 2005, 6:205 http://www.biomedcentral.com/1471-2105/6/205
Page 2 of 15
(page number not for citation purposes)
failure therapy for idiopathic heart failure (and vice
versa), cure rates will be markedly diminished, and
unwarranted toxicities problems will be encountered. It
may be critical to distinguish these characteristically simi-
lar but clinically somewhat distinct heart failures, to better
optimize therapy. The ability to distinguish between dif-
ferent etiologies of heart failure may be essential to guide
appropriate therapy and determine prognosis for success-
ful treatment.
A new approach to discriminating etiologies of heart fail-
ure is gene expression profiling using DNA microarray
technology, which has been shown to be promising in the
diagnosis of human diseases or subdiseases, especially in
cancer [10-12]. Recent genomic studies by three separate
groups have demonstrated a distinct etiology dependent
genomic pattern in the failing heart [13-16]. These studies
offer hope that the microarray gene expression analysis
could potentially add to conventional laboratory
approaches to diagnose different underlying etiologies of
heart failure while simultaneously enhance prognostic cri-
teria. It was hypothesized that heart failure arising from
different underlying etiologies present with different gene
expression patterns and that these differences could be
used as a diagnostic tool. Here we test the hypothesis with
two human heart failure datasets from different
institutions.
Sample classification with gene expression data is statisti-
cally challenging due to the "small n, large p" problem
[17]: the number of samples n is much smaller than the
number of genes or predictors p. In our first dataset, we
have n = 30 and p > 20000. Many new statistical methods
have been developed or adapted to face the challenge.
With more and more new methods emerging and existing
methods being adapted, it becomes increasingly compel-
ling for practitioners to compare and assess their perform-
ance, but there are few such comparative studies [18-20].
Huang and Pan [19] compared several methods, includ-
ing partial least squares (PLS) [21], nearest shrunken cen-
troid (SC) [12], and a penalized PLS (PPLS), for binary
classification of gene expression data. They found that
these methods are competitive. More recently, some
authors [22,20] have shown the promising performance
of least absolute shrinkage and selection operator
(LASSO) [23] and random forest (RF) [24]. It is our main
goal to evaluate and compare these methods using two
human heart failure datasets. For this purpose, we also
extend the PPLS, originally proposed for binary classifica-
tion, to multiclass classification. We found that the above
five statistical methods perform similarly. Furthermore,
our analysis stresses the importance of utilizing multiple
datasets for classification purposes.
Results
Minnesota data
Myocardial tissue samples from the left ventricular apex of
patients with severe refractory heart failure were collected
at the time of the left ventricular assist device (LVAD)
placement at the University of Minnesota Medical School.
A total of 30 tissue samples were processed for microarray
analysis on the Affymetrix Human Genome U133A chip
containing ~22,000 genes. The initial data analysis was
completed using Affymetrix Microarray Suite (MAS 5.0). A
more complete description on the data is provided in
[25].
The heart failure patients are divided into three classes
according to the underlying etiology. Patients with clinical
ECHO and EKG evidence, history of previous myocardial
infarctions, and direct observation of the heart for confir-
mation of infarction at the time of LVAD implantation are
defined as ischemic. Patients with an ischemic etiology
were further divided into two classes: patients with
ischemia but without acute myocardial infarctions
(ischemic class) and patients with ischemia that have had
an acute myocardial infarctions within ten days of LVAD
implant (IM class), the remaining patients were assigned
to the idiopathic class. Among the total 30 samples, 10 of
them are ischemic, 7 are IM and 13 are idiopathic.
PGA data
The PGA data were obtained in another heart failure study
conducted at the Cardio-Genomics PGA (Programs for
Genomic Applications) at the Harvard Medical School.
Myocardial samples were collected from patients under-
going heart transplantation whose failure arises from dif-
ferent etiologies (e.g. idiopathic, ischemic, alcoholic,
valvular, and hypertrophic) and from normal organ
donors whose hearts were not used for transplants. The
transcriptional profile of the mRNA in these samples was
also measured with Affymetrix oligonucleatide microarray
technology. HG-U133 plus 2 chips containing 54,675
probe sets were used and data were analyzed in MAS 5.0.
In the PGA data set there were 11 normal samples, 11
ischemic samples and 14 idiopathic samples. The PGA
dataset is publicly available at Genomics of Cardiovascu-
lar Development, Adaptation, and Remodelling. NHLBI
Program for Genomic Applications, Harvard Medical
School with URL: http://www.cardiogenomics.org.
In order to make the results comparable to those based on
the HG-U133A chips used in the Minnesota data, we
matched the probe sets on a HG-U133 plus 2 chip with
those on a HG-U133A chip. Only six probe sets on the
HG-U133 plus 2 chip could not be found on a HG-U133A
chip. Hence we used the remaining 22277 ( = 22283 - 6)
probe sets in the following analyses with the PGA data.

Page 3

BMC Bioinformatics 2005, 6:205http://www.biomedcentral.com/1471-2105/6/205
Page 3 of 15
(page number not for citation purposes)
Classification with Minnesota data: One-against-others
approach
Table 1 reports the LOOCV misclassification errors of the
five classification methods for the Minnesota data with
models starting from different numbers of top ranked
genes ranging from 50 to all genes. Four kinds of errors are
reported for each method: three one-against-others two-
class LOOCV misclassification errors (Ischemic vs. others,
IM vs. others and Idiopathic vs. others) and one three-
class LOOCV misclassification error. Note that through-
out this article, three-class classification results for SC and
RF were obtained by direct applications of SC and RF,
rather than by combining multiple binary classifications.
Based on Table 1, we first note that the performances of
any classifier is sensitive to the number of top ranked
genes one starts with. For example, in the three-class clas-
sification, LASSO made 8 errors when only the top 200
genes are considered but made 18 errors when all 22283
genes are used. But there is no obvious relationship
between the gene subset size and the five methods'
performances.
In the two-class classification problem of ischemic vs. oth-
ers, PPLS, PLS, SC, LASSO and RF yielded LOOCV errors
range from 7 to 15, 6 to 13, 8 to 13, 9 to 15 and 10 to 12
respectively. The five methods obtained similar numbers
of errors in all instances. In the two-class classification
problems of IM vs. others and idiopathic vs. others, all
five methods yielded similar numbers of errors. In the
overall three-class classification, again all the five classifi-
cation methods perform similarly. There is no clear evi-
dence that one method is clearly superior to others.
In order to assess whether a gene expression profile is
affected by gender, we classified the 23 samples from male
patients only. Among the 23 samples for male patients, 9
of them are ischemic, 7 are IM and 7 are idiopathic.
Table 2 reports the LOOCV misclassification errors of
these five methods for the Minnesota data with males
only. Again, the models start from different numbers of
top ranked genes. The three two-class LOOCV misclassifi-
cation errors and one three-class LOOCV misclassification
error are estimated for each method as described before.
Based on Table 2, again we note that the classification per-
formances of PPLS, PLS, SC, LASSO and RF can be quite
sensitive to the number of top ranked genes one uses. For
example, in the one-against-others two-class classification
problem of ischemic vs. others, PPLS made 10 errors
when the top 400 genes are considered but the number of
errors suddenly increases to 14 when the top 800 genes
are used. Again there is no obvious relationship between
the gene subset size and the five methods' performances.
Comparing PPLS, PLS, SC, LASSO and RF, we find that the
five methods perform very similarly in almost all
instances. There is no evidence that any method is clearly
the best. One thing we noticed about LASSO is that when
the model contains many genes, say top 12800, then it
gives more errors than PPLS, PLS, SC and RF in the three-
class classification. This could happen by chance since we
did not observe the same trend in the decomposed one-
against-others binary classifications.
As in Table 1, the results in Table 2 suggest that discrimi-
nating ischemic group from the other two groups was less
Table 1: LOOCV errors for three-class classification with Minnesota data: all patients.
# of top
genes
Isch vs other IM vs other Idio vs otherOverall
PPLSPLSSC LSO RF PPLSPLSSCLSO RF PPLSPLS SC LSO RFPPLSPLS SC LSORF
50
100
200
400
800
1600
3200
6400
9600
12800
16000
19200
22283
7
9
9
11
8
12
10
11
10
13
13
13
13
9
6
7
11
11
12
12
11
11
12
12
12
13
8
8
12
13
12
11
9
10
10
11
13
13
11
11
9
9
10
13
11
15
15
15
15
15
15
15
12
11
11
12
12
11
10
10
10
11
10
11
10
6
6
7
8
9
9
9
9
10
7
8
9
7
8
7
9
9
9
8
9
10
8
7
7
7
6
6
6
7
8
8
8
7
9
7
7
7
7
7
7
8
5
6
8
9
9
9
9
9
9
8
9
8
8
8
7
7
7
7
7
7
7
8
10
6
10
8
12
9
9
7
8
8
8
7
6
9
7
8
8
8
7
8
8
8
8
8
7
5
5
5
8
7
5
5
5
4
5
5
5
5
8
11
6
8
10
7
10
8
9
10
10
10
10
6
8
7
8
8
7
8
8
10
9
9
8
10
14
11
12
16
15
16
14
15
14
15
15
15
14
11
9
11
15
17
17
14
14
12
13
13
14
14
11
11
12
14
15
13
14
13
15
15
14
14
14
12
13
8
11
14
11
14
15
16
17
17
17
18
14
13
14
15
16
13
13
12
15
17
16
17
16
10
10
11

Page 4

BMC Bioinformatics 2005, 6:205http://www.biomedcentral.com/1471-2105/6/205
Page 4 of 15
(page number not for citation purposes)
accurate than distinguishing IM from the other two
groups and separating idiopathic group from the other
two groups.
If we compare Table 1 (30 patients) with Table 2 (23 male
patients), we can see that the misclassification error rates
of all the five methods in Table 2 are much higher than
those in Table 1. Reduced sample size is likely a factor.
To see whether the high prediction errors are due to the
presence of the three classes, we considered a binary clas-
sification problem. We applied all the five methods to the
10 ischemic and 13 idiopathic samples. We also assessed
the classification accuracy on the male patients with 9
ischemic and 7 idiopathic samples. The classification
results were shown in Table 3.
From Table 3, we can see that all the five methods have
very similar performances in classifying ischemic and idi-
opathic samples. If we compare the classification perform-
ances of these five methods with/without female patients,
taking the sample size into consideration, we can see that
the misclassification error rates with only male patients is
much higher than those with all patients. This again is
probably because the sample size (16) with only males is
smaller. In particular, we noticed that LASSO is more sen-
sitive to the small sample size.
Table 2: LOOCV errors for three-class classification with Minnesota data: males only.
# of top
genes
Isch vs other IM vs otherIdio vs otherOverall
PPLS PLS SCLSO RF PPLS PLSSC LSORFPPLS PLSSC LSO RFPPLS PLS SCLSORF
50
100
200
400
800
1600
3200
6400
9600
12800
16000
19200
22283
11
10
12
10
14
15
11
11
9
9
13
13
12
9 11
10
16
15
13
12
11
13
14
15
15
11
9
10
11
10
10
9
7
10
10
9
10
10
10
10
12
11
12
12
12
10
11
11
10
10
9
9
11
7
5
6
6
6
6
7
6
7
5
8
6
7
6
7
7
8
7
4
7
8
7
8
6
6
5
5
5
6
7
6
7
7
6
6
5
5
5
5
7
8
8
7
8
8
8
11
8
8
8
8
8
7
7
5
7
7
6
8
7
7
8
7
7
8
5
5
6
5
5
5
5
7
8
8
9
7
5
6
6
6
7
6
5
5
6
7
8
7
6
5
6
6
7
8
8
8
5
4
4
5
5
6
6
6
5
7
6
6
8
8
8
8
8
8
8
8
5
8
8
9
9
8
8
8
8
7
7
8
8
15
10
13
14
12
13
14
11
13
12
14
14
15
12
12
12
14
11
12
11
14
13
14
12
12
11
11
11
11
11
12
13
14
15
13
15
14
13
14
14
13
13
14
14
13
15
19
19
20
20
20
20
15
14
15
13
15
17
15
14
15
15
17
17
15
10
12
12
12
15
11
12
11
10
12
10
11
Table 3: LOOCV errors for two-class classification with Minnesota data: ischemic vs idio-pathic.
# of top
genes
All (23 samples) Males (16 samples)
PPLS PLSSC LASSO RF PPLS PLSSCLASSORF
50
100
200
400
800
1600
3200
6400
9600
12800
16000
19200
22283
10
7
9
6
6
5
9
7
9
8
8
11
9
6
7
9
8
9
8
10
7
8
8
7
8
10
5
5
7
7
8
8
9
9
8
8
7
6
5
6
7
11
11
4
8
8
8
7
8
8
7
7
6
6
6
8
7
8
10
9
8
11
9
11
9
10
10
9
9
10
10
8
6
6
6
7
8
8
9
9
9
9
10
10
8
6
6
6
6
7
8
11
11
11
8
9
9
10
10
10
10
10
9
9
811
12
9
9
10
10
9
9
8
10
11
9
7
11
11
12
12
12
12
12
12
12
12
12
12

Page 5

BMC Bioinformatics 2005, 6:205http://www.biomedcentral.com/1471-2105/6/205
Page 5 of 15
(page number not for citation purposes)
Other models
In the previous classification problems, we only included
linear terms of gene expression levels in a model. We also
considered expanded models including squared terms of
each gene's expression levels. The motivation is to possi-
bly improve model fitting, for instance, to avoid masking
in linear models [26]. In this way, the number of variables
in the new data is doubled (the original variables plus
their squared terms) and we have 44566 variables. We
repeated all the previous classification procedures and
found that the classification performance did not improve
(results not shown).
Classification with Minnesota data: pair-wise approach
We repeated the three-class classification with PPLS via
the pair-wise approach. The results are included in Table 4.
We assessed the classification of PPLS by including all 30
patients and with 23 male only. By comparing Table 4 to
Table 1 – 2, we can see that the PPLS via one-against-others
approach gives much smaller errors. This suggests that for
this specific problem, the one-against-others approach is
probably better. Again, we see the classification with male
patients gives much larger LOOCV misclassification error
rates.
Classification with PGA data: one-against-others
approach
Table 5 reports the LOOCV misclassification errors of the
five methods for the PGA data. Four kinds of errors are
reported for each method: three one-against-others two-
class LOOCV misclassification errors (Normal vs. others,
Ischemic vs. others, and Idiopathic vs. others) and one
three-class LOOCV misclassification error.
Based on Table 5, we can see that the classification per-
formances of PPLS, PLS, SC, LASSO and RF are quite sta-
ble with different numbers of top ranked genes one uses.
In the two-class classification problem of normal vs. oth-
ers, the five classification methods almost perform per-
fectly, where PPLS, PLS and RF have 0 errors in all
circumstances, SC has 0 errors in all situations except for
3 cases and LASSO has 1 error in one case and 0 errors in
all other cases. In the two-class classification problems of
ischemic vs. others and idiopathic vs. others, PPLS, PLS,
SC, LASSO and RF yielded 1–5 errors (mostly with 1–3
errors). That the problem of distinguishing normal from
the others is the easiest confirms that the normal class is
more separable from the other two classes. As for the
three-class classification, the errors range from 1 to 2 and
it is almost perfect.
Table 4: LOOCV errors for three-class classification: PPLS with
pair-wise approach.
#of top genes Minnesota data
All patientsMale
50
100
200
400
800
1600
3200
6400
9600
12800
16000
19200
22283
18
13
16
17
18
17
16
15
17
17
19
14
19
16
18
18
17
14
16
14
15
15
14
14
16
16
Table 5: LOOCV errors for three-class classification with PGA data: all patients.
# of top
genes
Normal vs other Isch vs other Idio vs otherOverall
PPLS PLSscLSO RFPPLSPLSSC LSORFPPLSPLSSC LSORFPPLSPLS SCLSO RF
50
100
200
400
800
1600
3200
6400
9600
12800
16000
19200
22277
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
2
2
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
4
5
3
3
2
3
3
2
3
2
2
2
1
2
1
1
1
1
2
2
1
1
1
1
1
2
3
2
2
2
2
2
2
2
2
2
2
2
2
1
1
4
1
1
2
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
5
3
2
2
2
2
2
2
3
3
3
2
2
1
1
2
1
1
2
3
2
2
2
2
1
1
5
3
2
2
2
2
3
3
2
2
2
2
2
1
1
2
2
2
2
2
2
2
2
2
2
2
2
1
1
2
2
2
2
2
2
2
2
2
2
2
1
1
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
2
2
1
1
1
1
1
1
1
1
1
1
1
1
2
1
1
1
1
1
1
1
2
2
1
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1