Page 1

BioMed Central

Page 1 of 9

(page number not for citation purposes)

BMC Bioinformatics

Open Access

Correspondence

A response to information criterion-based clustering with

order-restricted candidate profiles in short time-course microarray

experiments

Shyamal D Peddada*1, David M Umbach1 and Shawn F Harris2

Address: 1Biostatistics Branch, National Institute of Environmental Health Sciences Research Triangle Park, NC 27709, USA and 2SRA International

Inc, Durham, NC 27713,USA

Email: Shyamal D Peddada* - peddada@niehs.nih.gov; David M Umbach - umbach@niehs.nih.gov; Shawn F Harris - Shawn_Harris@sra.com

* Corresponding author

Abstract

Background: For gene expression data obtained from a time-course microarray experiment, Liu

et al. [1] developed a new algorithm for clustering genes with similar expression profiles over time.

Performance of their proposal was compared with three other methods including the order-

restricted inference based methodology of Peddada et al. [2,3]. In this note we point out several

inaccuracies in Liu et al. [1] and conclude that the order-restricted inference based methodology

of Peddada et al. (programmed in the software ORIOGEN) indeed operates at the desired nominal

Type 1 error level, an important feature of a statistical decision rule, while being computationally

substantially faster than indicated by Liu et al. [1].

Results: Application of ORIOGEN to the well-known breast cancer cell line data of Lobenhofer

et al. [4] revealed that ORIOGEN software took only 21 minutes to run (using 100,000 bootstraps

with p = 0.0025), substantially faster than the 72 hours found by Liu et al. [1] using Matlab. Also,

based on a data simulated according to the model and parameters of simulation 1 (σ2 = 1, M = 5)

in [1] we found that ORIOGEN took less than 30 seconds to run in stark contrast to Liu et al. who

reported that their implementation of the same algorithm in R took 2979.29 seconds.

Furthermore, for the simulation studies reported in [1], unlike the claims made by Liu et al. [1],

ORIOGEN always maintained the desired false positive rate. According to Figure three in Liu et al.

[1] their algorithm had a false positive rate ranging approximately from 0.20 to 0.70 for the

scenarios that they simulated.

Conclusions: Our comparisons of run times indicate that the implementations of ORIOGEN's

algorithm in Matlab and R by Liu et al. [1] is inefficient compared to the publicly available JAVA

implementation. Our results on the false positive rate of ORIOGEN suggest some error in Figure

three of Liu et al. [1], perhaps due to a programming error.

Background

A short-series time-course microarray experiment induces

a natural constraint on the mean expression of a gene or a

probe over time. Thus one may expect a systematic pattern

to the mean expression of a gene as long as the time points

are not too far apart to lose the biological relevance of a

time-course experiment. For example, for some genes the

mean expression may monotonically increase (or

Published: 22 December 2009

BMC Bioinformatics 2009, 10:438 doi:10.1186/1471-2105-10-438

Received: 6 August 2009

Accepted: 22 December 2009

This article is available from: http://www.biomedcentral.com/1471-2105/10/438

© 2009 Peddada et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0),

which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Page 2

BMC Bioinformatics 2009, 10:438 http://www.biomedcentral.com/1471-2105/10/438

Page 2 of 9

(page number not for citation purposes)

decrease) over time, whereas some others may display an

(inverted) umbrella shaped pattern, etc. Although, one

may consider using a parametric model to describe the

pattern of expression across time points, a simple nonpar-

ametric approach can be used to express the pattern of

expression across time using mathematical inequalities

(known as order restrictions). This strategy was first

exploited in [2] and subsequently software called ORIO-

GEN (Ordered Restricted Inference for Ordered Gene

ExpressioN) was developed ([3]). It has been publicly

available at ORIOGEN - Order Restricted Inference for

Ordered Gene ExpressioN http://www.niehs.nih.gov/

research/resources/software/oriogen/index.cfm

2005, and an upgraded version (2.2.1) has been available

since February 1, 2007.

since

Liu et al. [1] introduce an interesting alternative method-

ology for clustering genes using order-restricted inference

methodology. Their strategy differs from that of [2,3] in

that they avoid the bootstrap computation of p-value for

identifying significant genes. Instead, they use an infor-

mation theoretic model selection criterion to assign genes

to temporal patterns (clusters). Once genes are clustered,

they evaluate the reliability of each cluster using a boot-

strap algorithm along the lines of Kerr and Churchill [5].

Liu et al. [1] make several statements about our method-

ology [2,3] that are erroneous and require clarification.

The methodology of Peddada et al. [2], implemented in

ORIOGEN [3], controls the Type I error rate (false positive

rate) at any pre-specified level. The methodology of Liu et

al. [1], implemented in ORICC, does not control Type I

error rate explicitly. It thereby risks attributing differential

expression through time to an excessive proportion of

genes whose expression does not truly change. Given the

large multiple testing problem inherent in microarray

analyses (e.g., the Human Affymetrix chip has ~45,000

probes), use of procedures that control Type I error, or a

related quantity like the false discovery rate (FDR), is cru-

cial.

Results

Computation time

Throughout their paper, Liu et al. [1] claim that Peddada

et al. methodology is excessively computationally inten-

sive. The many bootstrap samples needed for precise esti-

mation of small p values is indeed computationally

demanding; but the run times mentioned by Liu et al. for

our methodology seemed extremely long in our experi-

ence. On page 2 they assert that their implementation of

"Peddada's method" required 72 hours to analyze the

breast cancer cell-line data of [4]. They do not state how

many bootstrap samples they used and what p-value they

used in their analysis. Although they cite our publicly

available software ORIOGEN, they appear to have written

their own code rather than using ORIOGEN. They state

that they implemented our methodology (calling it "Ped-

dada's methodology") in Matlab (page 2) and again in R

(page 5). Their performance estimates were not based on

the ORIOGEN software that is freely available from our

website. We were surprised that the authors made the

effort to re-code our algorithm in two different languages

when our software is available without charge.

To examine their claims about run times, we imple-

mented ORIOGEN on the breast cancer cell-line data of

[4], using 100,000 bootstraps, with a p-value of 0.0025.

We found that ORIOGEN took only 21 minutes to run,

not 72 hours as stated in [1] for their Matlab implementa-

tion. We also used ORIOGEN to analyze data simulated

exactly as described on pages 5-6 of [1]. The authors claim

on page 7 that their implementation of "Peddada's

method" in R took 2979.29 seconds to analyze the simu-

lated data when σ2 = 1 and M = 5; however, for the same

simulation conditions, ORIOGEN took about 30 seconds

to run. For these analyses, we employed a Dell desktop PC

with an Intel Xeon CPU 2.33 GHz with 3.00 GB of RAM.

ORIOGEN was developed by two professional computer

programmers who tested it repeatedly before making it

public. These exceptionally large discrepancies in run

times between ORIOGEN and Liu et al.'s implementation

of its algorithm lead us to conclude that either Liu et al.

either misinterpreted details of our methodology or their

coding of it is extremely inefficient.

False positive rates

In Figure three, Liu et al. [1] compare the false positive rate

of our method with theirs. They claim to have run our

method at a level of significance of 0.025 (for each simu-

lated gene) using 200 bootstraps (page 6). In their simu-

lation study they consider 200 "null" genes and 2000

"non-null" genes and define false positive rate to be the

proportion of null genes that are declared significant.

Based on their Figure three, they report that our method

can have a false positive rate as high as 0.50 at the nomi-

nal rate of 0.025. This result is incorrect. We generated the

2200 genes according to the patterns described by the

authors on page 9 which included 200 "null genes".

Exactly as in [1], we implemented ORIOGEN with a level

of significance of 0.025 and 200 bootstraps for the 6 pat-

terns of σ2 and as many replicates as in [1]. We found that

ORIOGEN always performed at the desired nominal level

of 0.025, as it was designed to do. It appears that Liu et al.

[1] misinterpreted our methodology, made some pro-

gramming errors in coding it, or miscalculated/misre-

ported the false positive rates.

Most statisticians and bioinformaticians recognize that

maintaining a pre-specified false positive rate is an impor-

tant requirement for statistical testing procedures. In fact,

scientists want to avoid reporting an excessive number of

genes as differentially expressed when they are not. Dur-

Page 3

BMC Bioinformatics 2009, 10:438 http://www.biomedcentral.com/1471-2105/10/438

Page 3 of 9

(page number not for citation purposes)

ing the past decade, much research has been devoted to

developing sound methods for controlling false discovery

rates in gene expression microarray studies. In contrast to

ORIOGEN, which maintains the nominal false positive

rate and provides estimates of q-values often used to con-

trol false discovery, ORICC, the method proposed by Liu

et al., does not control the false positive rate or the false

discovery rate at a pre-specified level. Consequently,

ORICC can sometimes have an unusually high false posi-

tive rate, as high as 0.70 according to Figure three of [1].

In the bottom right panel of Figure three of [1], the lowest

false positive rate that ORICC achieved for the simulated

data is 0.20.

Other erroneous statements

(a) Liu et al. make an incorrect assertion regarding the

universal domination property of order-restricted

maximum likelihood estimator (MLE) (page 14 of

[1]). They assert that the order-restricted MLE univer-

sally dominates the unrestricted MLE and wrongly

attribute this theoretical property to Hwang and Ped-

dada [6]. Hwang and Peddada did not prove the result

in the generality stated in [1]; they proved it only for

independently normally distributed data when the

means satisfy a monotone order. On the contrary, the

main emphasis of [6] is to demonstrate that the order-

restricted MLE may actually perform poorly under cer-

tain conditions. Hence in [6] Hwang and Peddada

introduced an alternative to the order-restricted MLE

for estimation of parameters satisfying constraints.

ORIOGEN uses this new estimation procedure instead

of the order-restricted MLE. When the order restriction

is monotonic, the two procedures coincide.

(b) Liu et al. also state incorrectly on page 5 that "Ped-

dada's method then carried out a bootstrap-based like-

lihood ratio test". Because ORIOGEN is not working

with order-restricted MLE's to begin with, there is no

explicit likelihood with which to construct a likeli-

hood-based test. Consequently, its test statistic is more

along the lines of a "Wald" type statistic and not a like-

lihood ratio. In the same sentence as stated above, on

page 5, Liu et al. suggest that Peddada et al. use the

bootstrap-likelihood ratio test to decide a gene's best

matched profile. This statement is not correct. Ped-

dada et al. used the bootstrap to select significant

genes but assigned genes to profiles using a goodness-

of-fit criterion.

A Simulation study

In their simulation study, Liu et al. [1] considered 200

"null" (or non-differentially expressed) genes and 2000

"non-null" (differentially expressed/true positive) genes.

Thus at most 10% are non-differentially expressed

whereas the overwhelming majority, about 90%, are dif-

ferentially expressed. If this were the true nature of the

data, then a biologist may want to skip any formal statis-

tical methodology and take all 2200 genes - this selection

rule will assure him/her 100% discovery of true genes at a

small price of at most 10% false discovery rate (FDR).

From our experience, it would be more realistic to expect

that most genes in a microarray study would be non-dif-

ferentially expressed.

Study design

In this simulation study we almost mimicked the simula-

tion experiment of Liu et al. [1] with the major exception

that we considered 12000 null (or non-differentially

expressed) genes and 4000 non-null (true positives)

genes. Thus, 25% are true positives and about 75% are

true nulls.

We generated our data according to the model and the

parameters used in Liu et al.[1] except that we have more

null genes than non-null genes as commonly observed in

gene expression studies:

In the above model, for a gene g, g = 1,2,...,16000, is

the observed expression of the jth replicate, j = 1,2,...,8, in

the ith treatment group, i = 1,2,...,6, and

mean expression of gth gene in the ith treatment group. We

considered 2 patterns of variance σ2(= 0.2,1.)' within each

mmmm

=

(,,,

12

…

is the true

pattern of described below.

Pattern 1: (Null) μg =(0,0,0,0,0,0)' - 6000 samples corre-

sponding to each variance pattern, and hence 12000 null

genes.

Pattern 2: (Increasing) μ = (0,0.5,1,1.5,2,2.5)' - 200 sam-

ples corresponding to each variance pattern, and hence

400 increasing genes.

Pattern 3: (Decreasing) μ = (0,-0.5,-1,-1.5,-2,-2.5)' - 200

samples corresponding to each variance pattern, and

hence 400 decreasing genes.

Pattern 4: (Umbrella Peak at 2) μ = (0,0.5,0,-0.5,-1,-1.5)'

- 200 samples corresponding to each variance pattern, and

hence 400 umbrella pattern genes.

Pattern 5: (Inverted Umbrella Min at 2) μ = (0,-

0.5,0,0.5,1,1.5)' - 200 samples corresponding to each var-

YN

ij

g

i

g

ij

g

ij

=+

mees

,~ ( ,0 ). with

2

Yij

g

mi

g

g

ggg

6

′

)

Page 4

BMC Bioinformatics 2009, 10:438 http://www.biomedcentral.com/1471-2105/10/438

Page 4 of 9

(page number not for citation purposes)

Table 1: Comparison of ORIOGEN and ORICC using a simulated data. (due to Peddada et al.)

Criteria ORIOGENORICC

Number of nulls selected

Number of true positives selected

95 3559

3957 3335

Total number of discoveries34307516

Number of discoveries with correct non-null cluster assignment2917 3322

Type I error rate

False discovery rate (FDR)

0.008

0.028

0.297

0.474

Power0.834 0.989

Proportion of discoveries with correct non-null cluster assignment0.729 0.831

Proportion of discoveries with correct non-null cluster assignment among the correctly selected non-null genes 0.875 0.840

Total error0.074 0.265

Simulation 1: The overall error rate of Peddada's method and the one-stage ORICC algorithm

Figure 1

Simulation 1: The overall error rate of Peddada's method and the one-stage ORICC algorithm. (due to Liu et al.)

Page 5

BMC Bioinformatics 2009, 10:438http://www.biomedcentral.com/1471-2105/10/438

Page 5 of 9

(page number not for citation purposes)

iance pattern, and hence 400 inverted umbrella pattern

genes.

Pattern 6: (Umbrella Peak at 3) μ = (0,0.5,1,0.5,0,-0.5)' -

200 samples corresponding to each variance pattern, and

hence 400 umbrella pattern genes.

Pattern 7: (Inverted Umbrella Min at 3) μ = (0,-0.5,-1,-

0.5,0,0.5)' - 200 samples corresponding to each variance

pattern, and hence 400 inverted umbrella pattern genes.

Pattern 8: (Umbrella Peak at 4) μ = (0,0.5,1,1.5,1,0.5)' -

200 samples corresponding to each variance pattern, and

hence 400 umbrella pattern genes.

Pattern 9: (Inverted Umbrella Min at 4) μ = (0,-0.5,-1,-

1.5,-1,-0.5)' - 200 samples corresponding to each variance

pattern, and hence 400 inverted umbrella pattern genes.

Pattern 10: (Umbrella Peak at 5) μ = (0,0.5,1,1.5,2,1.5)' -

200 samples corresponding to each variance pattern, and

hence 400 umbrella pattern genes.

Pattern 11: (Inverted Umbrella Min at 5) μ = (0,-0.5,-1,-

1.5,-2,-1.5)' - 200 samples corresponding to each variance

pattern, and hence 400 inverted umbrella pattern genes.

Thus the total number of genes considered in this simula-

tion study is 16000 consisting of 12000 null and 4000

non-null.

Results

We applied ORICC, by downloading the software from

the website provided in [1], and ORIOGEN 2.2.1. We

applied ORIOGEN using a p-value cut off (or level of sig-

nificance) of 0.01 and the reclassification p-value of 0.90

for patterns. Since we are using a cut-off of 0.01, it is suf-

Simulation 1: The false positive rate of Peddada's method and the one-stage ORICC algorithm

Figure 2

Simulation 1: The false positive rate of Peddada's method and the one-stage ORICC algorithm. (due to Liu et al.)