# The impact of quantile and rank normalization procedures on the testing power of gene differential expression analysis.

**ABSTRACT** Background Quantile and rank normalizations are two widely used pre-processingtechniques designed to remove technological noise presented ingenomic data. Subsequent statistical analysis such as genedifferential expression analysis is usually based on normalizedexpressions. In this study, we find that these normalizationprocedures can have a profound impact on differential expressionanalysis, especially in terms of testing power.Results We conduct theoretical derivations to show that the testing power ofdifferential expression analysis based on quantile or ranknormalized gene expressions can never reach 100% with fixed samplesize no matter how strong the gene differentiation effects are.We perform extensive simulation analyses and find theresults corroborate theoretical predictions.Conclusions Our finding may explain why genes with well documentedstrong differentiation are not always detected in microarrayanalysis. It provides new insights in microarray experimental design and will helppractitioners in selecting proper normalization procedures.

**0**Bookmarks

**·**

**166**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**Normalization procedures are widely used in high-throughput genomic data analyses to remove various technological noise and variations. They are known to have profound impact to the subsequent gene differential expression analysis. Although there has been some research in evaluating different normalization procedures, few attempts have been made to systematically evaluate the gene detection performances of normalization procedures from the bias-variance trade-off point of view, especially with strong gene differentiation effects and large sample size. In this paper, we conduct a thorough study to evaluate the effects of normalization procedures combined with several commonly used statistical tests and MTPs under different configurations of effect size and sample size. We conduct theoretical evaluation based on a random effect model, as well as simulation and biological data analyses to verify the results. Based on our findings, we provide some practical guidance for selecting a suitable normalization procedure under different scenarios.PLoS ONE 06/2014; 9(6):e99380. · 3.53 Impact Factor - SourceAvailable from: Johanna P Daily
##### Article: Plasmodium falciparum gene expression measured directly from tissue during human infection.

Daria Van Tyne, Yan Tan, Johanna P Daily, Steve Kamiza, Karl Seydel, Terrie Taylor, Jill P Mesirov, Dyann F Wirth, Danny A Milner[Show abstract] [Hide abstract]

**ABSTRACT:**During the latter half of the natural 48-h intraerythrocytic life cycle of human Plasmodium falciparum infection, parasites sequester deep in endothelium of tissues, away from the spleen and inaccessible to peripheral blood. These late-stage parasites may cause tissue damage and likely contribute to clinical disease, and a more complete understanding of their biology is needed. Because these life cycle stages are not easily sampled due to deep tissue sequestration, measuring in vivo gene expression of parasites in the trophozoite and schizont stages has been a challenge. We developed a custom nCounter® gene expression platform and used this platform to measure malaria parasite gene expression profiles in vitro and in vivo. We also used imputation to generate global transcriptional profiles and assessed differential gene expression between parasites growing in vitro and those recovered from malaria-infected patient tissues collected at autopsy. We demonstrate, for the first time, global transcriptional expression profiles from in vivo malaria parasites sequestered in human tissues. We found that parasite physiology can be correlated with in vitro data from an existing life cycle data set, and that parasites in sequestered tissues show an expected schizont-like transcriptional profile, which is conserved across tissues from the same patient. Imputation based on 60 landmark genes generated global transcriptional profiles that were highly correlated with genome-wide expression patterns from the same samples measured by microarray. Finally, differential expression revealed a limited set of in vivo upregulated transcripts, which may indicate unique parasite genes involved in human clinical infections. Our study highlights the utility of a custom nCounter® P. falciparum probe set, validation of imputation within Plasmodium species, and documentation of in vivo schizont-stage expression patterns from human tissues.Genome Medicine 11/2014; 6(11):110. · 4.94 Impact Factor - SourceAvailable from: Chandrima DasKirti K. Kulkarni, Kiran Gopinath Bankar, Rohit Nandan Shukla, Chandrima Das, Amrita Banerjee, Dipak Dasgupta, Madavan Vasudevan[Show abstract] [Hide abstract]

**ABSTRACT:**The role of Mithramycin as an anticancer drug has been well studied. Sarcoma is a type of cancer arising from cells of mesenchymal origin. Though incidence of sarcoma is not of significant percentage, it becomes vital to understand the role of Mithramycin in controlling tumor progression of sarcoma. In this article, we have analyzed the global gene expression profile changes induced by Mithramycin in two different sarcoma lines from whole genome gene expression profiling microarray data. We have found that the primary mode of action of Mithramycin is by global repression of key cellular processes and gene families like phosphoproteins, kinases, alternative splicing, regulation of transcription, DNA binding, regulation of histone acetylation, negative regulation of gene expression, chromosome organization or chromatin assembly and cytoskeleton.Genomics Data. 11/2014;

Page 1

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

RESEARCH ARTICLEOpen Access

The impact of quantile and rank normalization

procedures on the testing power of gene

differential expression analysis

Xing Qiu, Hulin Wu and Rui Hu*

Abstract

Background: Quantile and rank normalizations are two widely used pre-processing techniques designed to remove

technological noise presented in genomic data. Subsequent statistical analysis such as gene differential expression

analysis is usually based on normalized expressions. In this study, we find that these normalization procedures can

have a profound impact on differential expression analysis, especially in terms of testing power.

Results: We conduct theoretical derivations to show that the testing power of differential expression analysis based

on quantile or rank normalized gene expressions can never reach 100% with fixed sample size no matter how strong

the gene differentiation effects are. We perform extensive simulation analyses and find the results corroborate

theoretical predictions.

Conclusions: Our finding may explain why genes with well documented strong differentiation are not always

detected in microarray analysis. It provides new insights in microarray experimental design and will help practitioners

in selecting proper normalization procedures.

Background

Microarray technology has been widely adopted in many

genomic related studies in the past decade. Despite its

popularity, it is well known that various technical noises

exist in microarray experiments [1,2] due to the limi-

tation of technology. As a remedy, many normalization

procedures have been proposed to remove these system-

atic noises, thus improving the detection of differentially

expressed genes. Some efforts have been made to evaluate

different normalization procedures [3-6]. Interested read-

ers are referred to [7,8] for background and more detailed

reviews of normalization procedures.

Quantile normalization is perhaps the most widely

adopted method for analyzing microarray data generated

byAffymetrixGeneChipplatform.Motivatedbyquantile-

quantile plot, it makes the empirical distribution of gene

expressions pooled from each array to be the same [3].

It is the default option of BioConductor [9], which is a

*Correspondence: huruizg@hotmail.com

Department of Biostatistics and Computational Biology, University of

Rochester, 601 Elmwood Avenue, Box 630, Rochester, New York 14642, USA

very popular open source software for analyzing microar-

ray data implemented in R [10], the de facto standard

statistical computing language in the statistical research

community. This algorithm is also used for normalizing

Affymetrix exon arrays [11,12], Illumina BeadChip arrays

[13-15], Illumina transcriptome sequencing (mRNA-Seq)

data [16], Illumina Infinium whole genome genotyping

(WGG) arrays [17], and Solexa/Illumina deep sequenc-

ing technology [18], etc. In addition, several other popular

normalization procedures are variants of quantile nor-

malization, such as the enhanced quantile normalization

[19] and subset quantile normalization [20] designed for

microarrays, and the conditional quantile normalization

[21] designed primarily for normalizing RNA-seq data.

Rank normalization is an alternative to quantile nor-

malization. It replaces each observation by its frac-

tional rank (the rank divided by the total number of

genes) within array [22,23]. This normalization pro-

cedure achieves robustness to non-additive noise at

the expense of losing some parametric information

of expressions.

After normalization, a pertinent statistical test such as

Student’s t-test [24] is applied to these normalized gene

© 2013 Qiu et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative

Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and

reproduction in any medium, provided the original work is properly cited.

Page 2

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 2 of 10

expression levels. The resulting p-values are adjusted by

a multiple testing procedure (MTP) in order to con-

trol certain quantity of per-family Type I error, such as

family-wise error rate (FWER) [25-28] and false discov-

ery rate (FDR) [29]. Differentially expressed genes are

identified based on a pre-specified threshold of adjusted

p-values. More detailed introduction of statistical meth-

ods for detecting differentially expressed genes can be

found in [30-33].

Without compromising the control of type I error,

better testing power can be achieved by either increas-

ing sample size or improving the strength of gene

differentiation effect (fold changes between different phe-

notypes). Sometimes large expected differential effects

based on biological considerations are invoked as a rea-

son to justify a microarray study with very small sample

sizes.

In this study, we find that one cannot “trade” dif-

ferentiation effects with sample size. When the sam-

ple size is small, the statistical power for a gene

differentiation analysis will not reach 100% even when

the effect size approaches to infinity. This counter-

intuitive phenomenon is due to the nature of the

normalization procedures, which alters both sample

mean difference and pooled sample standard devia-

tion of the normalized expressions. As a result, they

both grow at most linearly as functions of effect size

and their effects cancel out. Our findings provide new

insights into microarray experimental design which may

help practitioners in selecting appropriate normalization

procedures.

Methods

Notations and biological data

Notations

We assume that all expression levels are log-transformed.

For convenience, the words “gene” and “gene expression”

are used interchangeably to refer to these log-transformed

random variables. These genes are indexed by i

1,2,...,m, where m is the total number of genes.

Let c = A,B be two different phenotypic groups. For

simplicity we assume that the number of arrays in both

groups are the same and denoted by n. Without loss of

generality, phenotypic group A is set to represent the phe-

notype of interest (usually the disease or the treatment

group) and group B the normal phenotype. So up (down)

regulation of a gene refers to its over (under) expression

in group A. We denote by yc

of the ith gene recorded on the jth array sampled from the

cth phenotypic group. The normalized counterpart of yc

is written as y∗c

The mean and standard deviation of yc

by E

ij

=

ijthe observed expression level

ij

ij.

ijare denoted

ic, respectively. Their

?

yc

?

= μc

iand var(yc

ij) = σ2

normalized sample counterparts are denoted by ¯ y∗c

1

n

tively.

In practice, the true level of gene differentiation is not

a constant. It depends on the biological settings. The

variance of gene expressions is nor constant either — it

depends on the accuracy of measuring instruments and

the homogeneity of biological subjects, just to name a

few factors. In terms of statistical power, the decrease of

gene expression variance is equivalent to the increase of

mean difference. For simplicity, we consider gene expres-

sion variance to be fixed and define the effect size, our

analysis tuning parameter, to be the expected mean differ-

ence of the ith gene expression between two phenotypes

ei:= μA

We divide genes into three sets:

i·

=

?n

k=1y∗c

ikand?ˆ σ∗c

i

?2=

1

n−1

?n

j=1(y∗c

ij− ¯ y∗c

i·)2, respec-

i− μB

i.

• G0, the set of non-differentially expressed genes

(abbreviated as NDEGs). For all i ∈ G0,

ei:= μA

• G+

ei> 0.

• G−

ei< 0.

i− μB

i= 0.

1, the set of up-regulated genes. For all i ∈ G+

1, the set of down-regulated genes. For all i ∈ G−

1,

1,

The set of differentially expressed genes (abbreviated

as DEGs) is the union of both up-regulated and down-

regulated genes, which is denoted by G1= G+

write the size of these gene sets by m0= |G0|, m+

m−

and m0+ m1= m.

Biologicaldata

The biological dataset used in this study is the childhood

leukemia dataset from the St. Jude Children’s Research

Hospital database [34]. We select three groups of data:

88patients(arrays)withhyperdiploidacutelymphoblastic

leukemia (HYPERDIP), 79 patients (arrays) with a spe-

cial translocation type of acute lymphoblastic leukemia

(TEL) and 45 patients (arrays) with a T lineage leukemia

(TALL). Each patient is represented by an array report-

ing the logarithm (base 2) of expression level on the set of

9005 genes.

1∪ G−

1= |G+

1+ m−

1. We

1|,

11= |G−

1|, and m1= |G1|. Apparently m1= m+

Analytic analysis of the impact of normalization

procedures on differential expression analysis

In this section, we evaluate the impact of quantile and

rank normalization on t-test. We are especially interested

in studying the asymptotic property of the t-statistic as

the effect size of differentiation approaches infinity while

other parameters such as n and σ2

evidences in Section “Results and discussion” show that

our findings are also valid for other statistical tests such as

Wilcoxon rank-sum test and permutation based test.

iare fixed. Empirical

Page 3

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 3 of 10

To simplify theoretical derivation, we assume that the

mean expression levels in the normal phenotype (group

B) are zeros (μB

μB

many hypothesis testing procedures such as t-test and

Wilcoxon rank-sum test are invariant under shift trans-

formation, so the mean difference between two groups,

ei, is much more important than the normal level of gene

expressions. For simplicity, we also assume that the effect

size is a constant e+> 0 for all up-regulated and e−< 0

for all down-regulated genes. In summary,

⎧

⎪⎪⎩

Therefore, the expected group differences of non-

normalized gene expression data are

⎧

⎩

We must point out that all these assumptions are

made only for the simplification of the theoretical deriva-

tions. Our findings essentially do not depend on these

assumptions. This has been confirmed in our biologi-

cal simulation study in Section “Results and discussion”

(SIMU-BIO).

For the ith normalized gene expression, its t-statistic is

defined as

?n

??ˆ σ∗A

standard deviation.

The testing power of a two-sided t-test is determined

by the absolute value of t-statistic. Based on Equation (3),

it is clear that the testing power converges to 100% when

n approaches infinity. For a fixed n (which also implies a

fixed number of degrees of freedom), the testing power

is determined by the absolute sample mean difference,

|¯ y∗A

we study the asymptoticproperties ofthese two quantities

for quantile and rank normalized expressions separately.

i= 0). This assumption implies that μA

i+ ei = ei. This simplification is reasonable because

i=

E

?

yc

ij

?

=

⎪⎪⎨

e+c = A, i ∈ G+

e−c = A, i ∈ G−

0c = A, i ∈ G0,

0c = B,

1,

1,

(1)

E

?

yA

i·− yB

i·

?

=

⎨

e+i ∈ G+

e−i ∈ G−

0i ∈ G0.

1,

1,

(2)

t∗

i=

2·¯ y∗A

i·− ¯ y∗B

ˆ σ∗

i·

i

, (3)

where ˆ σ∗

i

=

i

?2+?ˆ σ∗B

i

?2

2

is called the pooled sample

i·− ¯ y∗B

i·|, and the pooled sample variance,?ˆ σ∗

i

?2. Below

Quantilenormalization

With quantile normalization (QUANT), a reference array

of empirical quantiles, denoted as q = (q1,q2,...,qm),

is first computed by taking the average across all ordered

arrays. Let yc

ordered gene expression observations in the jth array (j =

(1),j? yc

(2),j? ··· ? yc

(m),jdenote the

1,2,...,n) of the cth (c = A,B) group, the rth (r =

1,2,...,m) element of this reference array is

qr=

1

2n

?n

k=1

?

yA

(r),k+

n

?

l=1

yB

(r),l

?

. (4)

The original expressions are replaced by the entries of

the reference array with the same rank. Denote rc

rankofyc

gene expressions are

ijas the

ijinthearraytowhichitbelongs.Thenormalized

y∗c

ij= qrc

ij=

1

2n

?n

k=1

?

yA

(rc

ij),k+

n

?

l=1

yB

(rc

ij),l

?

. (5)

We refer the reader to [3] for more details.

In group A, over(under)-expressed genes tend to have

high (low) ranks in each array. When the effect size is

small, the ranks of DEGs in group A are mixed with those

of NDEGs and the downstream testing power will be low.

When the effect size is large, the DEGs in group A effec-

tively take up all the top and bottom ranks, so the NDEGs

in group A can only compete for ranks between m−

and m − m+

almost always take the top m+

and the m−

the bottom m−

that the Student’s t-statistic of quantile normalized gene

expressions follows a mixture distribution in which the

doubly noncentral part converges to a distribution with

finite all order moments instead of infinity when the true

effect size becomes large.

We first investigate the asymptotic properties of sample

meandifference ¯ y∗A

malization ranks gene expressions first and then replace

them by a reference quantile computed from all arrays.

For an up-regulated DEG (i ∈ G+

among the top m+

(rB

In this case, the expectation of sample mean difference is

zero; otherwise it grows linearly as a function of e+. More

specifically, by using conditional expectation, we obtain

that for i ∈ G+

1+ 1

1. We assume that the m+

1up-regulated genes

1ranks with equal chances

1down-regulated genes almost always take

1ranks with equal chances. We will show

i·−¯ y∗B

i·.Roughlyspeaking,quantilenor-

1), its rank can be

1genes for all arrays in the normal group

1, j = 1,2,...,n) with probability?m+

ij> m − m+

1

m

?n.

1,

E(¯ y∗A

i·− ¯ y∗B

⎧

⎪⎪⎪⎩

i·|rB

i1,··· ,rB

in)

∝

⎪⎪⎪⎨

O(1),with probability

?

m+

m

1

?n

?

,

O(e+,e−), with probability 1 −

m+

m

1

?n

.

(6)

Similarly for down-regulated DEGs (i ∈ G−

1),

Page 4

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 4 of 10

E(¯ y∗A

i·− ¯ y∗B

⎧

⎪⎪⎪⎩

i·|rB

i1,··· ,rB

in)

∝

⎪⎪⎪⎨

O(1),with probability

?

m−

m

1

?n

?

,

O(e+,e−), with probability1 −

m−

m

1

?n

.

(7)

Detailed derivations can be found in Section 3 in the

Additional file 1.

Similarly, ˆ σ∗·

can either grow linearly as a function of e+and e−or

(with positive probability) stay as a constant. Heuristically

speaking, ˆ σ∗·

expressions are all in the top group (r·

group (m−

m−

sizes so they are canceled out. If the ranks are from dif-

ferent groups, some will have high expressions and some

are low, the standard deviation will be “stretched out”.

Since we assume up-regulated (down-regulated) genes in

group A almost always take up the top (bottom) ranks,

(ˆ σ∗A

For group B we have

??

⎧

O((e+)2,(e−)2), otherwise with probability

1 −?m0

i, the pooled sample standard deviation,

idoes not depend on e+or e−if the ranks of

ij> m−m+

1), middle

1< r·

ij? m − m+

1), or the bottom group (r·

ij?

1) because all expression levels have the same effect

i

)2∝?n

j=1(y∗A

ij− ¯ y∗A

i·)2does not depend on e+or e−.

E

ˆ σ∗B

O(1), allrB

i

?2|rB

i1,··· ,rB

i·∈ top m+

with probability?m0

in

?

∝

⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

1,or middlem0,or bottomm−

?n+

?n−

1

m

?

?

m+

m

1

?n

?n

+

?

?

m−

m

1

?n

?n

(8)

m

m+

m

1

−

m−

m

1

.

More detailed derivations can be found in Section 3 in the

Additional file 1.

According to Equations (6), (7) and (8), the sample

mean difference and pooled sample standard deviation

both grow at most linearly as functions of e+(e−). As a

result, the (absolute values of) t-statistics t∗

rB

mately have the following mixture of central, noncentral

and doubly noncentral forms:

iin (3) (given

1) approxi-

i1,··· ,rB

in) for up-regulated DEGs (i ∈ G+

t∗

i|rB

i1,··· ,rB

⎧

O(e+,e−)

O(1)

in

∼

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

O(1)

O(1), allrB

i·∈topm+

, allrB

with probability?m0

O(e+,e−),otherwise with probability

1 −?m0

1with probability

i·∈middlem0or bottomm−

?

m+

m

1

?n

?n

,

1

m

?n+

?n

?

m−

m

1

,

O(e+,e−)

m

?n−

?

m+

m

1

−

?

m−

m

1

?n

.

(9)

Similarly, the t-statistics t∗

G−

ifor down-regulated DEGs (i ∈

1) approximately have the following mixture forms:

t∗

in

⎧

O(e+,e−)

O(1)

with probability?m0

O(e+,e−),otherwise with probability

1 −?m0

i|rB

i1,··· ,rB

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

O(e+,e−)

∼

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

O(1)

O(1), allrB

i·∈bottomm−

, allrB

1with probability

?

?n

m−

m

1

?n

,

i·∈middlem0or topm+

1

?

m

?n+

?n

m+

m

1

,

m

?n−

?

m+

m

1

−

?

m−

m

1

?n

.

(10)

To see this mixture under the normality assumption,

we assume that all observed gene expressions yc

low a normal distribution. Then, the normalized gene

expressions y∗c

tion (See Section 2 in the Additional file 1). According to

Equation (9), the t-statistics t∗

G+

tral and doubly noncentral t-distributions with a density

function

?

mm

?

m

ijfol-

ijapproximately follow a normal distribu-

ifor up-regulated DEGs (i ∈

1) approximately follow a mixture of central, noncen-

ft∗

i≈

m+

1

?n

ft+

??m0

?n

?n

m+

m

+

?

m−

m

1

?n?

?

fT(γ)

?n?

+

1 −

?m0

−

?

1

?n

−

m−

m

1

fT(γ,λ).

Here ft, fT(γ) and fT(γ,λ) are the density functions of

central, noncentral and doubly noncentral t-distributions,

respectively, with ν = 2n − 2 degrees of freedom. γ ∝

O(e+,e−) is the numerator noncentrality parameter and

λ ∝ O((e+)2,(e−)2) is the denominator noncentrality

parameter (from noncentral χ2) [35]. Similarly, accord-

ing to Equation (10), the t-statistics t∗

DEGs (i ∈ G−

density function

?

mm

?

m

ifor down-regulated

1) approximately follow a distribution with a

ft∗

i≈

m−

1

?n

ft+

??m0

?n

?n

m+

m

+

?

m+

m

1

?n?

?

fT(γ)

?n?

+

1 −

?m0

−

?

1

?n

−

m−

m

1

fT(γ,λ).

In microarray analysis it is reasonable to assume m1 ?

m, i.e., the proportion of DEGs is small (m−

m+

is negligible. Empirical density functions of t∗

normalizedDEGexpressionswithdifferenteffectsizesare

shown in Figures 1 (b) and (d). For effect sizes 2 and 4,

the two peaks in the center represent the doubly noncen-

tralt-distributionpartT(γ,λ)andthetwopeakstothefar

left and right sides represent the noncentral t-distribution

1? m and

1? m).Sothecentralt-distributionpartinthemixture

ifor quantile

Page 5

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 5 of 10

Density

0

0.05

0.1

0.4

0

0.05

0.1

0.4

0

0.05

0.1

0.4

0

0.05

0.1

0.4

e=0

e=2

(a) Without normalization

Density

e=0

e=2

(b) With quantile normalization

Density

e=0

e=4

(c) Without normalization

Density

−60−40 −200 204060−60

(d) With quantile normalization

−40−200204060

−60−40 −20020 4060−60−40 −200 20 40 60

e=0

e=4

Figure 1 Empirical density estimates of the t-statistics before and after quantile normalization. Empirical density estimates of the t-statistics

before and after quantile normalization. Gene expression are simulated by using normal random numbers with standard deviation 0.35 and

gene-gene correlation 0.9. Total number of genes is m = 1000. Total numbers of truly differentially expressed genes are m+

genes and m−

on 200 repetitions.

1= 60 for up-regulated

1= 40 for down-regulated genes. The sample size is n = 10 and the true effect size is e+= −e−= e = {0,2,4}. Estimates are based

partT(γ).Thedoublynoncentralt-distributionconverges

to a distribution with finite all order moments when e+

(e−) approaches infinity. Figure 1 shows the convergence

of t∗

and the medians of the t statistic absolute values for DEGs

with and without quantile normalization are plotted in

Figure 2. Clearly, the median for data without normaliza-

tiongrowslinearlywhilethemedianfordatawithquantile

normalization is upper-bounded by a fixed constant when

effect size becomes large. Therefore, the testing power

associated with a two-sided t-test cannot reach 100%. The

derivation of this convergence can be found in Section

4 in the Additional file 1. This result suggests that even

if certain genes are known to have dramatically different

expression levels for different phenotypes, a typical differ-

ential expression analysis based on quantile normalized

expressions may not be able to detect them. In this case,

combining the results obtained from differential expres-

sion analysis without normalization may provide new

insight to the underlying biology.

i. Furthermore, we let e+and −e−vary from 0 to 3.6

Empirical evidences in Section “Results and discussion”

also show that the statistical power converges to a fixed

numberstrictlylessthan1.0;andthisconvergenceisinde-

pendent of the hypothesis testing methods and MTPs

being applied. Heuristically speaking, QUANT “borrows”

information from both NDEGs and DEGs to reduce data

variation, and as a result the normalized expressions

are complex mixture of both NDEGs and DEGs with

possibly very high true group differences. Consequently,

the variances of normalized DEGs are asymptotically

dominated by the differences between the NDEGs and

DEGs and become increasing functions of effect sizes.

Asymptotically, the increased variances cancel out the

contributions of the increased effect sizes to the testing

power.

Ranknormalization

With rank normalization (RANK), we replace each entry

in one array by its position (rank) in the ordered array

counted from the smallest value divided by the total

Page 6

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 6 of 10

0.00.51.0 1.5

Effect size

2.02.53.0 3.5

0

5

10

15

20

Median of t statistic absolute values

Without normalization

Quantile normalization

Figure2 Mediansofthet-statisticabsolutevalues. Medians of the

absolute values of the t-statistics for data with and without quantile

normalization. Gene expression are simulated by using normal

random numbers with standard deviation 0.35 and gene-gene

correlation 0.9. Total number of genes is m = 1000. Total numbers of

truly differentially expressed genes are m+

genes and m−

n = 10 and the true effect size is e+= −e−= e = {0,0.2,0.4,

...,3.6}. Estimates are based on 200 repetitions.

1= 60 for up-regulated

1= 40 for down-regulated genes. The sample size is

number of genes. Denote rc

to which it belongs, the normalized gene expressions are

ijas the rank of yc

ijin the array

y∗c

ij=

rc

m.

ij

(11)

This method was proposed by [22] and discussed further

in [23].

Compared with QUANT, RANK goes even further in

the nonparametric direction. It removes the noise by only

preserving the ordering of observations. We know m is

usuallyverylargeinatypicalmicroarraystudy.Iftheeffect

size is large such that the over-expressed genes always

take up the top m+

always take up the bottom m−

approximately has the following uniform distribution:

1ranks and the under-expressed genes

1ranks in group A, y∗c

ij

y∗c

ij∼

⎧

⎪⎪⎪⎪⎩

⎪⎪⎪⎪⎨

U(1 −m+

U(0,m−

U(m−

U(0,1),

1

m,1),

m),

m,1 −m+

c = A, i ∈ G+

c = A, i ∈ G−

m), c = A, i ∈ G0,

c = B.

1,

1,

1

11

(12)

Here for simplicity, again we assume that the genes take

the specified ranks with equal chances within each group.

Therefore, the normalized gene expressions no longer

depend on the effect size. The expected group differences

for rank normalized genes are

⎧

⎩

It is easy to check that the pooled standard deviation is

also independent of the effect size. As a result, the testing

power with rank normalization converges to a constant

strictly less than 1.0 as the effect size increases. More

details can be found in Section 5 in the Additional file 1.

E

?

y∗A

i·− y∗B

i·

?

≈

⎨

1

2−m+

m−

1

2m−1

1

2mi ∈ G+

2i ∈ G−

1,

1.

(13)

Simulation studies

Extensive simulations are conducted to verify above theo-

reticalpredictions.Wedocumentthesesimulationstudies

in this section.

Simulationdata

Two sets of simulated data are used in this study. Each

set of data has two groups of n arrays representing gene

expressions under two phenotypic groups (group A and

group B). The numbers of up and down regulated genes

are denoted by m+

generality, group B is set to represent the normal pheno-

type, so up (down) regulation of a gene refers to its over

(under) expression in group A.

• SIMU: Each array has m = 1000 genes. The number

of differentially expressed genes (DEGs) is set to be

100, which implies that the number of

non-differentially expressed genes (NDEGs) is

m0= 900. For both groups, all genes are normally

distributed with standard deviation σ = 0.35 which is

estimated from the biological data. Every two distinct

genes have correlation coefficient 0.9 which is

estimated from the biological data. As a reference, the

sample Pearson correlation coefficient averaged over

all pairs of genes for biological data used in this study

are: 0.91 for HYPERDIP, 0.93 for TEL, and 0.91 for

TALL. The algorithm used to generate these

correlated observations is stated in [36] and is similar

to the method used in [37]. This high correlation

between non-normalized gene expressions can

introduce high correlation between the test statistics

[38] and result in high instability of the list of DEGs.

This phenomenon was documented and discussed in

[39]. We also conduct simulations with

non-homogeneous gene correlation structure and the

results are similar to that of SIMU. Details can be

found in Section 6 of the Additional file 1.

The expectations of DEGs in group A (yA

2,...,m+

constant e for over-expressed genes (i = 1,...,m+

and −e for under-expressed genes (i = m−

1and m−

1, respectively. Without loss of

ij, i = 1,

1+ m−

1, j = 1,2,...,n) are set to be a

1)

1+ 1,...,

Page 7

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 7 of 10

100). Here the effect size e takes value in {0.2,0.4,··· ,

3.4,3.6}. (m+

(balanced differential expression structure) or (90,10)

(unbalanced differential expression structure). For all

genes in group B and NDEGs in group A, their

expectations are set to be 0. The sample size in each

group is set to be n, taking values in {5,10}.

• SIMU-BIO: To match the statistical properties of

real gene expression more closely and mimic other

noise sources such as non-additive noise, we apply

resampling method to the biological data to construct

an additional set of data.

We apply t-test to HYPERDIP and TEL (79 arrays

chosen from each set) without any normalization

procedure or multiple testing adjustment. At

significance level 0.05, 734 genes are detected as

1,m−

1) is set to be either (60,40)

DEGs with an unbalanced differential expression

structure (677 up-regulated and 57 down-regulated).

We record the mean difference across HYPERDIP

and TEL for each DEG as its effect size (ei). Then we

combine HYPERDIP and TEL data and randomly

permute the arrays. After that we randomly choose

2n arrays and divide them into two groups A and B

of n arrays each, mimicking two biological conditions

without differentially expressed genes. Here the

sample size n takes value in {5,10}. We add the

recorded effect sizes to 734 genes (identified earlier)

in group A. We also add addition effect size e to 677

up-regulated genes and −e to 57 down-regulated

genes in group A where e takes value in {0,0.2,0.4,···,

3.4,3.6}. These 734 genes are defined as our DEGs in

this simulation. Similarly, we apply this resampling

Effect Size

Number of True Positives

0.2 0.6 1.01.41.8 2.22.63.0 3.4 0.20.61.0 1.4

Effect Size

1.82.2 2.63.0 3.4

NONE

QUANT

RANK

(a) m+

1= 90 ,m−

1=10 ,n =5

Number of True Positives

0.20.61.01.4

Effect Size

1.82.2 2.63.0 3.40.2 0.61.01.4

Effect Size

1.8 2.22.63.03.4

NONE

QUANT

RANK

(b) m+

1= 60 ,m−

1=40 ,n =5

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Number of True Positives

NONE

QUANT

RANK

(c) m+

1= 90 ,m−

1=10 ,n =10

Number of True Positives

NONE

QUANT

RANK

(d) m+

1= 60 ,m−

1=40 ,n =10

Figure 3 Simulation results (SIMU). Average number of true positives as functions of effect size for SIMU. The error bar represents one standard

deviation above and below average. Total number of truly differentially expressed genes is 100 with m+

genes, respectively. Data replicates: 20.

1up-regulated and m−

1down-regulated

Page 8

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 8 of 10

Additional Effect Size

Number of True Positives

NONE

QUANT

RANK

(a) m+

1= 677 ,m−

1= 57 ,n = 5

Additional Effect Size

Number of True Positives

NONE

QUANT

RANK

(b) m+

1= 259 ,m−

1= 287 ,n = 5

Additional Effect Size

Number of True Positives

0

100 200 300 400 500 600 734

0

100 200 300 400 500 600 734

NONE

QUANT

RANK

(c) m+

1= 677 ,m−

1= 57 ,n = 10

Additional Effect Size

Number of True Positives

0.0

0

100

200

300

400

546

0

100

200

300

400

546

NONE

QUANT

RANK

(d) m+

1= 259 ,m−

1= 287 ,n = 10

0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.6

0.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.60.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.6

0.0 0.4 0.8 1.2 1.6 2.0 2.4 2.8 3.2 3.6

Figure 4 Simulation results (SIMU-BIO). Average number of true positives as functions of effect size for SIMU-BIO. The error bar represents one

standard deviation above and below average. Total number of truly differentially expressed genes is m+

down-regulated genes, respectively. Data replicates: 20.

1+ m1

1with m+

1up-regulated and m−

1

procedure to TALL and TEL (45 arrays chosen from

each set) and 546 genes are defined to be DEGs with

a balanced differential expression structure (259

up-regulated and 287 down-regulated). The sample

size n takes value in {5,10} and the additional effect

size e takes value in {0,0.2,0.4,··· ,3.4,3.6}.

Hypothesis testing methods

We use Student’s t-test to compute unadjusted p-values

andthenapplytheBonferronimultipletestingadjustment

to compute the adjusted p-values and control the family-

wise error rate (FWER) at 0.05 level.

Two alternative tests, namely the Wilcoxon rank-sum

test and permutation N-test are also used in this study.

The results are largely consistent with those obtained

from the t-test and can be found in Section 6 in the Addi-

tional file 1. The N-test is a multivariate nonparametric

test which has been used to successfully select dif-

ferentially expressed genes and gene combinations in

microarray data analysis [23,40-42]. A brief introduction

of this test can be found in Section 1 in the Additional

file 1.

Results and discussion

We randomly generate 20 sets of data per tuning param-

eter for SIMU and SIMU-BIO. We apply normalization

procedures first and then conduct hypothesis tests to

obtain raw p-values. After that, we apply the Bonfer-

roni multiple testing adjustment to get adjusted p-values.

We declare a gene to be differentially expressed if its

adjusted p-value is less than a prespecified significance

level 0.05. The estimated mean and standard deviation of

the true positives are reported in Figures 3 and 4. Various

results with additional tests (Wilcoxon rank-sum test and

Page 9

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 9 of 10

permutation N-test), sample sizes (n = 15,20) and non-

homogeneous gene correlation structure including false

positive plots can be found in Section 6 in the Additional

file 1.

By removing the noise from the observed gene

expressions, quantile and rank normalization procedures

improve the statistical power of the subsequent differen-

tial expression analyses when effect size is small. How-

ever, when e becomes large, the testing powers based on

the normalized expressions converge to fixed numbers

strictly less than 1.0. This confirms our previous theo-

retical derivations.

Conclusions

Microarray technology has been used in many areas of

biomedical research. Biomedical researchers rely on this

technology to identify differentially expressed genes. Due

tothe“largep,smalln”natureofthemicroarraydata,mul-

tiple testing correction must be applied in differentially

expression analysis. As we all know, stringent control of

Type I error invariably comes with the price of reduced

testing power. However, the success of most microar-

ray studies depends critically on the ability of differen-

tial expression analysis to identify the “right genes” and

researchers cannot afford to miss many these targets.

High statistical power can be achieved in a study with

the following properties.

1. An adequate sample size. Clearly, this is a reliable

way to increase statistical power. Everyone seems to

agree on it but not everyone practices it. Many years

ago this was due to the high cost of conducting

microarray experiments. Currently it only costs a

fraction to obtain the same number of arrays. In a

sense, the myth that “five arrays per group should be

good enough” only reflects the fact that it takes a long

time to change old, perhaps even anachronic habits.

2. Small variance. It is well known that a large

proportion of the variance of gene expression is

induced by undesirable systematic variations and

various technical noise. Microarray technology has

been evolving very fast in the past years and we think

it is not unreasonable to assume that the technical

noise level is getting lower. However, variance

induced by biological heterogeneity will not be

affected by the advances of technology. For certain

data, using a normalization procedure, such as

QUANT or RANK, can reduce this variance and

help detect DEGs. We must point out that these

elegant variance reduction procedures can also alter

the mean expression and increase sample variance

when the true effect size is large. This bias-variance

trade-off is common in different branches of

statistics and should not be conveniently ignored.

3. Strong true effect size. Based on our experience, this

is often invoked as a reason to justify the use of small

sample size in a study a priori. In our study, we

demonstrate that one cannot simply “trade” sample

size by effect size. Both our theoretical derivations

and simulation studies indicate that as long as the

sample size is small, the testing power of a typical

gene differential expression analysis based on

quantile or rank normalized data never reaches 100%

no matter how large the effect size is. A large n is still

critical for finding informative genes in this situation.

One main motivation of our study is to dismiss the dan-

gerous idea that “five arrays per-group ought to be good

enough for my study”. Our somewhat counter-intuitive

findings suggest that if data with dramatic gene differen-

tiation have only limited sample size (e.g., less than 10

per group), rank and quantile normalizations may not

be able to improve testing power as one expects. For

such a scenario we recommend conducting an additional

differential expression analysis with other normalization

procedure or even without normalization first, and then

compare/combine the results with the original analysis

with quantile or rank normalization.

Although we choose to focus on the Affymetrix

GeneChip platform throughout this paper, we believe

our conclusions should be valid for other array plat-

forms which require/recommend normalization, such as

Affymetrix exon arrays, Illumina BeadChip arrays and

many others. We hope this study can help biological

researchers choose an appropriate normalization proce-

dure in their experiments or even develop novel normal-

ization procedures with better downstream testing power

when the gene differential expression is dramatic.

Additional file

Additional file 1: Supplementary material.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

All three authors have equal contribution to this paper including the original

idea, study design, theoretical derivations, simulations and summary of the

findings. All authors read and approved the final manuscript.

Acknowledgements

This research is supported by the University of Rochester CTSA award number

UL1 RR024160 from the National Center for Research Resources and the

National Center for Advancing Translational Sciences of the National Institutes

of Health; NIH/NIAID HHSN272201000055C/N01-AI-50020 from the National

Institutes of Health; NIH 5 R01 AI087135-02 from the National Institutes of

Health; and NIH 2 R01 HL062826-09A2 from the National Institutes of Health.

The content is solely the responsibility of the authors and does not necessarily

represent the official views of the National Center for Research Resources or

the National Institutes of Health. We appreciate Ms. Christine Brower’s

technical assistance with computing. In addition, we would like to thank Ms.

Malora Zavaglia and Ms. Jing Che for their proofreading effort.

Page 10

Qiu etal. BMCBioinformatics 2013, 14:124

http://www.biomedcentral.com/1471-2105/14/124

Page 10 of 10

Received: 14 September 2012 Accepted: 7 February 2013

Published: 11 April 2013

References

1.Hartemink AJ, Gifford DK, Jaakkola TS, Young RA: Maximum likelihood

estimation of optimal scaling factors for expression array

normalization. ProcSPIEBIOS 2001, 132:Article4266.

2.Scherer A: BatchEffectsandNoiseinMicroarrayExperiments:Sourcesand

Solutions. Chichester: Wiley; 2009.

3.Bolstad B, Irizarry R, Astrand M, Speed T: A comparison of normalization

methods for high density oligonucleotide array data based on

variance and bias. Bioinformatics 2003, 19:185–193.

4.Park T, Yi S, Kang S, Lee S, Lee Y, Simon R: Evaluation of normalization

methods for microarray data. BMCBioinformatics 2003, 4:33.

5.Rao Y, Lee Y, Jarjoura D, Ruppert AS, Liu CG, Hsu JC, Hagan JP: A

comparison of normalization techniques for microRNA microarray

data. StatApplGenetMolBiol 2008, 7:Article22. [http://dx.doi.org/10.

2202/1544-6115.1287]

6.Pradervand S, Weber J, Thomas J, Bueno M, Wirapati P, Lefort K, Dotto GP,

Harshman K: Impact of normalization on miRNA microarray

expression profiling. RNA 2009, 15(3):493–501. [http://dx.doi.org/10.

1261/rna.1295509]

7.Quackenbush J: Microarray data normalization and transformation.

NatGenet 2002, 32(Suppl):496–501. [http://dx.doi.org/10.1038/ng1032]

8. Bilban M, Buehler LK, Head S, Desoye G, Quaranta V: Normalizing DNA

microarray data. CurrIssuesMolBiol 2002, 4(2):57–64.

9.Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S, Ellis B,

Gautier L, Ge Y, Gentry J, Hornik K, Hothorn T, Huber W, Iacus S, Irizarry R,

Leisch F, Li C, Maechler M, Rossini AJ, Sawitzki G, Smith C, Smyth G, Tierney

L, Yang JYH, Zhang J: Bioconductor: open software development for

computational biology and bioinformatics. GenomeBiol 2004,

5(10):R80. [http://dx.doi.org/10.1186/gb-2004-5-10-r80]

10. R Development Core Team: R:ALanguageandEnvironmentforStatistical

Computing. Vienna: R Foundation for Statistical Computing; 2006. [http://

www.R-project.org] [ISBN 3-900051-07-0].

11. Okoniewski M, Miller C: Comprehensive analysis of affymetrix exon

arrays using BioConductor. PLoSComputBiol 2008, 4:e6.

12. Robinson MD, Speed TP: A comparison of Affymetrix gene expression

arrays. BMCBioinformatics 2007, 8:449. [http://dx.doi.org/10.1186/1471-

2105-8-449]

13. Du P, Kibbe WA, Lin SM: Lumi: a pipeline for processing Illumina

microarray. Bioinformatics 2008, 24(13):1547–1548. [http://dx.doi.org/10.

1093/bioinformatics/btn224]

14. Schmid R, Baum P, Ittrich C, Fundel-Clemens K, Huber W, Brors B, Eils R,

Weith A, Mennerich D, Quast K: Comparison of normalization methods

for Illumina BeadChip HumanHT-12 v3. BMCGenomics 2010, 11:349.

[http://dx.doi.org/10.1186/1471-2164-11-349]

15. Dunning MJ, Smith ML, Ritchie ME, Tavar´ e S: beadarray: R classes and

methods for Illumina bead-based data. Bioinformatics 2007,

23(16):2183–2184. [http://dx.doi.org/10.1093/bioinformatics/btm311]

16. Bullard JH, Purdom E, Hansen KD, Dudoit S: Evaluation of statistical

methods for normalization and differential expression in mRNA-Seq

experiments. BMCBioinformatics 2010, 11:94. [http://dx.doi.org/10.1186/

1471-2105-11-94]

17. Staaf J, Vallon-Christersson J, Lindgren D, Juliusson G, Rosenquist R,

H¨ oglund M, Borg A, Ringn´ er M: Normalization of Illumina Infinium

whole-genome SNP data improves copy number estimates and

allelic intensity ratios. BMCBioinformatics 2008, 9:409. [http://dx.doi.

org/10.1186/1471-2105-9-409]

18. ’t Hoen P, Ariyurek Y, Thygesen H, Vreugdenhil E, Vossen R, De Menezes R,

Boer J, Van Ommen G, Den Dunnen J: Deep sequencing-based

expression analysis shows major advances in robustness, resolution

and inter-lab portability over five microarray platforms. NucleicAcids

Res 2008, 36(21):e141.

19. Hu J, He X: Enhanced quantile normalization of microarray data to

reduce loss of information in gene expression profiles. Biometrics

2007, 63:50–59.

20. Wu Z, Aryee M: Subset quantile normalization using negative control

features. IntJComputBiol 2010, 17(10):1385–1395.

21. Hansen K, Irizarry R, Wu Z: Removing technical variability in RNA-seq

data using conditional quantile normalization. Biostatistics 2011,

13(2):204–216.

22. Tsodikov A, Szabo A, Jones D: Adjustments and measures of

differential expression for microarray data. Bioinformatics 2002,

18(2):251–260.

23. Szabo A, Boucher K, Carroll W, Klebanov L, Tsodikov A, Yakovlev A:

Variable selection and pattern recognition with gene expression

data generated by the microarray technology. MathBiosci 2002,

176:71–98.

24. Tusher VG, Tibshirani R, Chu G: Significance analysis of microarrays

applied to the ionizing radiation response. ProcNatlAcadSciUSA

2001, 98(9):5116–5121. [http://dx.doi.org/10.1073/pnas.091062498]

25. Sidak Z: Rectangular confidence regions for the means of

multivariate normal distributions. JAmStatAssoc 1967, 62:626–633.

26. Holm S: A simple sequentially rejective multiple test procedure.

ScandJStat 1979, 6:65–70.

27. Simes R: An improved Bonferroni procedure for multiple tests of

significance. Biometrika 1986, 73(3):751.

28. Westfall PH, Young SS: Resampling-BasedMultipleTesting. New York: Wiley;

1993.

29. Benjamini Y, Hochberg Y: Controlling the false discovery rate: A

practical and powerful approach to multiple testing. JRStatSocSerB

1995, 57:289–300.

30. Dudoit S, Yang YH, Callow MJ, Speed TP: Statistical methods for

identifying differentially expressed genes in replicated cDNA

microarray experiments. StatSin 2002, 12:111–139.

31. Lee MLT: AnalysisofMicroarrayGeneExpressionData. New York: Springer;

2004.

32. Bremer M, Himelblau E, Madlung A: Introduction to the statistical

analysis of two-color microarray data. MethodsMolBiol 2010,

620:287–313. [http://dx.doi.org/10.1007/978-1-60761-580-4 9]

33. Yakovlev AY, Klebanov L, Gaile D: StatisticalMethodsforMicroarrayData

Analysis. New York: Springer; 2010.

34. Yeoh EJ, Ross ME, Shurtleff SA, Williams WK, Patel D, Mahfouz R, Behm FG,

Raimondi SC, Relling MV, Patel A, Cheng C, Campana D, Wilkins D, Zhou X,

Li J, Liu H, Pui CH, Evans WE, Naeve C, Wong L, Downing JR:

Classification, subtype discovery, and prediction of outcome in

pediatric acute lymphoblastic leukemia by gene expression

profiling. CancerCell 2002, 1(2):133–143.

35. Johnson NL, Kotz S, Balakrishnan N: ContinuousUnivariateDistributions,

Volumn2,secondedition. New York: John Wiley & SOns Inc.; 1995.

36. Hu R, Qiu X, Glazko G, Klebanov L, Yakovlev A: Detecting intergene

correlation changes in microarray analysis: a new approach to gene

selection. BMCBioinformatics 2009, 10:20. [http://dx.doi.org/10.1186/

1471-2105-10-20]

37. Tripathi S, Emmert-Streib F: Assessment method for a power analysis

to identify differentially expressed pathways. PloSone 2012,

7(5):e37510.

38. Qiu X, Hu R: Correlation between the true and false discoveries in a

positively dependent multiple comparison problem. In IMSAndrei

YakovlevCollection. Beachwood, Ohio, USA: Institute of Mathematical

Statistic; 2010.

39. Qiu X, Brooks AI, Klebanov L, Yakovlev A: The effects of normalization

on the correlation structure of microarray data. BMCBioinformatics

2005, 6:120. [http://dx.doi.org/10.1186/1471-2105-6-120]

40. Szabo A, Boucher K, Jones D, Tsodikov AD, Klebanov LB, Yakovlev AY:

Multivariate exploratory tools for microarray data analysis.

Biostatistics 2003, 4(4):555–567. [http://dx.doi.org/10.1093/biostatistics/4.

4.555]

41. Xiao Y, Frisina R, Gordon A, Klebanov L, Yakovlev A: Multivariate search

for differentially expressed gene combinations. BMCBioinformatics

2004, 5:164. [http://dx.doi.org/10.1186/1471-2105-5-164]

42. Klebanov L, Gordon A, Xiao Y, Land H, Yakovlev A: A permutation test

motivated by microarray data analysis. ComputStatandDataAnal

2006, 50(12):3619–3628.

doi:10.1186/1471-2105-14-124

Citethisarticleas:Qiuetal.:Theimpactofquantileandranknormalization

procedures on the testing power of gene differential expression analysis.

BMCBioinformatics 2013 14:124.