Generalized eta and omega squared statistics: measures of effect size for some common research designs.
ABSTRACT The editorial policies of several prominent educational and psychological journals require that researchers report some measure of effect size along with tests for statistical significance. In analysis of variance contexts, this requirement might be met by using eta squared or omega squared statistics. Current procedures for computing these measures of effect often do not consider the effect that design features of the study have on the size of these statistics. Because research-design features can have a large effect on the estimated proportion of explained variance, the use of partial eta or omega squared can be misleading. The present article provides formulas for computing generalized eta and omega squared statistics, which provide estimates of effect size that are comparable across a variety of research designs.
- [Show abstract] [Hide abstract]
ABSTRACT: Modafinil is a medication licensed for the treatment of narcolepsy. However, it has been reported that healthy individuals without wakefulness disorders are using modafinil off-label to enhance cognitive functioning. Although some studies have reported that modafinil improves cognitive task performance in healthy volunteers, numerous other studies have failed to detect cognitive enhancing effects of modafinil on several well-established neuropsychological tasks. Interestingly, several clinical and preclinical studies have found that improved cognitive task performance by modafinil is accompanied by slower response times. This observation raises the question as to whether this slowing of response time in healthy volunteers is a necessary and sufficient condition for cognitive enhancement with modafinil. The aim of the current experiment was to explore this question by investigating the effects of modafinil on the Hayling Sentence Completion Test (HSCT).PLoS ONE 11/2014; 9(11):e110639. · 3.53 Impact Factor - SourceAvailable from: Siobhan Vye[Show abstract] [Hide abstract]
ABSTRACT: Biological invasions, nutrient enrichment and ocean warming are known to threaten biodiversity and ecosystem functioning. The independent effects of these ecological stressors are well studied, however, we lack understanding of their cumulative effects, which may be additive, antagonistic or synergistic. For example, the impacts of biological invasions are often determined by environmental context, which suggests that the effects of invasive species may vary with other stressors such as pollution or climate change. This study examined the effects of an invasive seaweed (Sargassum muticum) on the structure and functioning of a synthetic macroalgal assemblage and tested explicitly whether these effects varied with nutrient enrichment and ocean warming. Overall, the presence of S. muticum increased assemblage productivity rates and warming altered algal assemblage structure, which was characterised by a decrease in kelp and an increase in ephemeral green algae. The effects of S. muticum on total algal biomass accumulation, however, varied with nutrient enrichment and warming, producing antagonistic cumulative effects on total algal biomass accumulation. These findings show that the nature of stressor interactions may vary with stressor intensity and among response variables, which leads to less predictable consequences for the structure and functioning of communities.Oikos 12/2014; · 3.56 Impact Factor - SourceAvailable from: Martin Nyolt
Article: Computational State Space Models for Activity and Intention Recognition. A Feasibility Study
[Show abstract] [Hide abstract]
ABSTRACT: Background Computational state space models (CSSMs) enable the knowledge-based construction of Bayesian filters for recognizing intentions and reconstructing activities of human protagonists in application domains such as smart environments, assisted living, or security. Computational, i. e., algorithmic, representations allow the construction of increasingly complex human behaviour models. However, the symbolic models used in CSSMs potentially suffer from combinatorial explosion, rendering inference intractable outside of the limited experimental settings investigated in present research. The objective of this study was to obtain data on the feasibility of CSSM-based inference in domains of realistic complexity. Methods A typical instrumental activity of daily living was used as a trial scenario. As primary sensor modality, wearable inertial measurement units were employed. The results achievable by CSSM methods were evaluated by comparison with those obtained from established training-based methods (hidden Markov models, HMMs) using Wilcoxon signed rank tests. The influence of modeling factors on CSSM performance was analyzed via repeated measures analysis of variance. Results The symbolic domain model was found to have more than states, exceeding the complexity of models considered in previous research by at least three orders of magnitude. Nevertheless, if factors and procedures governing the inference process were suitably chosen, CSSMs outperformed HMMs. Specifically, inference methods used in previous studies (particle filters) were found to perform substantially inferior in comparison to a marginal filtering procedure. Conclusions Our results suggest that the combinatorial explosion caused by rich CSSM models does not inevitably lead to intractable inference or inferior performance. This means that the potential benefits of CSSM models (knowledge-based model construction, model reusability, reduced need for training data) are available without performance penalty. However, our results also show that research on CSSMs needs to consider sufficiently complex domains in order to understand the effects of design decisions such as choice of heuristics or inference procedure on performance.PLoS ONE 11/2014; 9(11):e109381. · 3.53 Impact Factor
Page 1
Generalized Eta and Omega Squared Statistics: Measures of
Effect Size for Some Common Research Designs
Stephen Olejnik
University of Georgia
James Algina
University of Florida
The editorial policies of several prominent educational and psychological journals
require that researchers report some measure of effect size along with tests for
statistical significance. In analysis of variance contexts, this requirement might be
met by using eta squared or omega squared statistics. Current procedures for
computing these measures of effect often do not consider the effect that design
features of the study have on the size of these statistics. Because research-design
features can have a large effect on the estimated proportion of explained variance,
the use of partial eta or omega squared can be misleading. The present article
provides formulas for computing generalized eta and omega squared statistics,
which provide estimates of effect size that are comparable across a variety of
research designs.
It is often argued that researchers can enhance the
presentation of their research findings by including an
effect-size measure along with a test of statistical sig-
nificance. An effect-size measure is a standardized
index and estimates a parameter that is independent of
sample size and quantifies the magnitude of the dif-
ference between populations or the relationship be-
tween explanatory and response variables. Two broad
categories of effect size are standardized mean differ-
ences and measures of association or the proportion of
variance explained (Richardson, 1996). Kirk (1996)
has provided a nice summary of the variety of mea-
sures used as estimators for effect size. Olejnik and
Algina (2000) demonstrated the use and interpretation
of many of these effect-size indices. Although many
research methodologists have recommended the use
of effect-size measures, others have been critical of
their use and have cautioned that they may be easily
misinterpreted (Fern & Monroe, 1996; Maxwell,
Camp, & Arvey, 1981; O’Grady, 1982; Richardson,
1996). In particular, the research design of a study has
been identified as a potential source of confusion and
misuse. For example, suppose two different research-
ers conduct studies to estimate the effect of the same
treatment factor; one study includes a blocking factor
(e.g., gender), but the other does not. If gender pre-
dicts the dependent variable, the study with the block-
ing factor will have the smaller within-cell variance.
If, in each study, the within-cell standard deviation is
used as the denominator of the standardized-mean-
difference effect size, the two studies will estimate
different effect sizes even though the difference in
population means is the same. Because of this prob-
lem, Glass, McGaw, and Smith (1981) recommended
that meta-analysts ignore blocking factors in factorial
designs when computing the standard deviation used
as the denominator of a standardized mean difference.
Using simulated data, Morris and DeShon (1997)
demonstrated the magnitude of the error that is intro-
duced if the blocking factor is not ignored when es-
timating the standardized mean difference. Glass et al.
(1981, pp. 116–122) presented a similar recommen-
dation regarding the estimation of the standardized
mean difference obtained when using gain scores or
covariance-adjusted posttest measures in a random-
ized groups pretest–posttest design. Similarly, Dun-
lap, Cortina, Vaslow, and Burke (1996) discussed
matching and repeated measures designs and the im-
portance of taking into consideration the correlation
between measures when estimating the standardized
mean difference. Olejnik and Algina (2000) discussed
Stephen Olejnik, Department of Educational Psychology,
College of Education, University of Georgia; James Algina,
College of Education, University of Florida.
Correspondence concerning this article should be ad-
dressed to Stephen Olejnik, Department of Educational Psy-
chology, College of Education, University of Georgia, 325
Aderhold Hall, Athens, Georgia 30602-7143. E-mail:
olejnik@coe.uga.edu
Psychological Methods
2003, Vol. 8, No. 4, 434–447
Copyright 2003 by the American Psychological Association, Inc.
1082-989X/03/$12.00DOI: 10.1037/1082-989X.8.4.434
434
Page 2
these issues and presented recommendations regard-
ing the computation of the standardized mean differ-
ence in a wide variety of designs.
Design considerations are also important when the
effect size is estimated using a measure of the pro-
portion of variance explained, typically eta squared or
omega squared. The effect of ignoring nested factors
(Wampold & Serlin, 2000) and the importance of dis-
tinguishing between random and fixed factors (cf.
Charter, 1982; Dodd & Schultz, 1973; Dwyer, 1974;
Fleiss, 1969; Halderson & Glasnapp; 1972; Vaughan
& Corballis, 1969) has also been discussed.
The number of factors studied in the completely
randomized (CR) design is still another design issue
that influences the proportion of variance explained.
For example, consider the formula for eta squared in
a CR design:
? ˆ2=SSEffect
SST
,
(1)
where SSEffectis the sum of squares for the factor for
which the effect size is being estimated and SSTis the
total sum of squares. In a multifactor CR design, the
sum of squares due to each factor and due to multi-
factor interactions contributes to the total sum of
squares for the data set. Therefore, if the same factor
is investigated in several studies, but the number or
nature of any additional factors varies across the stud-
ies, there may be nonrandom differences in SSTfor
the studies. Consequently, eta squared for the factor of
interest may not be comparable across studies. The
same concern applies to omega squared because it is
computed as the ratio of the estimated variance due to
an effect to the estimated total variance.
Partial eta squared and partial omega squared have
been recommended (e.g., Keppel, 1991, pp. 222–224)
as solutions to the comparability problem. In a mul-
tifactor CR design, partial eta squared is computed as
the ratio of the effect sum of squares to the sum of the
effect sum of squares and the subjects-within-cells
sum of squares (SSs/Cells):
? ˆP
2=
SSEffect
SSEffect+ SSs?Cells.
A convenient formula for calculating partial omega
squared is
? ˆP
2=
SSEffect− dfEffectMSs?Cells
SSEffect+ ?N − dfEffect? MSs?Cells
(Keren & Lewis, 1979). Both the partial eta squared
and the partial omega squared eliminate the influence
of other factors in the design on the denominator of
these statistics. The General Linear Model program in
SPSS reports partial eta squared when the effect size
is requested.
Cohen (1973) cautioned however that the use of
partial eta squared may be inappropriate and can in
fact be misleading when the design includes a block-
ing factor. A blocking factor in a CR factorial design
has the effect of reducing the subjects-within-cells
sum of squares. As a result, computing partial eta- or
omega-squared statistics provides an estimate of ef-
fect size that is not comparable with effect sizes es-
timated in studies that do not include the blocking
variable. The partial eta or omega squared statistics
would provide estimates of effect size that can be
much larger than the effect size estimated from a
study that does not include the blocking factor. Al-
though some might view the larger effect-size mea-
sure as a reward for a stronger research design, the
increase in the effect size due to the blocking factor
causes difficulty when comparing or aggregating ef-
fect sizes across studies. Thus partial eta and omega
squared are subject to the same caution that Glass et
al. (1981) set forth for the standardized-mean-
difference effect size.
Although the importance of the Glass et al. (1981)
caution about choosing the appropriate standard
deviation for computing the standardized mean
difference has generally been recognized by meta-
analysts, Cohen’s caution on the use of partial eta
squared has not, in general, been acknowledged.
SPSS, for example, assumes that all factors in the
design are manipulated factors and computes the
partial eta squared statistics for each factor in the
factorial design. Textbooks presenting analysis of
variance (ANOVA) models (e.g., Keppel, 1991;
Maxwell & Delaney, 2000; Stevens, 1999) and ar-
ticles discussing and critiquing the use of effect-size
measures (e.g., Fern & Monroe, 1996; Richardson,
1996) have not addressed the issue. Consequently,
applied researchers who have been encouraged to
report measures of effect size (Wilkinson and the
Task Force on Statistical Inference, 1999) are most
likely reporting inappropriate effect-size measures
or an effect-size measure that cannot be compared
across studies that do not include the same blocking
factors. Effect-size measures quickly lose their use-
fulness unless they can be compared across a series of
studies.
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
435
Page 3
Purpose
The purpose of this article is to present generalized
eta squared (? ˆ2
alternatives to extant versions of eta and omega
squared. The coefficients ? ˆ2
designs in which there is at least one categorical in-
dependent variable. Each alternative provides an ef-
fect-size measure that is comparable across designs
that are used to investigate the effect of a factor or the
interaction of factors on the same population but vary
in terms of use of blocking factors, covariates, or the
inclusion of additional factors. That is, we seek to
remove the potential confounding of an effect-size
measure and the design used to investigate the effect.
Fleiss (1969), in essence, had the same objective. In
fact, the parameter we present in Equation 2 special-
izes to the parameters presented in Fleiss’s Equations
7 and 13. However, we show how to apply the pa-
rameter and how to estimate the parameter for a much
wider array of designs than was included in Fleiss.
Before introducing our procedures for calculating
? ˆ2
parameter is based on the point of view that the data
in a study arise due to two sources of variance: ma-
nipulated factors in the study and individual differ-
ences. Individual differences are due to stable and
transient characteristics of the participants as well as
to the uncontrolled characteristics of the experimental
setting that account for the fact that scores within the
finest combination of the manipulated factors are not
all equal. For example, in a study designed to inves-
tigate the efficacy of weight-training programs on
strength, measures of individual strength within a
treatment program might vary because of individual
differences in gender, body type, and prior activity
levels (stable characteristics). They may also differ
due to such transitory characteristics as motivational
levels and an individual’s temporary health condition.
Environmental factors such as differences in equip-
ment quality or instrument calibration offer still an-
other source of score variation among individuals who
are in the same weight-training program. Research
designs can differ in the degree to which sources of
individual differences are estimated or controlled. Be-
cause our objective is to provide an effect-size mea-
sure that is comparable across a variety of research
designs, our parameter recognizes and adjusts for dif-
ferences in the number and type of manipulated fac-
tors as well as for differences in the degree to which
sources of individual differences are estimated or con-
trolled.
G) and omega squared (? ˆ2
G), which are
Gand ? ˆ2
Gare intended for
Gand ? ˆ2
G, we define our effect-size parameter. This
Our effect-size parameter is
?Effect
+ ?Individual Differences
2
? ? ?Effect
22
,
(2)
where ? ? 1 if the effect involves only manipulated
factors, and ? ? 0 if the effect involves one or more
measured factors (e.g., gender, Gender × Manipulated
factor). The parameter ?2
of variance. For example, in a one-factor between-
subjects design with J levels,
Effectis defined as in analysis
?Effect
2
=?
j=1
J
??j− ??2
J
,
where ?jis the mean for the jth level of the factor,
and ? is the average of the J means. The variance
?2
due to measured factors, such as gender, interactions
of measured factors with other factors, covariates, and
variance within the cells of the design. Note that if
?2
a main effect of a measured factor, or an interaction of
a measured factor with any other factors, ?2
already have been included in ?2
ting ? ? 0 simply prevents it from being included in
the denominator twice. On the other hand, if ?2
a variance component for a main effect of a manipu-
lated factor, or an interaction involving only manipu-
lated factors, ?2
?2
denominator, because a manipulated factor that has an
effect adds variance to the data. Several examples
illustrate how the effect-size parameter and its com-
ponents work in context and, in subsequent sections,
we show how to estimate the parameter in a wide
variety of research designs.
Suppose the effect of four allergy medicines (factor
D) on psychomotor performance is investigated in a
balanced between-subjects design. The effect size is
Individual Differencesis the sum of variance components
Effect, on the one hand, is a variance component for
Effectwill
Individual Differences. Set-
Effectis
Effectwill not have been included in
Individual Differences. Setting ? ? 1 adds ?2
Effectto the
?D
D+ ?s?Cells
2
1 ? ?2
2
=
?D
2
?D
2+ ?s?Cells
2
.
(3)
Here ?2
variance. In this example, the denominator is the total
variance in the data (i.e., ?2
will become clear, this example does not imply that
the denominator will always comprise the total vari-
ance.
Now suppose that participants are classified by
s/Cells? ?2
Individual Differencesis the within-drug
Y? ?2
D+ ?2
s/Cells), but as
OLEJNIK AND ALGINA
436
Page 4
gender (factor G), and the two-factor design is still
balanced. The effect size for drug is now
?D
2+ ?DG
?D
2+ ?DG
2
1 ? ?D
2+ ?G
2
+ ?s/Cells
2
=
2
?D
2+ ?G
2
+ ?s?Cells
2
.
(4)
Here ?2
within a combination of drug and gender and does
not comprise all of the variation due to indi-
vidual differences. Rather, ?2
?2
the Gender × Drug interaction reflect the operation of
individual differences. Excluding the ?2
nominators of Equations 3 and 4 are equal and thus
estimates of the parameters defined by Equations 3
and 4 would be comparable across the two designs.
That is, even though the design is different in the
two examples, the magnitude of the effect size is
not confounded with the design used to investigate
the effect.
In addition, the design may affect ?2
either because of variation across experiments in con-
trol of characteristics of the experimental setting that
affect the dependent variable or because of variation
across experiments in the population that is sampled.
As an example of variation in control of characteris-
tics of the experimental setting, consider two designs
for investigating the effect of the allergy medicines. In
the first, the drugs are administered only in the morn-
ing. In the second, some participants self-select them-
selves into morning sessions, and some self-select
into afternoon sessions. In the second study, time of
day is not recorded. For both designs, the parameter
defined by Equation 3 would be estimated. If there is
a time-of-day effect, ?2
because the second experimenter has failed to control
for time of day. The parameters for the two designs
will not be equal even if it happens that ?2
for the two designs. As an example of investigating
different populations, suppose a study of the allergy
drugs is conducted using only participants age 60 or
older. In a second study, the participants were of ma-
jority age, but the age range was otherwise unre-
stricted. The effect-size parameter is again given by
Equation 3 and would be larger in the study of older
adults if ?2
?2
both examples, the meaning of a comparison of the
effect-size parameters for the two studies would nec-
s/Cellsis the variance of the criterion variable
G+ ?2
DG+ ?2
s/Cells?
Individual Differencesbecause both the gender effect and
Dterm, the de-
Individual Differences
s/Cellswill vary across designs
Dis the same
Dwas the same in the two studies, but
s/Cellswas smaller in the study of older adults. In
essarily be unclear. Our measures ? ˆ2
intended to address the comparability problem that
arises when different populations are sampled, and no
effect-size measure we are aware of can remove this
comparability problem. Our measures are only in-
tended to make effect-size measures comparable
when different factors are manipulated in two designs,
designs differ in manipulating factors as within-
subjects and between-subjects variables, and/or de-
signs differ in the use of blocking factor or covariates.
Consistent with current measures of eta and omega
squared, our estimates ? ˆ2
require a balanced design. However, it may be pos-
sible to develop alternative measures for unbalanced
designs by using Yates’s (1934) unweighted means
analysis: The cell means are treated as observations
and subjected to an ANOVA (Searle, 1971).
Now suppose that the investigator who used the
Drug × Gender design is interested in an effect size
for gender. The effect variance is ?2
vidual-differences variance is ?2
?2
Gand ? ˆ2
Gare not
Gand ? ˆ2
G, introduced below,
G, and the indi-
G+ ?2
DG+ ?2
s/Cells?
Individual Differences. Consequently the effect size is
?G
2+ ?DG
2
0 ? ?G
2+ ?G
2
+ ?s/Cells
2
=
?G
2
?G
2+ ?DG
2
+ ?s/Cells
2
.
Note that when the effect is a measured factor, the
variance due to that factor enters the denominator
as component of ?2
the denominator through ?2
? ? 0.
Individual Differences; its influence on
Effectis removed because
Eta Squared
To obtain effect sizes that are comparable across
designs, we propose
? ˆG
2=
SSEffect
? ? SSEffect+?
Meas
SSMeas+?
?
SS?
,
(5)
where ? ? 1 if the effect of interest is a manipulated
factor and zero otherwise. The index Meas runs over
all sources of variance that do not include subjects but
do involve a measured factors, (e.g., a blocking factor
or a Block × Manipulated factor interaction), and
SSMeasis the sum of squares for such an effect. The
index ? runs over all sources of variance that involve
subjects or covariates, and SS?is the sum of squares
for such a source of variation. As before, SSEffectis the
sum of squares for the factor for which the effect size
is being estimated (see Keppel, 1991, or Kirk, 1995,
for formulas to compute sum of squares).
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
437
Page 5
Between-Subjects Design With Crossed Factors
In a between-subjects design with crossed factors,
the only source of variance that involves subjects is
subjects-within cells, so
?
?
SS?= SSs?Cells.
In a CR design, all factors are manipulated factors
and, therefore, ? ? 1 for all effects, and
?
Meas
SSMeas= 0.
As a result, generalized eta squared is equal to partial
eta squared for a multifactor CR design. Consider a
two-factor design where factor A is manipulated and
factor b is a measured factor (here, we use lowercase
letters to represent a measured factor such as gender).
If an effect size for factor A is of interest, then SSEffect
? SSA, ? ? 1,
?
Meas
SSMeas= SSb+ SSAb,
and generalized eta squared would be computed by
using
? ˆG
2=
SSA
SSA+ SSb+ SSAb+ SSs?Cells=
SSA
SSTotal.
If an effect size for the measured factor is of interest,
then SSEffect? SSb, ? ? 0,
?
Meas
SSMeas= SSb+ SSAb,
and
? ˆG
2=
SSb
SSb+ SSAb+ SSs?Cells=
SSb
SSTotal− SSA.
If the interaction between the measured factor and
the manipulated factor is of interest, then SSEffect?
SSAb, ? ? 0,
?
Meas
SSMeas= SSb+ SSAb,
and
? ˆG
2=
SSAb
SSb+ SSAb+ SSs?Cells=
SSAb
SSTotal− SSA.
When both factors a and b are measured factors,
?
Meas
SSMeas= SSa+ SSb+ SSab,
and generalized eta squared can be computed by using
the ratio SSab/SST.
Equation 5 can be used to estimate generalized
eta squared for more complicated factorial designs
that include manipulated or measured factors. How-
ever, to facilitate use of generalized eta squared, Table
1 presents explicit formulas for estimating generalized
eta squared in the eight possible three-factor designs
involving none, one, two, or three measured factors.
In Table 1, Latin letters indicate the design, and Greek
letters indicate the effect of interest. To use the table,
the reader labels his or her factors to be consistent
with Table 1. If the goal is to calculate generalized eta
squared for a main effect, the factor of interest is
labeled as A if it is a manipulated factor and a if it is
a measured factor. For example, in a design with an
intervention, gender, and ethnic background as fac-
tors, if we are interested in an effect size for interven-
tion, the design would be Abc, where A represents the
manipulated intervention factor and b and c represent
the other two factors. Generalized eta squared would
equal
Table 1
Selected Formulas for Eta Squared Statistics in a Three-Factor Analysis of Variance
Designa
Source
??????
ABC
ABc
AbC
Abc
aBC
abC
aBc
abc
SSA/(SSA+ SSs/ABC)
SSA/(SST− SSB− SSAB)
SSA/(SST− SSC− SSAC)
SSA/SST
SSa/(SST− SSB− SSC− SSBC)
SSa/(SST− SSC)
SSa/(SST− SSB)
SSa/SST
SSAB/(SSAB+ SSs/ABC)
SSAB/(SST− SSA− SSB)
SSAb/(SST− SSA− SSC− SSAC)
SSAb/(SST− SSA)
SSaB/(SST− SSB− SSC− SSBC)
SSab/(SST− SSC)
SSaB/(SST− SSB)
SSab/SST
SSABC/(SSABC+ SSs/ABC)
SSABc/(SST− SSA− SSB− SSAB)
SSAbC/(SST− SSA− SSC− SSAC)
SSAbc/(SST− SSA)
SSaBC/(SST− SSB− SSC− SSBC)
SSabC/(SST− SSC)
SSaBc/(SST− SSB)
SSabc/SST
aUppercase letters indicate a manipulated factor; lowercase letters indicate a measured factor.
OLEJNIK AND ALGINA
438
Page 6
? ˆG
2=SSA
SST.
If gender was the factor of interest, the design would
be aBc (gender would be the a factor, intervention
would be the B factor, and ethnic background would
be the c factor) and
? ˆG
2=
SSa
SST− SSB.
For a concrete example, Keppel (1991, p. 438) pro-
vided an ANOVA table for a 3 × 2 × 2 factorial design
where the first two factors are manipulated, and the
third factor is measured (grade level). The design is
ABc. Partial eta squared for the first manipulated fac-
tor would be computed as SSA/(SSA+ SSs/ABc) ?
33.63/(33.63 + 84.00) ? .286. Generalized eta
squared would be computed as SSA/(SST− SSB−
SSAB) ? 33.63/(227.93 − 52.26 − 5.64) ? .198, a
31% reduction in the effect size estimated compared
with partial eta squared. The magnitude of the differ-
ence between partial eta and generalized eta squared
depends on the extent to which the measured factors
and the interactions between manipulated and mea-
sured factors reduce the subjects within-cell sum of
squares.
Analysis of Covariance
Although completely randomized designs are very
popular among applied researchers, they are ineffi-
cient. To improve the efficiency by reducing the
within-cell variance, many researchers design their
studies to include at least one preintervention measure
that is expected to correlate with the outcome(s) of
interest. Textbooks (e.g., Keppel, 1991, pp. 322–323)
frequently recommend computing partial eta squared
using adjusted sums of squares. However, the covari-
ate serves a purpose similar to that of a blocking
variable discussed in the previous section. Using the
adjusted sum of squares to compute eta squared re-
sults in a statistic that is not comparable with esti-
mates of effect size from studies that do not include a
covariate or from studies that include a covariate with
a different strength of relationship to the dependent
variable. The generalized formula (see Equation 5) for
eta squared can be used to provide an effect-size mea-
sure that is comparable across designs and covariates.
For example, for the randomized-groups pretest–
posttest design with a single covariate, if eta squared
for the manipulated treatment factor A was of interest
then SSEffect? SSA, ? ? 1,
?
Meas
SSMeas= 0
because A is the only factor, and
?
?
Then eta squared would be computed by using
SS?= SScovariate+ SSs?Cells.
? ˆG
2=
SSA
SSA+ SScovariate+ SSs?Cells.
Note we are assuming a model specifying that the
within-group regression slopes are equal (no Covari-
ate × Treatment interaction).
Repeated Measures Designs
Another approach that addresses the inefficiency of
CR designs is to have all participants complete each
treatment condition. These designs are referred to as
repeated measures or within-subjects designs. Be-
cause observations under the conditions being studied
are made on the same individuals, scores obtained
under the various levels of a factor are correlated. The
correlation between the measures reduces the error
sum of squares used in testing hypotheses about re-
peated measures factors, resulting in much more ef-
ficient tests. However, as a result of the reduction in
the error sum of squares, the eta squared calculated
for the within-subjects design may not be comparable
with the eta squared calculated if a between-subjects
design was used to estimate the effects of the same
factors. For example, if a one-factor between-subjects
design is used to investigate the effect of factor A,
then
? ˆ2=
SSA
SSA+ SSs?Cells
is the only eta squared that could be calculated, where
SSs/cellsestimates (N − J) ?2
factor is investigated using a repeated measures de-
sign, a possible eta squared measure is
Individual Differences. If the A
? ˆP
2=
SSA
SSA+ SSsA,
where SSsAis the error sums of squares for the A
effect. The effect sizes for the two designs are not
comparable because SSsAdoes not estimate (N −
J)?2
SSs/Cells.
To obtain eta squared for the repeated measures
design that is comparable with eta squared for the
between-subjects design, we should replace SSsAwith
SSs+ SSsA, because SSs+ SSsAequals the pooled sum
Individual Differencesand tends to be smaller than
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
439
Page 7
of squares within levels (SSs/Cells) of the repeated
measures factor (see Kirk, 1995, p. 254) and estimates
(N − J)?2
what happens when ? ˆ2
effect size for the repeated measures factor A is of
interest, SSEffect? SSA, ? ? 1,
?
Meas
?
?
Individual Differences. This replacement is exactly
Gis used. For example, if the
SSMeas= 0,
SS?= SSs+ SSsA,
and
? ˆG
2=
SSA
SSA+ SSs+ SSsA.
Most computer software (e.g., SPSS, SAS) does not
compute SSs. In a one-way design two alternative
methods for computing
?
?
SS?
are
?n − 1??
j=1
J
Sj
2,
where S2
and SST− SSA, where SSTis the sum of squared
deviations of the observations from the grand mean.
If there are two repeated measures factors, the
formulas in Table 1 can be used to estimate eta
squared. The repeated measures factors are manipu-
lated factors, and the subjects factor is a measured
factor, so the design is ABc. Maxwell and Delaney
(2000, p. 497) provided data for a 2 × 3 single-group
repeated measures design. For these data, partial eta
squared for the first factor would be computed as
jis the variance in the jth level of the factor
? ˆP
2=
SSA
SSA+ SSsA=
289,920
289,920 + 64,080= .819.
By contrast,
? ˆG
2=
SSA
SST− SSB− SSAB
=
289,920
1,133,940 − 285,660 − 105,120= .390.
The dramatic decrease in the effect size estimated is a
function of counteracting the correlation between the
observations.
Mixed Designs
When a between-subjects design is combined with
a repeated measures design, the result is often referred
to as a mixed design or a split-plot design. A repeated
measures factor, as described above, is almost never a
measured factor, but a between-subjects factor may be
either a manipulated or a measured factor.
Consider a study in which there is one repeated
measures factor (B), there is one between-subjects
factor (A), and subjects are nested within levels of
factor A (s/A). If the between-subjects factor is a ma-
nipulated factor then SSEffect? SSA, ? = 1,
?
Meas
SSMeas= 0,
and
?
?
SS?= SSs?A+ SSBs?A.
Generalized eta squared would then be computed as
? ˆG
2=
SSA
SSA+ SSs?A+ SSBs?A=
SSA
SST− SSB− SSAB.
Here SSTis the sum of the squared deviations of each
observation from the grand mean. This is the formula
for ? for design ABc (here c denotes subjects) in
Table 1.
If the between-subjects factor is a measured factor
(e.g., gender) and the interaction is of interest, then
SSEffect? SSaB, ? ? 0,
?
Meas
?
?
SSMeas= SSa+ SSaB,
SS?= SSs?a+ SSBs?a,
and generalized eta squared is computed by using
? ˆG
2=
SSaB
0 × SSaB+ SSa+ SSaB+ SSs?a+ SSBs?a
SSaB
SST− SSB,
=
which is the eta squared formula in Table 1 for ??
when the design is aBc.
Kirk (1995, p. 527) provided an ANOVA summary
table for a 2 × 2 × 4 mixed-model design with the last
factor being within subjects. If the first between-
subjects factor is manipulated (A) and the second be-
tween-subjects factor is measured (c), the design
OLEJNIK AND ALGINA
440
Page 8
could be described as ABc. Partial eta squared for the
first between-subjects factor would be computed as
SSA/(SSA+ SSs/Ac) ? 3.125/(3.125 + 7.250) ? .301.
Generalized eta squared would equal SSA/(SST− SSB
− SSAB) ? 3.125/(235.5 − 194.5 − 19.375) ? .145.
Omega Squared
Between-Subjects Design
Although eta squared is a very popular statistic for
reporting an effect size, it does provide an overesti-
mate of the population proportion of variance ex-
plained. Peters and Van Voorhis (1940, p. 322)
showed that when the explanatory and criterion vari-
ables are unrelated the bias is a function of the number
of levels of the explanatory factor (J) and the total
sample size, (J − 1)/(N − 1). Alternatively, Hays
(1963) suggested that omega squared be used in place
of eta squared. Omega squared is calculated by using
unbiased estimators of the variance components asso-
ciated with the sources of variation in the design.
However, the ratio of the unbiased estimators is not
itself an unbiased estimator of ?2(Winkler & Hays,
1975, p. 766). Nevertheless, omega squared tends to
be less biased than eta squared in small samples (Car-
roll & Nordholm, 1975; Keselman, 1975).
Omega squared, however, has the same limitations
as those discussed above associated with eta and par-
tial eta squared. A generalized form for estimated
omega squared that can be used with between-
subjects (e.g., completely randomized, randomized
block, analysis of covariance), repeated measures, and
mixed designs, when all factors except subjects are
fixed, is
? ˆG
2=
? ˆEffect
+?
Meas
2
? × ? ˆEffect
2
? ˆMeas
2
+ ? ˆ2,
(6)
where ? ? 1 if the effect of interest refers to a ma-
nipulated factor and ? ? 0 otherwise, the index Meas
runs over all sources of variance that do not include
subjects but do involve a measured factor, ?2
variance component for a source of variance that in-
volves a measured factor but does not involve sub-
jects, and ? ˆ2is obtained by pooling all sources of
variance that involve subjects and/or covariates. A
formula more convenient for calculation is
Measis a
? ˆG
2=
?SSEffect− dfEffect× MSError??N
???SSEffect− dfEffect× MSError?
+?
Meas?SSMeas− dfMeas× MSM_Error?
+ N × MSs?Cells??N,
where N is the total number of scores in the analysis.
We include N in the denominator and numerator of
Equation 7 to indicate the connection between com-
ponents of Equation 6 and components of Equation 7:
(7)
? ˆEffect
2
=SSEffect− dfEffect× MSError
N
,
?
Meas
? ˆMeas
2
=?
Meas
?SSMeas− dfMeas× MSM_Error?
N
,
and
? ˆ2= MSs?Cells.
To simplify the formulas that follow, N is deleted
from both numerator and denominator. In SSEffect−
dfEffect× MSError, the quantity MSErroris the error
mean square for testing the effect and in SSMeas−
dfMeas× MSM_Error, the quantity MSM_Erroris the error
mean square for testing the effect labeled Meas. In
fixed-effects between-subjects designs, without co-
variates, MSErrorand MSM_Errorare equal to MSs/Cells.
When covariates are included, MSErroris equal to the
within-cell variance adjusted by the covariates. In
other designs MSs/Cellsis computed from the sums of
squares and degrees of freedom for several of the
sources of variance.
Consider an example of a two-factor design where
factors A and B are manipulated. In a fixed effects
model, the pooled within-cell variance is the only
source of variation associated with subjects so ? ˆ2?
MSs/Cellsin Equation 6. Because A and B are manipu-
lated
?
Meas
?Meas
2
= 0,
and generalized omega squared for factor A is
? ˆG
2=
SSA− dfAMSs?cells
?SSA− dfAMSs?Cells? + N × MSs?Cells,
which simplifies to partial omega squared
? ˆG
2=
SSA− dfAMSs?Cells
SSA+ ?N − dfA?MSs?Cells.
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
441
Page 9
If factor A is a manipulated factor and factor b is a
measured factor, then using Equation 7
? ˆG
2=
SSA− dfAMSs?Cells
?SSA− dfAMSs?Cells? + ?SSb− dfbMSs?Cells?
+ ?SSAb− dfAbMSs?Cells? + N × MSs?Cells
,
which simplifies to omega squared rather than partial
omega squared
? ˆG
2=SSA− dfAMSs?Cells
SST+ MSs?Cells
.
For the measured factor b,
? ˆG
2=
SSb− dfbMSs?Cells
0?SSb− dfbMSs?Cells? + ?SSb− dfbMSs?Cells?
+ ?SSAb− dfAbMSs?Cells? + N × MSs?Cells
,
which simplifies to
? ˆG
2=
SSb− dfbMSs?Cells
SST+ SSA+ J × MSs?Cells,
where J is the number of levels in factor A.
The generalized omega squared formulas (Equa-
tions 6 and 7) can be applied to higher order factorial
designs. Table 2 provides formulas for omega squared
in a three-factor fixed-effects design for various com-
binations of manipulated (uppercase letters) and mea-
sured (lowercase letters) factors. Keppel (1991, p.
438) provided an ANOVA table for a 3 × 2 × 2
factorial design where the first two factors are ma-
nipulated, and the third factor is measured (grade
level). For factor A
? ˆP
2=
33.63 − 2?1.75?
33.63 + ?60 − 2?1.75= .223.
Generalized omega squared for design ABc is com-
puted as
? ˆG
2=
33.63 − 2?1.75?
227.93 − 52.26 − 5.64 + ?3 × 2 − 2?1.75= .170.
Partial omega squared is almost 24% greater than gen-
eralized omega squared. The omega effect sizes are
Table 2
Selected Formulas for Omega Squared Statistics in a Three-Factor Analysis of Variance
SourceDesign
? ˆ2
G
?
ABC
ABc
AbC
Abc
aBC
abC
aBc
abc
[SSA− dfA× MSs/Cells]/[SSA+ (N − dfA)MSs/Cells]
[SSA− dfA× MSs/Cells]/[SST− SSB− SSAB+ (JK − dfA)MSs/Cells]
[SSA− dfA× MSs/Cells]/[SST− SSC− SSAC+ (JL − dfA)MSs/Cells]
[SSA− dfA× MSs/Cells]/[SST+ MSs/Cells]
[SSa− dfa× MSs/Cells]/[SST− SSB− SSC− SSBC+ KL × MSs/Cells]
[SSa− dfa× MSs/Cells]/[SST− SSC+ L × MSs/Cells]
[SSa− dfa× MSs/Cells]/[SST− SSB+ K × MSs/Cells]
[SSa− dfa× MSs/Cells]/[SST+ MSs/Cells]
??
ABC
ABc
AbC
Abc
aBC
abC
aBc
abc
[SSAB− dfAB× MSs/Cells]/[SSAB+ (N − dfAB)MSs/Cells]
[SSAB− dfAB× MSs/Cells]/[SST− SSA− SSB+ (JK − dfAB)MSs/Cells]
[SSAb− dfAb× MSs/Cells]/[SST− SSA− SSC− SSAC+ JL × MSs/Cells]
[SSAb− dfAb× MSs/Cells]/[SST− SSA+ J × MSs/Cells]
[SSaB− dfAb× MSs/Cells]/[SST− SSB− SSC− SSBC+ KL × MSs/Cells]
[SSab− dfab× MSs/Cells]/[SST− SSC+ L × MSs/Cells]
[SSaB− dfaB× MSs/Cells]/[SST− SSB+ K × MSs/Cells]
[SSab− dfab× MSs/Cells]/[SST+ MSs/Cells]
???
ABC
ABc
AbC
Abc
aBC
abC
aBc
abc
[SSABC− dfABC× MSs/Cells]/[SSABC+ (N − dfABC)MSs/Cells]
[SSABc− dfABc× MSs/Cells]/[SST− SSA− SSB− SSAB+ JK × MSs/Cells]
[SSAbC− dfAbC× MSs/Cells]/[SST− SSA− SSC− SSAC+ JL × MSs/Cells]
[SSAbc− dfAbc× MSs/Cells]/[SST− SSA+ J × MSs/Cells]
[SSaBC− dfaBC× MSs/Cells]/[SST− SSB− SSC− SSBC+ KL × MSs/Cells]
[SSabC− dfabC× MSs/Cells]/[SST− SSC+ L × MSs/Cells]
[SSaBc− dfaBc× MSs/Cells]/[SST− SSB+ K × MSs/Cells]
[SSabc− dfabc× MSs/Cells]/[SST+ MSs/Cells]
Note. Uppercase letters indicate a manipulated factor; lowercase letters indicate a measured factor. Letters J, K, and L refer to the number of
levels of factors A, B, and C, respectively.
OLEJNIK AND ALGINA
442
Page 10
smaller than the eta effect sizes computed earlier with
the same data.
If a design contains random effects, Equation 6 can
be used but the formulas for computing the variance
components would be different than when all effects
are fixed. Dodd and Schultz (1973) showed how to
compute variance components when one or more of
the factors are random.
Analysis of Covariance
If analysis of covariance is used and omega squared
is the preferred effect size, Equation 7 can be used
with MSs/Cellscalculated as it is in the corresponding
design without any covariates. That is,
MSs?Cells=??nCell− 1?SCell
??nCell− 1?
where summation is over all cells of the design,
nCellis the sample size in a cell, and S2
adjusted variance in a cell. Alternatively, MSs/Cellscan
be obtained by pooling the sum of squares for the
covariates and the sum of squares error and dividing
by the pooled degrees of freedom for these sources.
That is, MSs/Cells? (SSCovariate+ SSError)/(N − J). As
noted in the discussion following Equation 7, MSError
would be the within-cell variance adjusted by the co-
variates.
The formulas for omega squared in Table 2 can be
used for both two- or three-factor designs with one or
more covariates. For three-factor designs with any
combination of manipulated and measured factors, the
formulas in Table 2 can be used noting that MSs/Cells
is computed using Equation 8. For two-factor designs,
the formulas in Table 2 are also appropriate with c
representing the covariate, a measured factor.
2
,
(8)
Cellis the un-
Repeated Measures Design
In repeated measures designs, subjects are crossed
with the repeated measures factors. Equation 6 can be
used to calculate generalized omega squared for a
repeated measures design.1The quantity ? ˆ2can be
estimated by the ratio of the pooled sum of squares for
all sources of variance involving the subjects factor to
the pooled degrees of freedom for these sources of
variance. For example, in a single-factor repeated
measures design having J levels, ? ˆ2? (SSs+ SSsA)/
(N − J). The value of N is the total number of obser-
vations in the study N ? Jn, where n is the number of
individuals in the study. The estimate ? ˆ2is the aver-
age of the variances in the J levels of the repeated
measures factor (see Kirk, 1995, p. 254), so Equation
8 can also be used to calculate ? ˆ2. For a two-factor
repeated measures design ? ˆ2? (SSs+ SSsA+ SsB+
SSsAB)/(N − JK). Equation 7 is more convenient to use
for calculating generalized omega squared. The quan-
tity MSs/Cellsis ? ˆ2and can be computed as just de-
scribed. When Equation 7 is applied in a repeated
measures design, MSErrorin SSEffect− dfEffect× MSError
is the mean square for the interaction of subjects and
the effect. For example, in a two-way repeated mea-
sures design, MSErrorfor the AB effect is MSsAB.
Typically, the terms involving Meas can be deleted
because repeated measures factors are typically not
measured factors.
Maxwell and Delaney (2000, p. 497) provided data
for a 2 × 3 single-group repeated measures design.
Partial omega squared for the three-level factor equals
[289,920 − 2(3,560)]/[289,920 + (60 − 2)3,560] ?
.570. Generalized omega squared can be computed
using the formula in Table 2 for effect ? and design
ABc where subjects is the c factor:
? ˆG
2=
289,920 − 2?3,560?
1,133,940 − 285,660 − 105,120
+ ??3??2? − 2?8,393.33
= .364.
1For a repeated measures design, assuming as typically
would be true that the repeated measures factors are ma-
nipulated, the parameter defined in Equation 2 is
?Effect
2
? × ?Effect
2
+?
?
??
2,
where ? ranges over all sources of variance that involve
subjects. Thus in a one-factor repeated measures design, the
denominator would include ?2
eter would equal
e, ?2
sA, and ?2
s, and the param-
?A
2+ ?s
2
?A
2+ ?e
2+ ?sA
2.
It can be shown that when Equation 6 is applied in a re-
peated measures design, the denominator underestimates
correct denominator. For example, for a one-factor repeated
measures design, the expected value of the denominator of
Equation 6 is
2+?
?A
2+ ?e
2+ ?s
J − 1
J??sA
2.
An alternative to Equation 6, on the basis of results in Dodd
and Schultz (1973) is
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
443
Page 11
Mixed Design
Equation 6 or 7 can be used to calculate generalized
omega squared for a mixed design. In Equation 6, ? ˆ2
is computed by pooling all sources of variance involv-
ing subjects. For example, in a design with one be-
tween-subjects factor and one repeated-measures fac-
tor, ? ˆ2? (SSs/A+ SSBs/A)/(dfs/A+ dfBs/A) ? MSs/Cells.
This formula for ? ˆ2is equivalent to pooling the vari-
ance over the cells formed by combinations of the
between-subjects and repeated measures factors. As
an example, consider a design in which factors A and
B are both manipulated. Because A and B are both
manipulated
?
Meas
?SSMeas− dfMeas× MSM_Error? = 0.
Then Equation 7 becomes
? ˆG
2=
SSEffect− dfEffect× MSError
??SSEffect− dfEffect× MSError? + N × MSs?Cells.
If the A factor is of interest, we have
? ˆG
2=
SSA− dfAMSs?A
?SSA− dfAMSs?A? + N × MSs?Cells,
where N is the total number of observations in the data
set.
Tables 3 and 4 provide the formulas derived from
Equation 7 for computing generalized omega squared
for selected between-subject and repeated measures
factors. In these tables N˜is the number of individuals
in the study.
Kirk (1995, p. 527) provided an ANOVA summary
table for a 2 × 2 × 4 mixed-model design with the last
factor being within-subjects. Assuming an AcB de-
sign, partial omega squared for factor A would equal
[3.125 − 1(1.812)]/[3.125 + (8 − 1)1.812] ? .083. By
contrast,
? ˆG
2=
3.125 − 1?1.812?
235.5 − 194.5 − 19.375
+ 1.812 + 2?2 − 1?0.396
= .054.
Randomized Block Design
Kirk (1995, pp. 293–298) distinguished between a
randomized block design (RBD) and a generalized
randomized block design (GRBD). In an RBD the
? ˆEffect
2
? × ? ˆEffect
2
+?
?
? ˆ?
2+?
?
??
NMS?
,
where ??is the product of the number of levels of the
factors defining mean square for the ? effect. Thus for
MSsA, ??? nJ. It can be shown that when the preceding
equation is applied in a repeated measures design, the de-
nominator overestimates the correct denominator. For ex-
ample, for a one-factor repeated measures design the ex-
pected value of the denominator of equation would be
2+?
?A
J + 1
J??e
2+ ?s
2+ ?sA
2.
We did not select Equation 6 because we believe it is the
better way to estimate the denominator of generalized
omega squared. Rather, we selected it because it can be
applied to all designs considered in this article. The alter-
native to Equation 6 does not apply to designs with covariates.
Table 3
Selected Formulas for Generalized Omega Squared Statistics for Between-Subjects Factors in Mixed-Model Designs
SourceDesign
? ˆ2
G
?
AB
aB
ACB
aCB
AcB
acB
[SSA− dfA× MSs/A]/[SSA+ (N˜− dfA× MSs/A+ N˜(K − 1) × MSBs/A]
[SSa− dfa× MSs/a]/[SST− SSB+ MSs/a+ (K − 1) × MSBs/a]
[SSA− dfA× MSs/AC]/[SSA+ (N˜− dfA) × MSs/AC+ N˜(K − 1)MSBs/AC]
[SSa− dfa× MSs/aC]/[SST− SSB− SSC− SSBC+ L × MSs/aC+ L(K − 1) × MSBs/AC]
[SSA− dfA× MSs/ac]/[SST− SSB− SSAB+ MSs/ac+ J(K − 1) × MSBs/Ac]
[SSa− dfa× MSs/ac]/[SST− SSB+ MSs/ac+ (K − 1) × MSBs/ac]
[SSAC− dfAC× MSs/AC]/[SSAC+ (N˜− dfAC) × MSs/AC+ N˜(K − 1) × MSBs/AC]
[SSaC− dfaC× MSs/aC]/[SST− SSB− SSC− SSBC+ L × MSs/aC+ L(K − 1) × MSBs/aC]
[SSAc− dfAc× MSs/Ac]/[SST− SSA− SSB− SSAB+ J × MSs/Ac+ J(K − 1) × MSBs/Ac]
[SSac− dfac× MSs/ac]/[SST− SSB+ MSs/ac+ (K − 1) × MSBs/ac]
Note. A and C are between-subjects factors, and B is a repeated measures factor. N˜is the total number of individuals in the study.
??
ACB
aCB
AcB
acB
OLEJNIK AND ALGINA
444
Page 12
number of participants within a block is equal to the
number of treatments to which the participants will be
assigned. In a GRBD the number of participants in a
block is a multiple of the number of treatments to
which the participants will be assigned. For example,
suppose a control treatment and two treatments de-
signed to reduce fear of spiders are to be compared.
Prior to assignment to treatments, researchers obtain
measurements of fear of spiders on 60 participants. If
the participants are ranked, placed into blocks of
three, and assigned from within blocks to the treat-
ments, the design is an RBD. If a median split is used
so that participants are placed in two blocks of 30
each and then assigned to treatments, the design is a
GRBD. For an RBD, generalized omega squared can
be computed by using the procedures for a repeated
measures design, with blocks replacing participants. It
is also possible for blocks to be assigned to levels of
one or more between-subjects factors. Then general-
ized omega squared can be computed by using the
procedures for a mixed design, with blocks replacing
participants. For a GRBD, generalized omega squared
can be computed by using the procedures for a between-
subjects design. If in addition there are repeated mea-
sures factors, generalized omega squared can be com-
puted by using the procedures for a mixed design.
Discussion
Providing a measure of an effect size is now being
required by the editors of several prominent journals
in education and psychology to enhance the meaning-
fulness of the results when statistical hypothesis tests
are used. The current practice of reporting the partial
eta or partial omega squared statistic to estimate an
effect size for each factor in multifactor research de-
sign is often misleading and inappropriate. Routine
use of these measures of effect size does not appro-
priately consider the design features of a research
study when estimating the magnitude of an effect. In
the case of measures of association in an ANOVA
context, little attention has been given to the design
features, which can affect the size of the estimated
effects. Textbooks and computer output often imply
that the partial eta or partial omega squared statistics
are the appropriate effect-size measures to use in all
research designs that include more than a single ex-
planatory variable. This is an appropriate recommen-
dation if the researcher manipulates all of the factors
in the design. However, if a research design includes
one or more measured factors, partial eta or omega
squared statistics often provide an undesirable effect-
size measure. Researchers often include one or more
measured factors as covariates or as blocking vari-
ables to increase the statistical power of their hypoth-
esis tests. These factors increase the statistical power
by reducing the error variance. However, although
power of the analysis is increased, the interpretation
of the hypothesis test is unchanged. The reduced error
variance as a result of blocking or covarying, how-
ever, does affect the interpretation of effect-size mea-
Table 4
Selected Formulas for Generalized Omega Squared Statistics for Within-Subjects Factors in Mixed-Model Designs
SourceDesign
? ˆ2
G
?
AB
aB
ACB
aCB
AcB
acB
[SSB− dfB× MSBs/A]/[SSB+ N˜× MSs/A+ (N˜− 1)(K − 1) × MSBs/A]
[SSB− dfB× MSBs/A]/[SST+ MSs/A]
[SSB− dfB× MSBs/AC]/[SSB+ N˜× MSs/AC+ (N˜− 1)(K − 1) × MSBs/AC]
[SSB− dfB× MSBs/aC]/[SST− SSC− SSCB+ L × MSs/aC+ (L − 1) (K − 1) × MSBs/aC]
[SSB− dfB× MSBs/Ac]/[SST− SSA− SSAB+ J × MSs/ac+ (L − 1) (K − 1) × MSBs/Ac]
[SSB− dfB× MSBs/ac]/[SST+ MSs/ac]
[SSAB− dfAB× MSBs/A]/[SSAB+ N˜× MSs/A+ (N˜− dfA) (K − 1)× MSBBs/A]
[SSaB− dfaB× MSBs/a]/[SST+ N˜× MSs/a+ (K − 1) × MSBs/a]
[SSAB− dfAB× MSBs/AC]/[SSAB+ N˜× MSs/AC+ (N˜− dfA) (K − 1) × MSBs/AC]
[SSaB− dfaB× MSBs/aC]/[SST− SSB− SSC− SSBC+ L × MSs/aC+ L(K − 1) × MSBs/aC]
[SSAB− dfAB× MSBs/Ac]/[SST− SSA− SSB+ J × MSs/Ac+ (K − 1) × MSBs/Ac]
[SSaB− dfaB× MSBs/ac]/[SST− SSB+ MSs/ac+ (K − 1) × MSBs/ac]
??
AB
aB
ACB
aCB
AcB
acB
???
ACB
aCB
AcB
acB
[SSACB− dfACB× MSBs/AC]/[SSACB+ Ñ × MSs/AC+ (Ñ − dfA)(K − 1) × MSBs/AC]
[SSaCB− dfaCB× MSBs/aC]/[SST− SSB− SSC− SSBC+ L × MSs/aC+ L(K − 1) × MSBs/aC]
[SSAcB− dfAcB× MSBs/Ac]/[SST− SSA− SSB+ J × MSs/Ac+ (K − 1) × MSBs/Ac]
[SSacB− dfacB× MSBs/ac]/[SST− SSB− MSs/ac+ (K − 1) × MSBs/ac]
Note. A and C are between-subjects factors, and B is a repeated measures factor. N˜is the total number of individuals in the study.
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
445
Page 13
sures. Reducing the error variance restricts the popu-
lation for whom the effect is being estimated.
Partial eta and omega squared also can provide indi-
ces that are inappropriately compared to Cohen’s (1988)
guidelines for defining the small, medium, and large
effects. In suggesting the guidelines, Cohen cited studies
that compared unrestricted populations (e.g., differences
between men and women). Cohen used differences be-
tween men and women on several variables to help de-
fine .2? as a small difference. He did not restrict these
populations, for example by using age as a covariate or
a blocking variable. To do so would have changed the
magnitude of the difference that today is considered
small. If researchers continue to use the same guidelines
to define small, medium, and large effects across a va-
riety of research designs, then it seems reasonable to
expect researchers to provide effect-size estimates rela-
tive to the full range of the population rather than a
restricted population. Consequently, the use of partial
eta or partial omega squared in many situations dimin-
ishes the usefulness of the new editorial policy.
The generalized eta and omega squared statistics
have two major advantages. First, these statistics pro-
vide measures of effect size that are comparable
across a wide variety of research designs that are
popular in education and psychology. Second, these
effect-size measures provide indices of effect that are
consistent with Cohen’s (1988) guidelines for defin-
ing the magnitude of the effect. Cohen pointed out
three decades ago that design considerations must be
used when computing an effect-size measure. Cur-
rently, most researchers who choose to report an ef-
fect size as the proportion of variance explained have
ignored Cohen’s caution. Using the procedures out-
lined in the present article, researchers can correct this
omission and provide more comparable effect-size
measures. A final cautionary note should be restated.
Generalized eta and omega squared statistics can pro-
vide comparable effect-size measures for studies that
use different outcome measures and different research
designs when studying a common target population. If
the target populations studied differ substantially with
respect to their variances, effect-size measures would
not be comparable. Professional judgment must be
used when determining when it is reasonable to com-
pare effect sizes across a series of studies.
References
Carroll, R. M., & Nordholm, L. A. (1975). Sampling char-
acteristics of Kelly’s ?2and Hays’ ?2. Educational and
Psychological Measurement, 35, 541–554.
Charter, R. A. (1982). Practical formulas of strength of as-
sociations measures. Educational and Psychological
Measurements, 42, 969–974.
Cohen, J. (1973). Eta-squared and partial eta-squared in
fixed factor ANOVA designs. Educational and Psycho-
logical Measurement, 33, 107–112.
Cohen, J. (1988). Statistical power analysis for the behav-
ioral sciences (2nd ed.). New York: Academic Press.
Dodd, D. H., & Schultz, R. F. (1973). Computational pro-
cedures for estimating magnitude of effects for some
analysis of variance designs. Psychological Bulletin, 79,
391–395.
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J.
(1996). Meta-analysis of experiments with matched
groups or repeated measures designs. Psychological
Methods, 1, 170–177
Dwyer, J. H. (1974). Analysis of variance and the magni-
tude of effects: A general approach. Psychological Bul-
letin, 81, 731–737.
Fern, F. E., & Monroe, K. L. B. (1996). Effect-size esti-
mates: Issues and problems in interpretation. Journal of
Consumer Research, 23, 89–105.
Fleiss, J. L. (1969). Estimating the magnitude of experimen-
tal effects. Psychological Bulletin, 72, 273–276.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-
analysis in social research. Beverly Hills, CA: Sage.
Halderson, J. S., & Glasnapp, D. R. (1972). Generalized
rules for calculating the magnitude of an effect in facto-
rial and repeated measures ANOVA designs. American
Educational Research Journal, 9, 301–310.
Hays, W. L. (1963). Statistics for psychologists. New York:
Holt, Rinehart & Winston.
Keppel, G. (1991). Design and analysis: A researcher’s
handbook. Englewood Cliffs, NJ: Prentice Hall.
Keren, G., & Lewis, C. (1979). Partial omega squared for
ANOVA designs. Educational and Psychological Mea-
surement, 39, 119–128.
Keselman, H. (1975). A Monte Carlo investigation of three
estimates of treatment magnitude: Epsilon squared, eta
squared, and omega squared. Canadian Psychological
Review, 16, 44–48.
Kirk, R. E. (1995). Experimental design: Procedures for the
behavioral sciences (3rd ed.). Belmont, CA: Wadsworth.
Kirk, R. E. (1996). Practical significance: A concept whose
time has come. Educational and Psychological Measure-
ment, 56, 746–759.
Maxwell, S. E., Camp, C. J., & Arvey, R. D. (1981). Mea-
sures of strength of association: A comparative examina-
tion. Journal of Applied Psychology, 66, 525–534.
Maxwell, S. E., & Delaney, H. D. (2000). Designing experi-
OLEJNIK AND ALGINA
446
Page 14
ments and analyzing data. A model comparison perspec-
tive. Belmont, CA: Wadsworth.
Morris, S. B., & DeShon, R. P. (1997). Correcting effect
sizes computed from factorial analysis of variance for use
in meta-analysis. Psychological Methods, 2, 192–199.
O’Grady, K. E. (1982). Measures of explained variance:
Cautions and limitations. Psychological Bulletin, 92,
766–777.
Olejnik, S., & Algina, J. (2000). Measures of effect size for
comparative studies: Applications, interpretations and
limitations. Contemporary Educational Psychology, 25,
241–286.
Peters, C. C., & Van Voorhis, W. R. (1940). Statistical pro-
cedures and their mathematical bases. New York: Mc-
Graw-Hill.
Richardson, J. T. E. (1996). Measures of effect size. Behav-
ioral Research Methods, Instruments, & Computers, 28,
12–22.
Searle, S. R. (1971). Linear models. New York: Wiley.
Stevens, J. (1999). Intermediate statistics: A modern ap-
proach (2nd ed.). Mahwah, NJ: Erlbaum.
Vaughan, G. M., & Corballis, M. C. (1969). Beyond tests of
significance: Estimating strength of effects in selected
ANOVA designs. Psychological Bulletin, 72, 204–213.
Wampold, B. E., & Serlin, R. C. (2000). The consequences
of ignoring a nested factor on measures of effect size in
analysis of variance. Psychological Methods, 425–433.
Wilkinson, L., & the Task Force on Statistical Inference.
(1999). Statistical methods in psychology journals:
Guidelines and explanations. American Psychologist, 54,
594–604.
Winkler, R. L., & Hays, W. L. (1975). Statistics: Probabil-
ity, inference, and decision (2nd ed.). New York: Holt.
Yates, F. (1934). The analysis of multiple classifications
with unequal numbers in the different classes. Journal of
the American Statistical Association, 29, 51–66.
Received September 17, 2001
Revision received April 11, 2003
Accepted June 23, 2003 I
SPECIAL SECTION: ETA AND OMEGA SQUARED STATISTICS
447
View other sources
Hide other sources
- Available from James Algina · May 19, 2014
- Available from umn.edu