Content uploaded by Gerben Mulder
Author content
All content in this area was uploaded by Gerben Mulder on Apr 14, 2019
Content may be subject to copyright.
Sample Size Planning for Precise Contrast
Estimates: An Introduction and Tutorial for One
and Two Way Designs
Gerben Mulder
Department of Language, Literature, and Communication
VU Amsterdam
April 14, 2019
Contrast analysis provides a straightforward way of answering focused re-
search questions (Rosenthal, Rosnow, & Rubin, 2000). When combined with an
estimation approach the results of a contrast analysis are particularly informative
(Wiens & Nilsson, 2017), whether the researcher uses a Bayesian approach (e.g.
Kruschke, 2015) or a frequentist approach (Calin-Jageman & Cumming, 2019)
to estimation. Indeed, the results of a contrast analysis using the estimation
approach give the researcher both a point estimate and an interval estimate,
revealing plausible candidate values for the population and (thereby) input that
can be used for the evaluation of the theoretical and practical importance of the
results.
Of course, the informativeness of the results depends to a large extent on
the precision with which one estimates the contrast. That is, a wide interval
estimate, with a large range of plausible values, expresses more uncertainty
than a relatively narrow interval estimate, which makes it harder to assess the
theoretical and practical importance of the results. For this reason, it is advisable
to design a study with as large a precision as is necessary for the purposes of
the study, or at least to design a study with as high a precision as is practically
feasible.
The goal of the present contribution is to introduce sample size planning
for precision for contrast estimates. The general background to the approach
advocated here is Accuracy in Parameter Estimation (AIPE) (Kelley, 2013;
Maxwell, Kelley, & Rausch, 2008; see also Cumming, 2012, for a basic introduc-
tion). In essence, AIPE is about optimizing the design of studies focusing on the
estimation of model parameters in terms of the width of the confidence interval
of the estimate. In the context of contrast estimates, then, the focus of sample
size planning for precision is on determing an optimal sample size for estimating
contrast values with as narrow a confidence interval width as is necessary or
feasible.
This tutorial focuses specifically on contrast estimates in common experimen-
tal designs. It will consider one way within and between subjects design and
two way within, between, and mixed designs (split-plot designs). The tutorial
includes R-functions for sample size planning for precise contrast estimates in
these designs. Alternatively, planning for precision can be done with the Shiny
1
application that accompanies the article1
The structure of the tutorial is as follows. The tutorial starts with introducing
the central concepts underlying the planning approach by making reference to
a one factor between participants design. This is followed by examples of the
application of the R-functions for sample size planning in single factor and
factorial between participants design. The remainder is structured in the same
way, first the mathematical formulas are presented and then these expressions
are applied to sample size planning for precision.
Central Concepts and Between Subject Designs
It is worthwile to introduce some of the basic concepts with a between subjects
design with a single fixed factor with three levels and a continuous response
variable X. The purpose of the study is to estimate the contrast between the
population mean of treatment level 1 (
µ1
) and the mean of the other two
treatment levels (
µ2
+
µ3
)
/
2. These population means are the expected values
of three independent random variables
X1
,
X2
, and
X3
, one for each treatment
condition, and we assume them to be normally distributed with equal variance
σ2
. The assumptions of normality and equal variances (and covariances, for the
within and mixed designs) will be made throughout, and for that reason will not
be made explicit everytime).
The to be estimated contrast
ψ
can be formulated as a linear combination
of the population means (see for introduction to contrast estimation Maxwell,
Delaney, & Kelley, 2017; Rosenthal, Rosnow, & Rubin, 2000; or, for a gentle more
practically oriented introduction: Haans, 2018):
ψ
=
µΛT
, where in the present
example,
µ
= [
µ1, µ2, µ3
], a vector of population means, and
Λ
= [
λ1, λ2, λ3
]a
vector of contrast or lambda weights. Thus, ψ=µΛT=µ1λ1+µ2λ2+µ2λ3.
For example, if
µ
= [2
,
1
,
1] and we want to compare the first population mean
to the average of the other two, we could use contrast weights
Λ
= [1
,−1
2,−1
2
].
This would give
ψ
= 2
∗
1 + 1
∗ −1
2
+ 1
∗ −1
2
= 1. By the same token, if
only the second and third expected values were to be contrasted, we could use
Λ
= [0
,
1
,−
1], which would give
ψ2
= [2
,
1
,
1][0
,
1
,−
1]
T
= 2
∗
0+1
∗
1+1
∗−
1 = 0.
The estimated value of the contrast between population means is simply the
corresponding linear combination of the sample means. Thus, to obtain the
point estimate, we multiply the vector of sample means
¯
X
by the transpose of
the contrast weights vector: ˆ
ψ=¯
XΛT.
As our expression of the uncertainty of the point estimate, we will use the
95% confidence interval (CI). Using terminology of Cumming (2012), the 95%
CI of the contrast estimate can be obtained by adding to and subtracting from
the point estimate the (estimated) Margin or Error (MoE). In other words, MoE
is the half-width of the 95% CI.
The expected value of MoE equals
MoE =t.975,df σˆ
ψ,(1)
where
σˆ
ψ
=
qE[( ˆ
ψ−ψ)2]
, the standard error of the contrast estimate, and
t.975,df the .975 quantile of a central t-distribution with df degrees of freedom.
1
The application is available here: https://gmulder.shinyapps.io/PlanningFactorialContrasts.
The app is restricted to planning for precision in one and two-way factorial designs.
2
The standard error of the contrast estimate is the square root of the variance
of the contrast estimate
σ2
ˆ
ψ
=
E
[(
ˆ
ψ−ψ
)
2
]. The value of the variance is a function
of both the sample size and the contrast weights. In particular, the variance of
the contrast estimate is equal to the sum of squared contrast weights multiplied
by the within-condition variance
σ2
divided by the per condition sample size (n):
σ2
ˆ
ψ=ΛΛTσ2
n= (Xλ2
i)σ2
n=1
n(Xλ2
i)σ2.(2)
If we combine (1) and (2) we get an expression for expected MoE
MoE =t.975,df r1
n(Xλ2
i)σ2.(3)
From (1) and (3) it is clear that we need to specify the degrees of freedom.
For the one-way between participants design these are the degrees of freedom
that are used to estimate the within condition variances, i.e. with a levels of the
treatment factor the df equal
df
=
a
(
n−
1) =
N−a
. The degrees of freedom
for the other designs are given in the Appendix.
The idea of sample size planning for precision is already quite apparent from
(3). We determine a value for MoE and solve (3) for
n
. There are, however, a
few complications. The first complication is that in order to use (3) we need
to specify the value of the within condition variance, whose value is generally
unknown. This problem can be solved quite easily: in stead of working with
an absolute value for MoE, we express MoE as a number of within condition
standard deviations (Cumming, 2012):
f=MoE/σ =t.975,df r1
n(Xλ2
i)√σ2
√σ2=t.975,df r1
nXλ2
i.(4)
The second complication is that the sample size enters (4) twice, once as
the denominator of the fraction involved in the expression for the standardized
standard error and once in the degrees of freedom
a
(
n−
1), as Cumming (2012)
explains, this makes it necessary to use an iterative routine in order to solve for
n in (4). With software like R, however, this is easily implemented.
A final complication is that if we set f in (4) and solve for n using the
iterative routine, we get the sample size necessary to obtain our target MoE on
average. This means, that in approximately half the studies we undertake with
the required sample size, we will get a Margin of Error larger than the value
we want. This is where we will need the concept of assurance (Cumming, 2012;
Kelley, 2013): we will plan for a sample size that will give us a MoE that will
not exceed our target MoE with probability
γ
. We will make use of the fact that
the sampling distribution of the within condition variance is a scaled chi-squared
distribution. That is,
ˆσ2∼σ2χ2
df /df,
which means that if we standardize MoE by the within condition standard
deviation, that the
γ
-quantile of the sampling distribution of estimated MoE is
fγ=t.975,df r1
n(Xλ2
i)χ2
γ,df /df. (5)
3
All told, then, we set a value for f and assurance
γ
, meaning that we specify
the
γ
-quantile of the sampling distribution of standardized MoE, we specify
the vector of lambda weights, and we use these specifications with an iterative
routine to solve for n in (5). This will give us the required per condition sample
size necessary to obtain estimated standardized MoE no larger than our target
MoE with
γ
% assurance. Note that if we fill in the sample size in (4), we
get a value for expected MoE. Sample size planning for the other designs uses
variations of (5), see Appendix.
Planning for Contrasts in Between Participants Designs
Single factor designs
Let’s return to our example. We have a three condition between subjects design
and we want to estimate the difference between the population mean of the
first condition and the average of the expected values of the other two random
variables. We specify the vector of contrast weights as follows
Λ
= [1
,−1
2,−1
2
]
and let us suppose that we want to plan for a target MoE of f = .50 and assurance
γ
=
.
80. The R-function sampleSizeBetween provided in the Appendix can do
the job for us. Alternatively, use the Shiny application for sample size planning
that accompanies this article.
lambda =c(1,-1/2,-1/2)
f=.50
gamma =.80
nlevs =3
sampleSizeBetween(lambda, f, gamma, nlevs)
## [1] 27
The planning function gives us a value of n = 27 participants per treatment
level, so in total
N
= 27
∗
3 = 81. If we use n = 27 in (4), expected MoE equals
f = 0.4692.
We can use a simple simulation to see whether target MoE will be below f
= .50 in 80% of the replication experiments. This can be done as follows, note,
however, that since the random seed is not set, running this code will lead to a
somewhat different result than the one reported here.
nReps =10000
n=27
df =3*(n -1)
lambda =c(1,-1/2,-1/2)
sum.sq.w =sum(lambda^2)
MoEs =rep(0, nReps)
for (idx in 1:nReps) {
a1 =rnorm(n)
4
a2 =rnorm(n)
a3 =rnorm(n)
v=c(var(a1), var(a2), var(a3))
v=v%*% rep(n -1,3)/df
MoEs[idx] =qt(.975, df)*sqrt(v/n*sum.sq.w)
}
quantile(MoEs, .80)
## 80%
## 0.4988301
Another possibility is to make use of data that confirms exactly to our
specifications. Below, the three condition means are all equal (zero), and the
within condition variances are equal to one. The first contrast (X1) estimates
the contrast with weights [1
,−1
2,−1
2
]. The result of the estimate is zero, and
the lower and upper limits of the 95% confidence intervals should therefore
correspond to minus MoE and plus MoE, respectively. Note that because the
within condition variances equals 1, the Margin of Error of the estimate is equal
to the value of
f
= 0
.
4692 above. Likewise, the value of MoE for the second
contrast (X2) should be equal to MoE = 1.9908 ∗q1
27 ∗2=0.5418
library(MASS) #needed for ginv function below
n=27
# random variables scaled to have
# mean = 0 and var = 1
a1 =scale(rnorm(n))
a2 =scale(rnorm(n))
a3 =scale(rnorm(n))
# response variable:
Y=c(a1, a2, a3)
# independent treatment factor:
X=factor(c(rep(1, n), rep(2, n), rep(3, n)))
#define full set of orthogonal (helmert) contrasts
contrasts(X) =ginv(rbind(c(1,-.5,-.5), c(0,1,-1)))
#estimate model:
myMod =lm(Y ~X)
#obtain intervals
confint(myMod)
## 2.5 % 97.5 %
## (Intercept) -0.2212052 0.2212052
## X1 -0.4692472 0.4692472
## X2 -0.5418399 0.5418399
5
Factorial designs
The main purpose of using a factorial design is that these designs allow for the
studying interactions between factors. Studying interactions should take center
stage, and, therefore, from the estimation perspective the main focus of the
analysis is estimating interaction contrasts and their confidence intervals. Of
course, that is not to say that contrasts of the marginal means (main effect
contrasts) should not be considered, or to return to the topic of this tutorial,
sample size planning for obtaining estimates of these contrasts should be ignored.
Planning for precise contrast estimates in factorial between participants
designs can easily be done with the R-function provided above. There are just a
few things to consider. The first is that in specifying the weights vector, it is
convenient to conceptualize the design as a single factor design with number
of levels equal to the product of the number of levels of the factors involved.
Restricting the discussion to the two-way factorial designs, suppose that there
are two treatment factors each with two levels, we would conceptualize the design
not as a 2 x 2 factorial design, but as a 1 x 9 one-way design. The second thing
to consider is that if we use the sample size planning function provide above,
the nlevs-argument should be set equal to the product of the two factors.
Thus, if we were to plan for
f.80
=
.
50 for a contrast of marginal means in a
two-by-two design, the weights vector for the main effect of one of the treatments
could be set to
Λ
= [
1
2,1
2,−1
2,−1
2
]and the number of levels to 4. Using this
input in the sample size planning function, gives the following result.
lambda =c(1/2,1/2,-1/2,-1/2)
f=.50
gamma =.80
nlevs =4
sampleSizeBetween(lambda, f, gamma, nlevs)
## [1] 19
So, we need
n
= 19 observations per treatment level, making for a total of
N= 4 ∗19 = 76 observations.
If we want to plan for an interaction contrast, the thing to consider is that for
mean comparisons the absolute value of the sum of the contrast weights equals
4.0 (Kline, 2013), whereas for main effect contrasts this sum should equal 2.0.
Thus, if we plan for
f.80
=
.
50 for an interaction contrast in a 2 x 3 design, we
should use Λ= [1,−1
2,−1
2,−1,1
2,1
2], with the number of levels set to 2∗3=6:
lambda =c(1,-1/2,-1/2,-1,1/2,1/2)
f=.50
gamma =.80
nlevs =6
sampleSizeBetween(lambda, f, gamma, nlevs)
## [1] 50
Thus, according to the planning results, we need
n
= 50 participants per
level, makeing for a total of N= 300.
6
Within Participants Designs
For contrast analysis in within participants designs a choice has to be made
between two approaches (Maxwell, Delaney, & Kelley, 2017). The first approach
is to fit a mixed model, with treatment(s) as fixed factor and participant as
random factor (here, we consider designs with one observation per treatment
condition per participant only, so we model only two sources of random variance,
i.e. variance due to participant and due to error), and use the results of fitting
and all the underlying statistical assumptions to estimate the margin of error of
each contrast. If all assumptions of the model apply, the precision of the estimate
will be (somewhat) higher, i.e. MOE will have a (somewhat) smaller value and
the confidence interval will be (somewhat) narrower than in the alternative
approach. However, if the assumptions do not apply, the estimated precision
cannot be trusted (Maxwell, Delaney, & Kelley, 2017), so it may not be worthwile
to use the mixed model approach. The alternative is to transform the scores
for each participant to a contrast score, and to subsequently estimate the mean
contrast score over participants. This requires less assumptions than the mixed
model approach, and is shown to be superior to the mixed model approach when
the assumptions of the latter approach do not obtain (Maxwell, Delaney, &
Kelley, 2017).
In the planning approach described in this article, it is assumed that the
mixed model assumptions hold, but that the contrast analysis itself uses the
alternative transformed scores approach. The assumptions made for the within
participant designs (and for the within factors of the mixed design) is that the
random variables are multivariate normal variables with equal variances (
σ2
)
and covariances. The covariances are equal to
ρσ2
, where
ρ
is Pearson’s product
moment correlation.
These assumptions lead to the following expression of the Margin of Error of
a contrast estimate:
MoE =t.975,df r1
n(Xλ2
i)(σ2−ρσ2).(6)
It is easy to demonstrate that the variance of the contrast scores equals
Pλ2
i
(
σ2−ρσ2
). Take for instance the variance of the contrast score based
on subtracting one random variable from the other, say
X1−X2
. We have
Λ
= [1
,−
1], and
var
(
X1−X2
) =
var
(
X1
) +
var
(
X2
)
−
2
cov
(
X1, X2
) = 2
σ2−
2
ρσ2
= 2(
σ2−ρσ2
) =
Pλ2
i
(
σ2−ρσ2
)
.
Or, with three conditions, with
Λ
=
[1
,−1
2,−1
2
], we have
var
(
X1−1
2X2−1
2X3
) =
var
(
X1
)+
var
(
1
2X2
)+
var
(
1
2X3
)
−
2
cov
(
X1,1
2X2
)
−
2(
cov
(
X1,1
2X3
)
−
2
cov
(
1
2X2,1
2X3
) =
σ2
+
1
4σ2
+
1
4σ2
+
2
2ρσ2
+
2
2ρσ2−2
4ρσ2= 1 1
2σ2−11
2ρσ2=Pλ2
i(σ2−ρσ2).
If MoE is standardized by the within condition standard deviation, and we
take into account the assurance, we get:
fγ=t.975,df r1
n(Xλ2
i)(1 −ρ)χ2
γ,df /df. (7)
We will use (7) for sample size planning, with degrees of freedom equal to
n−
1(the number of participants minus 1), for the completely within participants
designs, and
df
=
an −
1for the mixed design, with
a
being the number of levels
7
of the between-participants factor. In order to use (7) we need to assume that
ρ <
1. Note that contrary to sample size planning for between participants
designs we will have to specify a value for a population parameter, i.e. the
population correlation between the treatment conditions.
Planning for Contrasts in Within Participants Designs
Single Factor Designs
The R-function sampleSizeWithin provided in the Appendix can be used to
plan for the sample size in whithin participants designs, alternatively, the Shiny
application can be used (see footnote 1). The R-function provided can be applied
for planning in both single factor and multi-factor designs. The main difference
between these designs in how the function is used, is in the way the contrast
weights are specified, as will be demonstrated momentarily.
Let us consider planning for a target MoE of f = .50 with 80% assurance,
in a single factor within participants design with three conditions. We use the
following additional input
Λ
= [1
,−1
2,−1
2
]and
ρ
=
.
60. Here, the value of
ρ
is arbitrarily set to equal
.
60, in practice one would either use a value that is
informed by previous research, or plan for a range of plausible values for the
correlation.
f=.50
gamma =.80
lambda =c(1,-1/2,-1/2)
rho =.60
sampleSizeWithin(lambda, f, gamma, rho)
## [1] 15
If 15 participants are selected for the study, the expected standardized MoE
will be equal to
t.975,14q1
15 1.5(1 −.60)
= 2
.
1448
∗
0
.
2000 = 0
.
4290. Let us
generate an ideal dataset for which the assumptions about the variance
σ2
= 1
and covariance
ρσ2
= 0
.
60 are true. A contrast analysis of this ideal data will
show that the value for expected MoE is correct.
# library(MASS)
n=15
lambda =c(1,-1/2,-1/2)
# specify means
mu =c(0,0,0)
# variance covariance matrix
Sigma =matrix(c(1,.6,.6,.6,1,.6,.6,.6,1), 3,3)
# generate multivariate data
theData =mvrnorm(n, mu, Sigma, empirical=TRUE)
# calculate contrast score
contrastScore =theData %*% lambda
# t.test, upper limit of CI equals expected standardized MoE
t.test(contrastScore)
8
##
## One Sample t-test
##
## data: contrastScore
## t = 4.349e-16, df = 14, p-value = 1
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
## -0.4289573 0.4289573
## sample estimates:
## mean of x
## 8.698012e-17
Factorial Within Participants Designs
Planning in a factorial within participants design can be done with the same
sample planning for precision function as we used above for the single factor
within participants design. The main difference is that in the specification
of the contrast weights we need to specify a number of weights equal to the
product of the number of levels of each factor. Let us consider a design with
two within participant factors, one with two levels and one with three. The
number of lambda weights is therefore equal to 6. For contrast analysis of the
main treatment effects, the sum of the absolute weights should equal 2.0 and for
interaction contrasts that sum should be 4.0 (Kline, 2013).
The following example with R-code shows a convenient way of obtaining
sample sizes for all of a full set of orthogonal contrasts for the 3 by 2 within
participants design. This is specifically convenient if the value for
fγ
is the same
for all contrasts. If the value differs from contrast to contrast, use the planning
function on a per contrast basis. We will plan for
f.95
=
.
30 and arbitrarily set
the correlation to equal ρ=.75
# this function can be used to make
# sure that the sum of absolute values
# of the lambda weights sum to 2 (default)
forceAbsoluteSum =function(lambda,sum.value=2) {
lambda *sum.value /sum(abs(lambda))
}
levelsa =3
levelsb =2
# note that for the planning functions
# the order of the weights is irrelevant,
# only the sum of squared weights needs to be correct.
A1 =c(1,-1/2,-1/2)
A2 =c(0,1,-1)
B1 =c(1,-1)
9
# start with interaction contrasts
interA1B1 =as.vector(outer(A1, B1))
interA2B1 =as.vector(outer(A2, B1))
# the absolute values of the interaction lambda weights
# sum to 4 already, so the following is not necessary
# in this case.
# just to illustrate
interA1B1 =forceAbsoluteSum(interA1B1, 4)
# make sure that that each weights factor
# contains 6 elements and that absolute
# values of weights sum to two
A1 =rep(A1, levelsb)
A1 =forceAbsoluteSum(A1)
A2 =rep(A2, levelsb)
A2 =forceAbsoluteSum(A2)
B1 =rep(B1, levelsa)
B1 =forceAbsoluteSum(B1)
# make matrix
contrasts =cbind(A1, A2, B1, interA1B1, interA2B1)
apply(contrasts, 2, sampleSizeWithin,
f=.30,gamma =.95,rho =.75)
## A1 A2 B1 interA1B1 interA2B1
## 16 20 15 47 59
The planning results show that with 59 participants (the second interaction
contrast), the margin of error of each individual contrast will not exceed more
than .30 standard deviations, with per contrast assurance at least equal to 95%.
These samples sizes are of course indicative to the extent that the assumptions
made for planning are reasonable, including the assumption that the population
correlation equals ρ=.70.
Factorial Mixed Designs
As we have seen, the sample size planning functions above can be used for
one and two-way between and within participants factorial designs. The only
difference between one and two way designs is that for planning in between
participants designs the number of levels is the product of the number of levels
of the factors involved, i.e. in a two-way factorial design, with
a
levels for the
first factor and
b
levels for the second factor, the number of levels equals
ab
. In
the case of mixed factorial designs, however, the situation is more complex.
Suppose we have a mixed factorial design with two factors. Treatment A is a
10
between participants factor with
a
= 2 levels, Treatment B a within participants
factor with
b
= 3 levels, and we suppose that for each combination of a level of
A and a level of B we have
n
observations (say one of each of
n
participants).
Let us also assume that each of the
ab
variables has equal variance
σ2
, and that
the covariance between the levels of Treatment B equals ρσ2.
Let us first consider estimating a contrast between the two population means
associated with the levels of Treatment A. The scores in each level of A are
the participants means of the
b
= 3 variables corresponding to Treatment B.
So, if we let
Xa.k
stand for the score of participant
k
in level
a
of Treatment
A averaged of the
b
levels of Treatment B, we know that the variance of the
Xa.k
must equal the variance of the mean of the
b
= 3 participant scores. Thus,
var
(
Xa.k
) =
var
([
Xa1k
+
Xa2k
+
Xa3k
)
/
3]) = (3
σ2
+6
ρσ2
)
/
9 =
σ2
(
b
+
b
(
b−
1)
ρ
)
/b2
.
We can use the latter in the expression of the (unstandardized) MoE of the
contrast between the condition means of Treatment A:
MoE =t.975,df r1
n(Xλ2
i)σ2(b+b(b−1)ρ)/b2.(8)
In standardized form, that is standardizing MoE by dividing by
σ
, and
taking into account the assurance, the expression for the margin of error of the
between-participants contrast becomes
fγ=t.975,df r1
n(Xλ2
i)(b+b(b−1)ρ)/b2χ2
γ,df /df, (9)
where the degrees of freedom equal
df
=
a
(
n−
1). Thus, with, for example,
a
= 2,
b
= 3,
n
= 10,
σ2
= 1
.
5,
ρ
=
.
60, and contrast weights for the between
factor set to
Λ
= [1
,−
1], the expected value of MoE is of the contrast equals
MoE =t.975,18 ∗q1
10 ∗2∗1.5∗(3 + 6 ∗.60)/9=0.9854.
For the within-participants factor B, we would transform the scores for each
of the
an
participants and use the resulting contrast scores to estimate the
expected value of the contrast. This means that we can simply use (7) for sample
size planning, with degrees of freedom equal to df =an −1.
An interaction contrast of the two factors A and B can be obtained by first
transforming the treatment B scores to contrast scores and subsequently estimate
a between participants contrast on the resulting scores. We will need to formulate
two contrast vectors, one for each factor. The expression for MoE is
MoE =t.975,df r1
n(Xλ2
a)(Xλ2
b)(σ2−ρσ2),(10)
wich degrees of freedom equal to
df
=
a
(
n−
1). If we standardize MoE and take
into account assurance, we get
fγ=t.975,df r1
n(Xλ2
a)(Xλ2
b)(1 −ρ)χ2
df /df. (11)
Planning for Precise Contrast estimates in Mixed Designs
The complexities of the mixed design call for three different planning functions,
one for the contrast analysis of the between participants factor, one for the
11
within participant factor, and one for the interaction of these factors. These
functions can only be used for designs with two factors .
Between Participant contrasts
Suppose we want our MoE to not exceed .25 standard deviations, with assurance
.90. In terms of the assumptions above, where we assumed that
σ2
= 1
.
5, this
value of f corresponds to a MoE of
.
25
∗√1.5
= 0
.
3062. Let us use (9) to plan for
a target MoE of
f.90
= 0
.
25. The following R-code can be used for that purpose.
lambda =c(1,-1)
f=0.25
gamma =.90
rho =.60
nlevsa =2
nlevsb =3
sampleSizeMixedBetween(lambda, f, gamma, rho,
nlevsa, nlevsb)
## [1] 103
The result of the planning function is that we need
n
= 103 participants per
between-participants condition, so,
N
= 206 in total. This will give an expected
MoE of MoE =t.975,204 ∗q1
103 ∗2∗1.5∗(3 + 6 ∗.60)/9=0.2882.
Within participant contrasts
For the within participants factor B, which has three levels, we will plan for two
Helmert contrasts. The first Helmert contrasts has
Λ1
= [1
,−1
2,−1
2
], and for the
second contrasts we specify
Λ2
= [0
,
1
,−
1]. We want to have 90% assurance per
contrast that MoE will not exceed .40 standard deviations, so we set
f.90
=
.
40.
Note that since the precision of the second contrasts will be lower (= higher
value for MoE) than for the first contrast, due to the fact that the sum of squared
weigths of the second contrast is larger than that of the first (and assuming
everthing else equal), so we only need to plan for the second contrast, since the
sample size needed to obtain our precision goal for the second contrast, will
necessarily be sufficient for the first contrast as well. We will again assume an
arbitrary value for the population correlation, and set
ρ
=
.
50. The following
code for R uses the sample size planning function provided in the Appendix.
lambda =c(0,1,-1)
f=.40
gamma =.90
rho =.50
nlevsa =2# number of levels of between factor
sampleSizeMixedWithin(lambda, f, gamma, rho, nlevsa)
## [1] 31
The results are that we need at least 31 participants. Of course, since these
31 participants will be randomly assigned to one of the two treatment conditions,
12
it is more convenient to select an even number of participants, the most obvious
candidate value being N= 32 or n= 16.
Interaction contrasts in the Mixed Design
Let us suppose that we want to plan for a target MoE of
f
=
.
40, with assurance
γ
=
.
80, for the estimate of a interaction contrast comparing the levels of
Treatment A with respect to the value of a Treatment B contrast in which the
first level of B is compared to the two other levels. For treatment B with have
the contrast weights vector
ΛB
= [1
,−1
2,−1
2
], and to compare the value of this
contrast between the levels of A we formulate a weights vector
ΛA
= [1
,−
1].
We will assume that the the value of the population correlation equals ρ=.50
We can use this input with the following R-code:
f=.40
gamma =.80
lambdaA =c(1,-1)
lambdaB =c(1,-.5,-.5)
rho =.50
nlevsa =2
nlevsb =3
sampleSizeMixedInt(lambdaA,
lambdaB, f, gamma, rho, nlevsa, nlevsb)
## [1] 42
According to the results of the planning function, then, we need
n
= 42
participants per level of the between participants factor, or 84 participants in
total. Let us check this sample size with a simple simulation study.
# library(MASS)
var =1
cov =0.5
#between-factor
nlevsa =2
#within-factor
nlevsb =3
weightsB =c(1,-.5,-.5)
# condition means
means =rep(0, nlevsb)
#variance-covariance-matrix
Sigma =matrix(c(1,.5,.5,.5,1,.5,.5,.5,1), 3,3)
# sample size per independent group equals 42
# total sample size equals 84
n=42
N=nlevsa *n
MoEs =NULL
# 10000 replications
13
nReps =10000
for (idx in 1:nReps) {
scores =mvrnorm(N, means, Sigma)
contrast =scores %*% weightsB
#calculate pooled variance
SS =(var(contrast[1:n])*(n -1)+var(contrast[(n+1):N])*(n -1))
var =SS/(N -nlevsa)
# SE and Margin of Error
SE =sqrt(2*var/n)
MoEs[idx] =qt(.975,N-nlevsa)*SE }
# the .80 quantile should be lower than f = .40
quantile(MoEs, .80)
## 80%
## 0.3998363
The result of this (rather small scale) simulation study is in agreement with
the results of the planning function: Selecting
n
= 42 participants per level of
the between participants factor gives us 80% assurance that standardized MoE
will not exceed
f
=
.
40. Again, whether this will be obtained in practice depends
on all the statistical assumptions, includeing that the population correlation
equals ρ=.50.
References
Cumming, G. (2012). Understanding the new statistics. Effect sizes, confidence
Intervals, and meta-analysis. New York: Routledge.
Calin-Jageman, R. & Cumming, G. (2019). The new statistics for better science:
Ask how much, how uncertain, and what else is known. The American
Statistician,73, 271-280.
Haans, A. (2018). Contrast analysis: A tutorial. Practical Assessment Research
& Evaluation,23, 1-21.
Kelley, K. (2013). Effect size and sample size planning. In T. D. Little (Ed.),
Oxford handbook of quantitative methods (Vol. 1.: Foundations, pp. 206-
222). New York: Oxford University Press.
Kruschke, J.K. (2015). Doing bayesian data analysis. A tutorial with R, Jags,
and Stan. Second Edition. Amsterdam, Netherlands: Academic Press.
Maxwell, S.E., Delaney, H., & Kelley, K. (2018). Designing experiments and
analyzing data. A model comparison perspective. (Third Edition). New
York: Routledge.
Maxwell, S.E., Kelley, K., & Rausch, J.R. (2008). Sample size planning for
statistical power and accuracy in parameter estimation. Annual Review of
Psychology,59, 537-563.
14
Rosenthal, R., Rosnow, R.L., & Rubin, D.B. (2000). Contrasts and effect sizes
in behavioral research. A correlational approach. New York: Cambridge
University Press.
Wiens, S., & Nilsson, M.E. (2017). Performing contrast analysis in factorial
designs: From NHST to confidence intervals and beyond. Educational and
Psychological Measurement, 77, 690-715.
15
Appendix
R-code for sample size planning functions
sampleSizeBetween <- function(lambda,
f.gamma,
gamma,
nlevs,
conf.level=.95) {
qMoE <- function(n) {
df =nlevs*(n -1)
ct =qt((1+conf.level)/2, df)
cchi =qchisq(gamma, df)
var =cchi/df *sum(lambda^2)/n
MoE =ct*sqrt(var)} # end qMoE
sqloss <- function(n) {
(qMoE(n) -f.gamma)^2
}# end squared loss function
SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)
return(SampleSize)
}
sampleSizeWithin <- function(lambda,
f.gamma,
gamma,
rho,
conf.level=.95) {
qMoE <- function(n) {
df =n-1
ct =qt((1+conf.level)/2, df)
cchi =qchisq(gamma, df)
var =(1-rho)*cchi/df *sum(lambda^2)/n
MoE =ct*sqrt(var) }
sqloss <- function(n) {
(qMoE(n)-f.gamma)^2}
SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)
return(SampleSize)
}
sampleSizeMixedBetween <- function(lambda,
f.gamma,
gamma,
rho,
nlevsa,
nlevsb,
conf.level=.95) {
a=nlevsa
b=nlevsb
var =(b +b*(b -1)*rho) /b^2
16
qMoE <- function(n) {
df =nlevsa *(n -1)
ct =qt((1+conf.level)/2, df)
cchi =qchisq(gamma, df)
ct*sqrt(sum(lambda^2)*var/n*(cchi/df))
}#end qMOE
sqloss <- function(n) {
(qMoE(n)-f.gamma)^2}
SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)
return(SampleSize) }
sampleSizeMixedWithin <- function(lambda,
f.gamma,
gamma,
rho,
nlevsa,
conf.level=.95) {
qMoE <- function(n) {
df =(nlevsa*n) -1
ct =qt((1+conf.level)/2, df)
cchi =qchisq(gamma, df)
var =(1-rho)*cchi/df *sum(lambda^2)/n
MoE =ct*sqrt(var) }
sqloss <- function(n) {
(qMoE(n)-f.gamma)^2}
SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)
return(SampleSize)
}
sampleSizeMixedInt <- function(lambda.a,
lambda.b,
f.gamma,
gamma,
rho,
nlevsa,
nlevsb,
conf.level =.95) {
qMoE <- function(n) {
df =nlevsa *(n -1)
ct =qt((1+conf.level)/2, df)
cchi =qchisq(gamma, df)
MoE =ct*sqrt(sum(lambda.a^2)*sum(lambda.b^2)*
(1-rho)/n*(cchi/df)) } #end qMOE
sqloss <- function(n) {
(qMoE(n)-f.gamma)^2}#end sqloss
SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)
return(SampleSize) }
17
