Content uploaded by Gerben Mulder

Author content

All content in this area was uploaded by Gerben Mulder on Apr 14, 2019

Content may be subject to copyright.

Sample Size Planning for Precise Contrast

Estimates: An Introduction and Tutorial for One

and Two Way Designs

Gerben Mulder

Department of Language, Literature, and Communication

VU Amsterdam

April 14, 2019

Contrast analysis provides a straightforward way of answering focused re-

search questions (Rosenthal, Rosnow, & Rubin, 2000). When combined with an

estimation approach the results of a contrast analysis are particularly informative

(Wiens & Nilsson, 2017), whether the researcher uses a Bayesian approach (e.g.

Kruschke, 2015) or a frequentist approach (Calin-Jageman & Cumming, 2019)

to estimation. Indeed, the results of a contrast analysis using the estimation

approach give the researcher both a point estimate and an interval estimate,

revealing plausible candidate values for the population and (thereby) input that

can be used for the evaluation of the theoretical and practical importance of the

results.

Of course, the informativeness of the results depends to a large extent on

the precision with which one estimates the contrast. That is, a wide interval

estimate, with a large range of plausible values, expresses more uncertainty

than a relatively narrow interval estimate, which makes it harder to assess the

theoretical and practical importance of the results. For this reason, it is advisable

to design a study with as large a precision as is necessary for the purposes of

the study, or at least to design a study with as high a precision as is practically

feasible.

The goal of the present contribution is to introduce sample size planning

for precision for contrast estimates. The general background to the approach

advocated here is Accuracy in Parameter Estimation (AIPE) (Kelley, 2013;

Maxwell, Kelley, & Rausch, 2008; see also Cumming, 2012, for a basic introduc-

tion). In essence, AIPE is about optimizing the design of studies focusing on the

estimation of model parameters in terms of the width of the conﬁdence interval

of the estimate. In the context of contrast estimates, then, the focus of sample

size planning for precision is on determing an optimal sample size for estimating

contrast values with as narrow a conﬁdence interval width as is necessary or

feasible.

This tutorial focuses speciﬁcally on contrast estimates in common experimen-

tal designs. It will consider one way within and between subjects design and

two way within, between, and mixed designs (split-plot designs). The tutorial

includes R-functions for sample size planning for precise contrast estimates in

these designs. Alternatively, planning for precision can be done with the Shiny

1

application that accompanies the article1

The structure of the tutorial is as follows. The tutorial starts with introducing

the central concepts underlying the planning approach by making reference to

a one factor between participants design. This is followed by examples of the

application of the R-functions for sample size planning in single factor and

factorial between participants design. The remainder is structured in the same

way, ﬁrst the mathematical formulas are presented and then these expressions

are applied to sample size planning for precision.

Central Concepts and Between Subject Designs

It is worthwile to introduce some of the basic concepts with a between subjects

design with a single ﬁxed factor with three levels and a continuous response

variable X. The purpose of the study is to estimate the contrast between the

population mean of treatment level 1 (

µ1

) and the mean of the other two

treatment levels (

µ2

+

µ3

)

/

2. These population means are the expected values

of three independent random variables

X1

,

X2

, and

X3

, one for each treatment

condition, and we assume them to be normally distributed with equal variance

σ2

. The assumptions of normality and equal variances (and covariances, for the

within and mixed designs) will be made throughout, and for that reason will not

be made explicit everytime).

The to be estimated contrast

ψ

can be formulated as a linear combination

of the population means (see for introduction to contrast estimation Maxwell,

Delaney, & Kelley, 2017; Rosenthal, Rosnow, & Rubin, 2000; or, for a gentle more

practically oriented introduction: Haans, 2018):

ψ

=

µΛT

, where in the present

example,

µ

= [

µ1, µ2, µ3

], a vector of population means, and

Λ

= [

λ1, λ2, λ3

]a

vector of contrast or lambda weights. Thus, ψ=µΛT=µ1λ1+µ2λ2+µ2λ3.

For example, if

µ

= [2

,

1

,

1] and we want to compare the ﬁrst population mean

to the average of the other two, we could use contrast weights

Λ

= [1

,−1

2,−1

2

].

This would give

ψ

= 2

∗

1 + 1

∗ −1

2

+ 1

∗ −1

2

= 1. By the same token, if

only the second and third expected values were to be contrasted, we could use

Λ

= [0

,

1

,−

1], which would give

ψ2

= [2

,

1

,

1][0

,

1

,−

1]

T

= 2

∗

0+1

∗

1+1

∗−

1 = 0.

The estimated value of the contrast between population means is simply the

corresponding linear combination of the sample means. Thus, to obtain the

point estimate, we multiply the vector of sample means

¯

X

by the transpose of

the contrast weights vector: ˆ

ψ=¯

XΛT.

As our expression of the uncertainty of the point estimate, we will use the

95% conﬁdence interval (CI). Using terminology of Cumming (2012), the 95%

CI of the contrast estimate can be obtained by adding to and subtracting from

the point estimate the (estimated) Margin or Error (MoE). In other words, MoE

is the half-width of the 95% CI.

The expected value of MoE equals

MoE =t.975,df σˆ

ψ,(1)

where

σˆ

ψ

=

qE[( ˆ

ψ−ψ)2]

, the standard error of the contrast estimate, and

t.975,df the .975 quantile of a central t-distribution with df degrees of freedom.

1

The application is available here: https://gmulder.shinyapps.io/PlanningFactorialContrasts.

The app is restricted to planning for precision in one and two-way factorial designs.

2

The standard error of the contrast estimate is the square root of the variance

of the contrast estimate

σ2

ˆ

ψ

=

E

[(

ˆ

ψ−ψ

)

2

]. The value of the variance is a function

of both the sample size and the contrast weights. In particular, the variance of

the contrast estimate is equal to the sum of squared contrast weights multiplied

by the within-condition variance

σ2

divided by the per condition sample size (n):

σ2

ˆ

ψ=ΛΛTσ2

n= (Xλ2

i)σ2

n=1

n(Xλ2

i)σ2.(2)

If we combine (1) and (2) we get an expression for expected MoE

MoE =t.975,df r1

n(Xλ2

i)σ2.(3)

From (1) and (3) it is clear that we need to specify the degrees of freedom.

For the one-way between participants design these are the degrees of freedom

that are used to estimate the within condition variances, i.e. with a levels of the

treatment factor the df equal

df

=

a

(

n−

1) =

N−a

. The degrees of freedom

for the other designs are given in the Appendix.

The idea of sample size planning for precision is already quite apparent from

(3). We determine a value for MoE and solve (3) for

n

. There are, however, a

few complications. The ﬁrst complication is that in order to use (3) we need

to specify the value of the within condition variance, whose value is generally

unknown. This problem can be solved quite easily: in stead of working with

an absolute value for MoE, we express MoE as a number of within condition

standard deviations (Cumming, 2012):

f=MoE/σ =t.975,df r1

n(Xλ2

i)√σ2

√σ2=t.975,df r1

nXλ2

i.(4)

The second complication is that the sample size enters (4) twice, once as

the denominator of the fraction involved in the expression for the standardized

standard error and once in the degrees of freedom

a

(

n−

1), as Cumming (2012)

explains, this makes it necessary to use an iterative routine in order to solve for

n in (4). With software like R, however, this is easily implemented.

A ﬁnal complication is that if we set f in (4) and solve for n using the

iterative routine, we get the sample size necessary to obtain our target MoE on

average. This means, that in approximately half the studies we undertake with

the required sample size, we will get a Margin of Error larger than the value

we want. This is where we will need the concept of assurance (Cumming, 2012;

Kelley, 2013): we will plan for a sample size that will give us a MoE that will

not exceed our target MoE with probability

γ

. We will make use of the fact that

the sampling distribution of the within condition variance is a scaled chi-squared

distribution. That is,

ˆσ2∼σ2χ2

df /df,

which means that if we standardize MoE by the within condition standard

deviation, that the

γ

-quantile of the sampling distribution of estimated MoE is

fγ=t.975,df r1

n(Xλ2

i)χ2

γ,df /df. (5)

3

All told, then, we set a value for f and assurance

γ

, meaning that we specify

the

γ

-quantile of the sampling distribution of standardized MoE, we specify

the vector of lambda weights, and we use these speciﬁcations with an iterative

routine to solve for n in (5). This will give us the required per condition sample

size necessary to obtain estimated standardized MoE no larger than our target

MoE with

γ

% assurance. Note that if we ﬁll in the sample size in (4), we

get a value for expected MoE. Sample size planning for the other designs uses

variations of (5), see Appendix.

Planning for Contrasts in Between Participants Designs

Single factor designs

Let’s return to our example. We have a three condition between subjects design

and we want to estimate the diﬀerence between the population mean of the

ﬁrst condition and the average of the expected values of the other two random

variables. We specify the vector of contrast weights as follows

Λ

= [1

,−1

2,−1

2

]

and let us suppose that we want to plan for a target MoE of f = .50 and assurance

γ

=

.

80. The R-function sampleSizeBetween provided in the Appendix can do

the job for us. Alternatively, use the Shiny application for sample size planning

that accompanies this article.

lambda =c(1,-1/2,-1/2)

f=.50

gamma =.80

nlevs =3

sampleSizeBetween(lambda, f, gamma, nlevs)

## [1] 27

The planning function gives us a value of n = 27 participants per treatment

level, so in total

N

= 27

∗

3 = 81. If we use n = 27 in (4), expected MoE equals

f = 0.4692.

We can use a simple simulation to see whether target MoE will be below f

= .50 in 80% of the replication experiments. This can be done as follows, note,

however, that since the random seed is not set, running this code will lead to a

somewhat diﬀerent result than the one reported here.

nReps =10000

n=27

df =3*(n -1)

lambda =c(1,-1/2,-1/2)

sum.sq.w =sum(lambda^2)

MoEs =rep(0, nReps)

for (idx in 1:nReps) {

a1 =rnorm(n)

4

a2 =rnorm(n)

a3 =rnorm(n)

v=c(var(a1), var(a2), var(a3))

v=v%*% rep(n -1,3)/df

MoEs[idx] =qt(.975, df)*sqrt(v/n*sum.sq.w)

}

quantile(MoEs, .80)

## 80%

## 0.4988301

Another possibility is to make use of data that conﬁrms exactly to our

speciﬁcations. Below, the three condition means are all equal (zero), and the

within condition variances are equal to one. The ﬁrst contrast (X1) estimates

the contrast with weights [1

,−1

2,−1

2

]. The result of the estimate is zero, and

the lower and upper limits of the 95% conﬁdence intervals should therefore

correspond to minus MoE and plus MoE, respectively. Note that because the

within condition variances equals 1, the Margin of Error of the estimate is equal

to the value of

f

= 0

.

4692 above. Likewise, the value of MoE for the second

contrast (X2) should be equal to MoE = 1.9908 ∗q1

27 ∗2=0.5418

library(MASS) #needed for ginv function below

n=27

# random variables scaled to have

# mean = 0 and var = 1

a1 =scale(rnorm(n))

a2 =scale(rnorm(n))

a3 =scale(rnorm(n))

# response variable:

Y=c(a1, a2, a3)

# independent treatment factor:

X=factor(c(rep(1, n), rep(2, n), rep(3, n)))

#define full set of orthogonal (helmert) contrasts

contrasts(X) =ginv(rbind(c(1,-.5,-.5), c(0,1,-1)))

#estimate model:

myMod =lm(Y ~X)

#obtain intervals

confint(myMod)

## 2.5 % 97.5 %

## (Intercept) -0.2212052 0.2212052

## X1 -0.4692472 0.4692472

## X2 -0.5418399 0.5418399

5

Factorial designs

The main purpose of using a factorial design is that these designs allow for the

studying interactions between factors. Studying interactions should take center

stage, and, therefore, from the estimation perspective the main focus of the

analysis is estimating interaction contrasts and their conﬁdence intervals. Of

course, that is not to say that contrasts of the marginal means (main eﬀect

contrasts) should not be considered, or to return to the topic of this tutorial,

sample size planning for obtaining estimates of these contrasts should be ignored.

Planning for precise contrast estimates in factorial between participants

designs can easily be done with the R-function provided above. There are just a

few things to consider. The ﬁrst is that in specifying the weights vector, it is

convenient to conceptualize the design as a single factor design with number

of levels equal to the product of the number of levels of the factors involved.

Restricting the discussion to the two-way factorial designs, suppose that there

are two treatment factors each with two levels, we would conceptualize the design

not as a 2 x 2 factorial design, but as a 1 x 9 one-way design. The second thing

to consider is that if we use the sample size planning function provide above,

the nlevs-argument should be set equal to the product of the two factors.

Thus, if we were to plan for

f.80

=

.

50 for a contrast of marginal means in a

two-by-two design, the weights vector for the main eﬀect of one of the treatments

could be set to

Λ

= [

1

2,1

2,−1

2,−1

2

]and the number of levels to 4. Using this

input in the sample size planning function, gives the following result.

lambda =c(1/2,1/2,-1/2,-1/2)

f=.50

gamma =.80

nlevs =4

sampleSizeBetween(lambda, f, gamma, nlevs)

## [1] 19

So, we need

n

= 19 observations per treatment level, making for a total of

N= 4 ∗19 = 76 observations.

If we want to plan for an interaction contrast, the thing to consider is that for

mean comparisons the absolute value of the sum of the contrast weights equals

4.0 (Kline, 2013), whereas for main eﬀect contrasts this sum should equal 2.0.

Thus, if we plan for

f.80

=

.

50 for an interaction contrast in a 2 x 3 design, we

should use Λ= [1,−1

2,−1

2,−1,1

2,1

2], with the number of levels set to 2∗3=6:

lambda =c(1,-1/2,-1/2,-1,1/2,1/2)

f=.50

gamma =.80

nlevs =6

sampleSizeBetween(lambda, f, gamma, nlevs)

## [1] 50

Thus, according to the planning results, we need

n

= 50 participants per

level, makeing for a total of N= 300.

6

Within Participants Designs

For contrast analysis in within participants designs a choice has to be made

between two approaches (Maxwell, Delaney, & Kelley, 2017). The ﬁrst approach

is to ﬁt a mixed model, with treatment(s) as ﬁxed factor and participant as

random factor (here, we consider designs with one observation per treatment

condition per participant only, so we model only two sources of random variance,

i.e. variance due to participant and due to error), and use the results of ﬁtting

and all the underlying statistical assumptions to estimate the margin of error of

each contrast. If all assumptions of the model apply, the precision of the estimate

will be (somewhat) higher, i.e. MOE will have a (somewhat) smaller value and

the conﬁdence interval will be (somewhat) narrower than in the alternative

approach. However, if the assumptions do not apply, the estimated precision

cannot be trusted (Maxwell, Delaney, & Kelley, 2017), so it may not be worthwile

to use the mixed model approach. The alternative is to transform the scores

for each participant to a contrast score, and to subsequently estimate the mean

contrast score over participants. This requires less assumptions than the mixed

model approach, and is shown to be superior to the mixed model approach when

the assumptions of the latter approach do not obtain (Maxwell, Delaney, &

Kelley, 2017).

In the planning approach described in this article, it is assumed that the

mixed model assumptions hold, but that the contrast analysis itself uses the

alternative transformed scores approach. The assumptions made for the within

participant designs (and for the within factors of the mixed design) is that the

random variables are multivariate normal variables with equal variances (

σ2

)

and covariances. The covariances are equal to

ρσ2

, where

ρ

is Pearson’s product

moment correlation.

These assumptions lead to the following expression of the Margin of Error of

a contrast estimate:

MoE =t.975,df r1

n(Xλ2

i)(σ2−ρσ2).(6)

It is easy to demonstrate that the variance of the contrast scores equals

Pλ2

i

(

σ2−ρσ2

). Take for instance the variance of the contrast score based

on subtracting one random variable from the other, say

X1−X2

. We have

Λ

= [1

,−

1], and

var

(

X1−X2

) =

var

(

X1

) +

var

(

X2

)

−

2

cov

(

X1, X2

) = 2

σ2−

2

ρσ2

= 2(

σ2−ρσ2

) =

Pλ2

i

(

σ2−ρσ2

)

.

Or, with three conditions, with

Λ

=

[1

,−1

2,−1

2

], we have

var

(

X1−1

2X2−1

2X3

) =

var

(

X1

)+

var

(

1

2X2

)+

var

(

1

2X3

)

−

2

cov

(

X1,1

2X2

)

−

2(

cov

(

X1,1

2X3

)

−

2

cov

(

1

2X2,1

2X3

) =

σ2

+

1

4σ2

+

1

4σ2

+

2

2ρσ2

+

2

2ρσ2−2

4ρσ2= 1 1

2σ2−11

2ρσ2=Pλ2

i(σ2−ρσ2).

If MoE is standardized by the within condition standard deviation, and we

take into account the assurance, we get:

fγ=t.975,df r1

n(Xλ2

i)(1 −ρ)χ2

γ,df /df. (7)

We will use (7) for sample size planning, with degrees of freedom equal to

n−

1(the number of participants minus 1), for the completely within participants

designs, and

df

=

an −

1for the mixed design, with

a

being the number of levels

7

of the between-participants factor. In order to use (7) we need to assume that

ρ <

1. Note that contrary to sample size planning for between participants

designs we will have to specify a value for a population parameter, i.e. the

population correlation between the treatment conditions.

Planning for Contrasts in Within Participants Designs

Single Factor Designs

The R-function sampleSizeWithin provided in the Appendix can be used to

plan for the sample size in whithin participants designs, alternatively, the Shiny

application can be used (see footnote 1). The R-function provided can be applied

for planning in both single factor and multi-factor designs. The main diﬀerence

between these designs in how the function is used, is in the way the contrast

weights are speciﬁed, as will be demonstrated momentarily.

Let us consider planning for a target MoE of f = .50 with 80% assurance,

in a single factor within participants design with three conditions. We use the

following additional input

Λ

= [1

,−1

2,−1

2

]and

ρ

=

.

60. Here, the value of

ρ

is arbitrarily set to equal

.

60, in practice one would either use a value that is

informed by previous research, or plan for a range of plausible values for the

correlation.

f=.50

gamma =.80

lambda =c(1,-1/2,-1/2)

rho =.60

sampleSizeWithin(lambda, f, gamma, rho)

## [1] 15

If 15 participants are selected for the study, the expected standardized MoE

will be equal to

t.975,14q1

15 1.5(1 −.60)

= 2

.

1448

∗

0

.

2000 = 0

.

4290. Let us

generate an ideal dataset for which the assumptions about the variance

σ2

= 1

and covariance

ρσ2

= 0

.

60 are true. A contrast analysis of this ideal data will

show that the value for expected MoE is correct.

# library(MASS)

n=15

lambda =c(1,-1/2,-1/2)

# specify means

mu =c(0,0,0)

# variance covariance matrix

Sigma =matrix(c(1,.6,.6,.6,1,.6,.6,.6,1), 3,3)

# generate multivariate data

theData =mvrnorm(n, mu, Sigma, empirical=TRUE)

# calculate contrast score

contrastScore =theData %*% lambda

# t.test, upper limit of CI equals expected standardized MoE

t.test(contrastScore)

8

##

## One Sample t-test

##

## data: contrastScore

## t = 4.349e-16, df = 14, p-value = 1

## alternative hypothesis: true mean is not equal to 0

## 95 percent confidence interval:

## -0.4289573 0.4289573

## sample estimates:

## mean of x

## 8.698012e-17

Factorial Within Participants Designs

Planning in a factorial within participants design can be done with the same

sample planning for precision function as we used above for the single factor

within participants design. The main diﬀerence is that in the speciﬁcation

of the contrast weights we need to specify a number of weights equal to the

product of the number of levels of each factor. Let us consider a design with

two within participant factors, one with two levels and one with three. The

number of lambda weights is therefore equal to 6. For contrast analysis of the

main treatment eﬀects, the sum of the absolute weights should equal 2.0 and for

interaction contrasts that sum should be 4.0 (Kline, 2013).

The following example with R-code shows a convenient way of obtaining

sample sizes for all of a full set of orthogonal contrasts for the 3 by 2 within

participants design. This is speciﬁcally convenient if the value for

fγ

is the same

for all contrasts. If the value diﬀers from contrast to contrast, use the planning

function on a per contrast basis. We will plan for

f.95

=

.

30 and arbitrarily set

the correlation to equal ρ=.75

# this function can be used to make

# sure that the sum of absolute values

# of the lambda weights sum to 2 (default)

forceAbsoluteSum =function(lambda,sum.value=2) {

lambda *sum.value /sum(abs(lambda))

}

levelsa =3

levelsb =2

# note that for the planning functions

# the order of the weights is irrelevant,

# only the sum of squared weights needs to be correct.

A1 =c(1,-1/2,-1/2)

A2 =c(0,1,-1)

B1 =c(1,-1)

9

# start with interaction contrasts

interA1B1 =as.vector(outer(A1, B1))

interA2B1 =as.vector(outer(A2, B1))

# the absolute values of the interaction lambda weights

# sum to 4 already, so the following is not necessary

# in this case.

# just to illustrate

interA1B1 =forceAbsoluteSum(interA1B1, 4)

# make sure that that each weights factor

# contains 6 elements and that absolute

# values of weights sum to two

A1 =rep(A1, levelsb)

A1 =forceAbsoluteSum(A1)

A2 =rep(A2, levelsb)

A2 =forceAbsoluteSum(A2)

B1 =rep(B1, levelsa)

B1 =forceAbsoluteSum(B1)

# make matrix

contrasts =cbind(A1, A2, B1, interA1B1, interA2B1)

apply(contrasts, 2, sampleSizeWithin,

f=.30,gamma =.95,rho =.75)

## A1 A2 B1 interA1B1 interA2B1

## 16 20 15 47 59

The planning results show that with 59 participants (the second interaction

contrast), the margin of error of each individual contrast will not exceed more

than .30 standard deviations, with per contrast assurance at least equal to 95%.

These samples sizes are of course indicative to the extent that the assumptions

made for planning are reasonable, including the assumption that the population

correlation equals ρ=.70.

Factorial Mixed Designs

As we have seen, the sample size planning functions above can be used for

one and two-way between and within participants factorial designs. The only

diﬀerence between one and two way designs is that for planning in between

participants designs the number of levels is the product of the number of levels

of the factors involved, i.e. in a two-way factorial design, with

a

levels for the

ﬁrst factor and

b

levels for the second factor, the number of levels equals

ab

. In

the case of mixed factorial designs, however, the situation is more complex.

Suppose we have a mixed factorial design with two factors. Treatment A is a

10

between participants factor with

a

= 2 levels, Treatment B a within participants

factor with

b

= 3 levels, and we suppose that for each combination of a level of

A and a level of B we have

n

observations (say one of each of

n

participants).

Let us also assume that each of the

ab

variables has equal variance

σ2

, and that

the covariance between the levels of Treatment B equals ρσ2.

Let us ﬁrst consider estimating a contrast between the two population means

associated with the levels of Treatment A. The scores in each level of A are

the participants means of the

b

= 3 variables corresponding to Treatment B.

So, if we let

Xa.k

stand for the score of participant

k

in level

a

of Treatment

A averaged of the

b

levels of Treatment B, we know that the variance of the

Xa.k

must equal the variance of the mean of the

b

= 3 participant scores. Thus,

var

(

Xa.k

) =

var

([

Xa1k

+

Xa2k

+

Xa3k

)

/

3]) = (3

σ2

+6

ρσ2

)

/

9 =

σ2

(

b

+

b

(

b−

1)

ρ

)

/b2

.

We can use the latter in the expression of the (unstandardized) MoE of the

contrast between the condition means of Treatment A:

MoE =t.975,df r1

n(Xλ2

i)σ2(b+b(b−1)ρ)/b2.(8)

In standardized form, that is standardizing MoE by dividing by

σ

, and

taking into account the assurance, the expression for the margin of error of the

between-participants contrast becomes

fγ=t.975,df r1

n(Xλ2

i)(b+b(b−1)ρ)/b2χ2

γ,df /df, (9)

where the degrees of freedom equal

df

=

a

(

n−

1). Thus, with, for example,

a

= 2,

b

= 3,

n

= 10,

σ2

= 1

.

5,

ρ

=

.

60, and contrast weights for the between

factor set to

Λ

= [1

,−

1], the expected value of MoE is of the contrast equals

MoE =t.975,18 ∗q1

10 ∗2∗1.5∗(3 + 6 ∗.60)/9=0.9854.

For the within-participants factor B, we would transform the scores for each

of the

an

participants and use the resulting contrast scores to estimate the

expected value of the contrast. This means that we can simply use (7) for sample

size planning, with degrees of freedom equal to df =an −1.

An interaction contrast of the two factors A and B can be obtained by ﬁrst

transforming the treatment B scores to contrast scores and subsequently estimate

a between participants contrast on the resulting scores. We will need to formulate

two contrast vectors, one for each factor. The expression for MoE is

MoE =t.975,df r1

n(Xλ2

a)(Xλ2

b)(σ2−ρσ2),(10)

wich degrees of freedom equal to

df

=

a

(

n−

1). If we standardize MoE and take

into account assurance, we get

fγ=t.975,df r1

n(Xλ2

a)(Xλ2

b)(1 −ρ)χ2

df /df. (11)

Planning for Precise Contrast estimates in Mixed Designs

The complexities of the mixed design call for three diﬀerent planning functions,

one for the contrast analysis of the between participants factor, one for the

11

within participant factor, and one for the interaction of these factors. These

functions can only be used for designs with two factors .

Between Participant contrasts

Suppose we want our MoE to not exceed .25 standard deviations, with assurance

.90. In terms of the assumptions above, where we assumed that

σ2

= 1

.

5, this

value of f corresponds to a MoE of

.

25

∗√1.5

= 0

.

3062. Let us use (9) to plan for

a target MoE of

f.90

= 0

.

25. The following R-code can be used for that purpose.

lambda =c(1,-1)

f=0.25

gamma =.90

rho =.60

nlevsa =2

nlevsb =3

sampleSizeMixedBetween(lambda, f, gamma, rho,

nlevsa, nlevsb)

## [1] 103

The result of the planning function is that we need

n

= 103 participants per

between-participants condition, so,

N

= 206 in total. This will give an expected

MoE of MoE =t.975,204 ∗q1

103 ∗2∗1.5∗(3 + 6 ∗.60)/9=0.2882.

Within participant contrasts

For the within participants factor B, which has three levels, we will plan for two

Helmert contrasts. The ﬁrst Helmert contrasts has

Λ1

= [1

,−1

2,−1

2

], and for the

second contrasts we specify

Λ2

= [0

,

1

,−

1]. We want to have 90% assurance per

contrast that MoE will not exceed .40 standard deviations, so we set

f.90

=

.

40.

Note that since the precision of the second contrasts will be lower (= higher

value for MoE) than for the ﬁrst contrast, due to the fact that the sum of squared

weigths of the second contrast is larger than that of the ﬁrst (and assuming

everthing else equal), so we only need to plan for the second contrast, since the

sample size needed to obtain our precision goal for the second contrast, will

necessarily be suﬃcient for the ﬁrst contrast as well. We will again assume an

arbitrary value for the population correlation, and set

ρ

=

.

50. The following

code for R uses the sample size planning function provided in the Appendix.

lambda =c(0,1,-1)

f=.40

gamma =.90

rho =.50

nlevsa =2# number of levels of between factor

sampleSizeMixedWithin(lambda, f, gamma, rho, nlevsa)

## [1] 31

The results are that we need at least 31 participants. Of course, since these

31 participants will be randomly assigned to one of the two treatment conditions,

12

it is more convenient to select an even number of participants, the most obvious

candidate value being N= 32 or n= 16.

Interaction contrasts in the Mixed Design

Let us suppose that we want to plan for a target MoE of

f

=

.

40, with assurance

γ

=

.

80, for the estimate of a interaction contrast comparing the levels of

Treatment A with respect to the value of a Treatment B contrast in which the

ﬁrst level of B is compared to the two other levels. For treatment B with have

the contrast weights vector

ΛB

= [1

,−1

2,−1

2

], and to compare the value of this

contrast between the levels of A we formulate a weights vector

ΛA

= [1

,−

1].

We will assume that the the value of the population correlation equals ρ=.50

We can use this input with the following R-code:

f=.40

gamma =.80

lambdaA =c(1,-1)

lambdaB =c(1,-.5,-.5)

rho =.50

nlevsa =2

nlevsb =3

sampleSizeMixedInt(lambdaA,

lambdaB, f, gamma, rho, nlevsa, nlevsb)

## [1] 42

According to the results of the planning function, then, we need

n

= 42

participants per level of the between participants factor, or 84 participants in

total. Let us check this sample size with a simple simulation study.

# library(MASS)

var =1

cov =0.5

#between-factor

nlevsa =2

#within-factor

nlevsb =3

weightsB =c(1,-.5,-.5)

# condition means

means =rep(0, nlevsb)

#variance-covariance-matrix

Sigma =matrix(c(1,.5,.5,.5,1,.5,.5,.5,1), 3,3)

# sample size per independent group equals 42

# total sample size equals 84

n=42

N=nlevsa *n

MoEs =NULL

# 10000 replications

13

nReps =10000

for (idx in 1:nReps) {

scores =mvrnorm(N, means, Sigma)

contrast =scores %*% weightsB

#calculate pooled variance

SS =(var(contrast[1:n])*(n -1)+var(contrast[(n+1):N])*(n -1))

var =SS/(N -nlevsa)

# SE and Margin of Error

SE =sqrt(2*var/n)

MoEs[idx] =qt(.975,N-nlevsa)*SE }

# the .80 quantile should be lower than f = .40

quantile(MoEs, .80)

## 80%

## 0.3998363

The result of this (rather small scale) simulation study is in agreement with

the results of the planning function: Selecting

n

= 42 participants per level of

the between participants factor gives us 80% assurance that standardized MoE

will not exceed

f

=

.

40. Again, whether this will be obtained in practice depends

on all the statistical assumptions, includeing that the population correlation

equals ρ=.50.

References

Cumming, G. (2012). Understanding the new statistics. Eﬀect sizes, conﬁdence

Intervals, and meta-analysis. New York: Routledge.

Calin-Jageman, R. & Cumming, G. (2019). The new statistics for better science:

Ask how much, how uncertain, and what else is known. The American

Statistician,73, 271-280.

Haans, A. (2018). Contrast analysis: A tutorial. Practical Assessment Research

& Evaluation,23, 1-21.

Kelley, K. (2013). Eﬀect size and sample size planning. In T. D. Little (Ed.),

Oxford handbook of quantitative methods (Vol. 1.: Foundations, pp. 206-

222). New York: Oxford University Press.

Kruschke, J.K. (2015). Doing bayesian data analysis. A tutorial with R, Jags,

and Stan. Second Edition. Amsterdam, Netherlands: Academic Press.

Maxwell, S.E., Delaney, H., & Kelley, K. (2018). Designing experiments and

analyzing data. A model comparison perspective. (Third Edition). New

York: Routledge.

Maxwell, S.E., Kelley, K., & Rausch, J.R. (2008). Sample size planning for

statistical power and accuracy in parameter estimation. Annual Review of

Psychology,59, 537-563.

14

Rosenthal, R., Rosnow, R.L., & Rubin, D.B. (2000). Contrasts and eﬀect sizes

in behavioral research. A correlational approach. New York: Cambridge

University Press.

Wiens, S., & Nilsson, M.E. (2017). Performing contrast analysis in factorial

designs: From NHST to conﬁdence intervals and beyond. Educational and

Psychological Measurement, 77, 690-715.

15

Appendix

R-code for sample size planning functions

sampleSizeBetween <- function(lambda,

f.gamma,

gamma,

nlevs,

conf.level=.95) {

qMoE <- function(n) {

df =nlevs*(n -1)

ct =qt((1+conf.level)/2, df)

cchi =qchisq(gamma, df)

var =cchi/df *sum(lambda^2)/n

MoE =ct*sqrt(var)} # end qMoE

sqloss <- function(n) {

(qMoE(n) -f.gamma)^2

}# end squared loss function

SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)

return(SampleSize)

}

sampleSizeWithin <- function(lambda,

f.gamma,

gamma,

rho,

conf.level=.95) {

qMoE <- function(n) {

df =n-1

ct =qt((1+conf.level)/2, df)

cchi =qchisq(gamma, df)

var =(1-rho)*cchi/df *sum(lambda^2)/n

MoE =ct*sqrt(var) }

sqloss <- function(n) {

(qMoE(n)-f.gamma)^2}

SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)

return(SampleSize)

}

sampleSizeMixedBetween <- function(lambda,

f.gamma,

gamma,

rho,

nlevsa,

nlevsb,

conf.level=.95) {

a=nlevsa

b=nlevsb

var =(b +b*(b -1)*rho) /b^2

16

qMoE <- function(n) {

df =nlevsa *(n -1)

ct =qt((1+conf.level)/2, df)

cchi =qchisq(gamma, df)

ct*sqrt(sum(lambda^2)*var/n*(cchi/df))

}#end qMOE

sqloss <- function(n) {

(qMoE(n)-f.gamma)^2}

SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)

return(SampleSize) }

sampleSizeMixedWithin <- function(lambda,

f.gamma,

gamma,

rho,

nlevsa,

conf.level=.95) {

qMoE <- function(n) {

df =(nlevsa*n) -1

ct =qt((1+conf.level)/2, df)

cchi =qchisq(gamma, df)

var =(1-rho)*cchi/df *sum(lambda^2)/n

MoE =ct*sqrt(var) }

sqloss <- function(n) {

(qMoE(n)-f.gamma)^2}

SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)

return(SampleSize)

}

sampleSizeMixedInt <- function(lambda.a,

lambda.b,

f.gamma,

gamma,

rho,

nlevsa,

nlevsb,

conf.level =.95) {

qMoE <- function(n) {

df =nlevsa *(n -1)

ct =qt((1+conf.level)/2, df)

cchi =qchisq(gamma, df)

MoE =ct*sqrt(sum(lambda.a^2)*sum(lambda.b^2)*

(1-rho)/n*(cchi/df)) } #end qMOE

sqloss <- function(n) {

(qMoE(n)-f.gamma)^2}#end sqloss

SampleSize <- ceiling(optimize(sqloss, c(1,5000))$minimum)

return(SampleSize) }

17