QUANTITATIVE RESEARCH | CHEAT SHEET
PR ER EQ UI SI TE S
•Write clear research questions, order them by priority and
importance, and write them out in full.
DE SI GN AND A NAL YS IS
•Design is more important than analysis.
•Before collecting data, ensure that your analysis matches
your design, and vice versa.
•Obtaining more data is always better, no matter what.
•Check whether your proposed study is ORANGEORANGE or GREENGREEN
in each row of the table on the right. For explanation, see
•If your study has at most TWO orange cells and no REDRED cell
in the table on the right, then proceed with caution. If your
study has more than two orange cells or one red cell, go
back and reconsider your design and analysis.
•Beware of order eects (priming, learning, emerging
strategies, fatigue, boredom, etc) within a participant’s
session and across multiple sessions for the same
participant. Test for these eects in your analyses.
•Check ALL assumptions of a statistical test or model BEFORE
conducting that test or tting that model.
LAX, PERMISSIVE, LIBERAL STRICT, RESTRICTIVE, CONSERVATIVE NOTES MY STUDY IS...
1No prior evidence against H0 (signicant
outcome may be false positive)
Strong prior evidence against H0 (signicant
outcome may be true positive)
If most of H0’s (!) being tested are true, a priori, then
most of signicant outcomes are false positives
(Ioannidis, 2005). See point 4.
2Key factors vary between participants Key factors vary within participants See tables below, and see Quené (2010) ORANGEORANGE
3Large variation between participants (items) Small variation between participants (items) Larger variation requires larger numbers of participants
(items), see point 5.
Consider (i.e. balance) both internal and external validity.
4Exploratory research, developing tentative
Experimental research, testing pre-existing
5Few participants OR few items Many participants AND many items See 3. Should be GREEN for GLMM or LMM, for
participants AND items.
NB “few” means 12 or fewer, “many” means 30 or more
6Low power High power NB “low” means .8 or less, “high” means .9 or more ORANGEORANGE
7Dependent variable (response) measured on
Dependent variable (response) measured on
Related to point 5.
“categorical” or qualitative response: e.g.
correct~incorrect response, scale with 5 or fewer
options; “continuous” or numerical response: e.g.
response time in ms, scale with 7 or more options, most
8Predicted eect is small in size:
small dierence, large variation
Predicted eect is large in size:
large dierence, small variation
Obtain estimates of variation from previous studies, or
from pilot work (see Quené, 2010).
Background: Quené & Van den Bergh (2020), §13.8.
9Many factors or predictors:
risk of overtting
Few factors or predictors:
“less is more”, “keep it simple”, robust
with k number of continuous predictors, m number
of levels of categorical factors, and N number of
N > 20(k+m), or, (k+m) < N/20
(Cohen, 1990; Quené, 2010)
10 Some concepts mentioned in this table are
not familiar to me
I have learned about and I fully understand all
concepts mentioned in this table
H0, variation, variance, eect size, power, signicance,
predictor, levels, response, n and N, model, test,
inference, sample, participants, stimuli, groups,
Hugo Quené (firstname.lastname@example.org) 2021.01.28 version 0.20 | license: CC: BY-SA (https://creativecommons.org/licenses/by-sa/4.0/)
RE FE RE NC ES
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12),
Gries, S. Th. (2015). Quantitative Linguistics. In International Encyclopedia of the
Social & Behavioral Sciences (2nd ed., Vol. 19, pp. 725–732). Oxford:
Ioannidis, J. (2005). Why Most Published Research Findings Are False. PLoS
Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Quené, H. (2010). How to design and analyze language acquisition
studies. In E. Blom & S. Unsworth (Eds.), Experimental Methods
in Language Acquisition Research (pp. 269–287). Amsterdam:
Quené, H. & Van den Bergh, H. (2020). Quantitative Methods and Statistics.
Retrieved 27 January 2021 from <https://hugoquene.github.io/
Winter, B. (2019). Statistics for Linguists: An Introduction Using R. Routledge.
Zuur, A. F., Ieno, E. N., & Elphick, C. S. (2010). A protocol for data exploration to
avoid common statistical problems. Methods in Ecology and Evolution,
1(1), 3-14. https://doi.org/10.1111/j.2041-210X.2009.00001.x
TREATMENT VARIES WITHIN PARTICIPANTS
group 1 (n=48) 1.A 1.B
group 2 (n=48) 2.A 2.B
total N=96 participants
AC KN OW LE DG EM EN TS
Thanks to Maaike Schoorlemmer, Kirsten Schutter and Piet van Tuijl for helpful comments and suggestions.
The following two tables illustrate row 2 of the table above.
(power >.8, sd=.5 for xed eects, sd=1.0 for random eects)
TREATMENT VARIES BETWEEN PARTICIPANTS
groups 1+2 (each n=32) 1.A 2.B
groups 3+4 (each n=32) 3.A 4.B
total N=128 participants