Content uploaded by Bas Van Hooren

Author content

All content in this area was uploaded by Bas Van Hooren on May 18, 2018

Content may be subject to copyright.

Magnitude-based inference: What is it? How does it work and is it appropriate?

Magnitude-based inference: What is it? How does

it work and is it appropriate?

B. Van Hooren 1

1NUTRIM School of Nutrition and Translational Research in Metabolism, Maastricht University Medical Centre+, Department of Nutrition and Movement Sciences, Maastricht,

The Netherlands

Magnitude-based inference |Video |Tutorial

Headline

Research in the ﬁeld of sports science is frequently per-

formed on a relatively small number of individuals. We

are usually however interested in knowing whether the eﬀect

found in our sample of individuals also applies to a larger

group: the population from which the sample is drawn. For

this purpose, we use statistical inferential methods. There

are several statistical inferential methods available. The most

widely used method is arguably null-hypothesis signiﬁcance

testing. This method has been widely criticized since its in-

troduction, most prominently because statistically signiﬁcant

results are not necessarily clinically relevant and statistically

non-signiﬁcant results can still be clinically relevant (Figure

1). (1, 2)

Magnitude-Based Inference. Motivated by the limitations of

null-hypothesis signiﬁcance testing, Batterham and Hopkins

(1) developed a new statistical inferential method in 2006 enti-

tled “Magnitude-Based Inference”. In this method, conﬁdence

intervals are interpreted in relation to a smallest worthwhile

change (Figure 1). The method has seen a large uptake in the

sports science community and is also increasingly used in other

research ﬁelds. Despite this large uptake, not all researchers

and practitioners fully understand how the method works. Un-

derstanding how Magnitude-Based Inference works is however

important as it helps researchers and practitioners to correctly

interpret the results of studies that have used this method. In

a new video, I therefore explain what Magnitude-Based Infer-

ence is and how it works. Click here for the link to the video.

Criticism

Several researchers have criticized Magnitude-Based Inference,

in particular for interpreting a frequentist conﬁdence interval

as a Bayesian credible interval and for having high rates of

type I errors (false positive were you conclude there is a sub-

stantial eﬀect while there is no substantial eﬀect).(3-6) These

researchers therefore advised to use other statistical inferential

methods such as a full Bayesian analysis (which has recently

been performed with Magnitude-Based Inference (7)) or equiv-

alence testing (see for example (8)). Batterham and Hopkins

have responded to these criticisms. (9-13) They justify their

Bayesian inferences with the conﬁdence interval by claiming to

use Bayesian methods, but with a non-informative prior dis-

tribution which results in the Bayesian credible interval to be

equivalent to the frequentist conﬁdence interval, when several

other assumptions are met.(5) Further, they re-analysed the

inferential error rates using diﬀerent deﬁnitions of the errors

(12) as used by the other researchers (4) and argued that the

deﬁnitions of errors used in a recent critique paper (6) are also

not entirely appropriate.(13)

The debate around these statistical issues may be diﬃ-

cult to follow for a sport scientist who does not have a

strong background in statistics. Understanding this de-

bate is however important as it allows researchers to de-

cide on whether they should use Magnitude-Based Infer-

ence or other statistical inferential methods for their stud-

ies. Therefore, in a second video I discuss some of the crit-

icisms on Magnitude-Based Inference and the responses by

Batterham and Hopkins in a (hopefully) understandable way.

Click here for the link to the second video.

Other approaches

Researchers that like the idea of Magnitude-Based Inference,

but who do not want to use it based on the criticisms can use

several other methods which are roughly similar. For exam-

ple, Mengersen, Drovandi, Robert, Pyne and Gore (7) recently

performed a full Bayesian analysis of Magnitude-Based Infer-

ence with a ﬂat prior distribution. Other approaches that

are roughly similar to Magnitude-Based Inference include us-

ing regions of practical equivalence (ROPE) in a Bayesian ap-

proach or equivalent testing in a frequentists approach (8, 14).

Twitter: Follow Bas Van Hooren @BasVanHooren

Fig. 1. Magnitude-Based Inference. Decisions in Magnitude-Based

Inference are made based on conﬁdence intervals (represented by the blue horizontal

lines) in relation to a smallest worthwhile change (represented by the dashed vertical

lines on each side of the trivial area). Consider the following example: a study has

investigated the eﬀects of 4 weeks resistance training on back squat 1 repetition max-

imum performance. Any increase or decrease larger than 5 kg is considered relevant,

while all changes smaller than 5 kg are too small to be of practical relevance (i.e.,

trivial). In Magnitude-Based Inference, conﬁdence intervals deﬁne the likely range of

the population value. If the study ﬁnds the eﬀect illustrated by the ﬁrst conﬁdence

interval, the conclusion is therefore that the intervention is (very likely) eﬀective as

the conﬁdence interval is entirely in the beneﬁcial area. For the second interval, the

conﬁdence interval overlaps the trivial and beneﬁcial areas. However, the overlap in

the beneﬁcial area is larger and the intervention is therefore more likely to be beneﬁcial

as trivial. The conclusion could therefore be to use the intervention because it might

be beneﬁcial and in the worst case scenario the intervention will have a trivial eﬀect.

In the 3rd and 4th interval, the overlap of the conﬁdence interval into the trivial area

has increased, so the intervention could have a trivial eﬀect, but it could also be ben-

eﬁcial. If the training intervention would not require much time and money, a coach

could still decide to use the intervention. However, the results are not statistically

signiﬁcant. Conversely, in the 5th interval the intervention has likely only a trivial

eﬀect, but it is signiﬁcant. These latter examples illustrate the mismatch between

practical relevance and statistical signiﬁcance. Figure adapted from Batterham and

Hopkins (1).

sportperfsci.com 1 SPSR - 2018 |May |x|v1

Magnitude-based inference: What is it? How does it work and is it appropriate?

References

1. Batterham AM, Hopkins WG. Making meaningful in-

ferences about magnitudes. Int J Sports Physiol Perform

2006;1(1):50-57.

2. Stang A, Poole C, Kuss O. The ongoing tyranny of sta-

tistical signiﬁcance testing in biomedical research. Eur J Epi-

demiol 2010;25(4):225-30.

3. Barker RJ, Schoﬁeld MR. Inference about magnitudes of

eﬀects. Int J Sports Physiol Perform 2008;3(4):547-57.

4. Welsh AH, Knight EJ. ”Magnitude-based inference”: a

statistical review. Med Sci Sports Exerc 2015;47(4):874-84.

5. Butson M. Will the numbers really love you back: Re-

examining Magnitude-based Inference. 2017

6. Sainani KL. The Problem with ”Magnitude-Based Infer-

ence”. Med Sci Sports Exerc 2018

7. Mengersen KL, Drovandi CC, Robert CP, et al. Bayesian

Estimation of Small Eﬀects in Exercise and Sports Science.

PLoS One 2016;11(4):e0147311.

8. Lakens D. Equivalence Tests: A Practical Primer for t

Tests, Correlations, and Meta-Analyses. Soc Psychol Personal

Sci 2017;8(4):355-62.

9. Hopkins WG, Batterham AM. An imaginary Bayesian

monster. Int J Sports Physiol Perform 2008;3(4):411-2.

10. Batterham AM, Hopkins WG. The case for magnitude-

based inference. Med Sci Sports Exerc 2015;47(4):885.

11. Hopkins WG, Batterham AM. Magnitude-based inference

under attack 2014 [Available from: sportsci.org accessed 28-2-

15.

12. Hopkins WG, Batterham AM. Error Rates, Decisive Out-

comes and Publication Bias with Several Inferential Methods.

Sports Med 2016;46(10):1563-73.

13. Hopkins WG, Batterham AM. The Vindication of

Magnitude-Based Inference (draft 2) 2018 [Available from:

sportsci.org/2018/mbivind.htm accessed 16-5- 2018.

14. Kruschke JK. Rejecting or Accepting Parameter Values

in Bayesian Estimation. Advances in Methods and Practices

in Psychological Science: 251524591877130, 2018.

Copyright: The articles published on Science Performance and Science

Reports are distributed under the terms of the Creative Commons Attribu-

tion 4.0 International License (http://creativecommons.org/licenses/by/4.0/),

which permits unrestricted use, distribution, and reproduction in any medium,

provided you give appropriate credit to the original author(s) and the

source, provide a link to the Creative Commons license, and indicate if

changes were made. The Creative Commons Public Domain Dedication

waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the

data made available in this article, unless otherwise stated.

sportperfsci.com 2 SPSR - 2018 |May |x|v1