Content uploaded by František Bartoš
Author content
All content in this area was uploaded by František Bartoš on Jul 25, 2022
Content may be subject to copyright.
No evidence for nudging after adjusting for publication bias
Maximilian Maier
a,1,2
, Franti
sek Barto
s
b,1
, T. D. Stanley
c,d
, David R. Shanks
a
, Adam J. L. Harris
a
, and
Eric-Jan Wagenmakers
b
Thaler and Sunstein’s“nudge”(1) has spawned a revolution
in behavioral science research. Despite its popularity, the
“nudge approach”has been criticized for having a “limited
evidence base”(e.g., ref. 2). Mertens et al. (3) seek to
address that limitation with a timely and comprehensive
metaanalysis. Mertens et al.’s headline finding is that
“choice architecture [nudging] is an effective and widely
applicable behavior change tool”(p. 8). We propose their
finding of “moderate publication bias”(p. 1) is the real
headline; when this publication bias is appropriately cor-
rected for, no evidence for the effectiveness of nudges
remains (Fig. 1).
Mertens et al. (3) find significant publication bias,
through Egger regression. Their sensitivity analysis (4)
indicates that the true effect size could be as low as
d=0.08 (if publication bias is severe). Mertens et al.
argue that severe publication bias is only partially sup-
ported by the funnel plot and proceed largely without
taking publication bias into account in their subsequent
analyses. However, the reported Egger coefficient (b=2.10) is
“severe”(5).
A newly proposed bias correction technique, robust Bayes-
ian metaanalysis (RoBMA) (6), avoids an all-or-none debate
over whether or not publication bias is “severe.”RoBMA
simultaneously applies 1) selection models that estimate rela-
tive publication probabilities (7) and 2) models of the relation-
ship between effect sizes and SEs [i.e., Precision Effect Test
and Precision Effect Estimate with Standard Error (6, 8, 9)].
Multimodel inference is then guided mostly by those models
that predict the observed data best (6, 9, 10). RoBMA makes
multimodel inferences about the presence or absence of an
effect, heterogeneity, and publication bias (6, 9).
Table 1 compares the unadjusted results to the publica-
tion bias–adjusted results.* Since publication bias–corrected
three-level selection models are computationally intracta-
ble, we analyzed the data in two ways: 1) ignoring the three-
level structure (column 2) and 2) using only the most
precise estimate from studies with multiple results (column
3). Strikingly, there is an absence of evidence for an overall
effect and evidence against an effect in the “information”
and “assistance”intervention categories, whereas the evi-
dence is undecided for “structure”interventions. When
using only the most precise estimates, we further find evi-
dence against an effect in most of the domains, apart from
“other,”“food,”and “prosocial”(the evidence is indecisive)
and weak evidence for the overall effect.
†
However, all inter-
vention categories and domains apart from “finance”show
evidence for heterogeneity, which implies that some nudges
might be effective, even when there is evidence against the
−0.4 −0.2 0.0 0.2 0.4
Cohen's d
Combined 0.04 [0.00, 0.14]
BF01 =0.95
Intervention
category:
Information
Structure
Assistance
0.00 [0.00, 0.00]
BF01 =33.84
0.12 [0.00, 0.43]
BF01 =1.12
0.01 [0.00, 0.07]
BF01 =9.05
Domain:
Health
Food
Environment
Finance
Pro−social
Other
0.01 [0.00, 0.10]
BF01 =8.98
0.02 [−0.09, 0.32]
BF01 =5.16
0.01 [−0.18, 0.25]
BF01 =4.41
0.00 [0.00, 0.00]
BF01 =41.23
0.00 [0.00, 0.05]
BF01 =11.93
0.08 [0.00, 0.33]
BF01 =1.38
Fig. 1. RoBMA
PSMA
model-averaged posterior mean effect size estimates
with 95%credible intervals and Bayes factors for the absence of the effect
for the combined sample or split by either the domain or intervention cate-
gory (ignoring the clustering of SEs). BF
01
quantifies evidence for the null
hypothesis. BF
01
larger than one corresponds to evidence in favor of the
null hypothesis, and BF
01
lower than one corresponds to evidence in favor
of the alternative hypothesis (evidence for the alternative hypothesis can
be obtained by reciprocating the Bayes factor; BF
10
=1/BF
01
). As a rule of
thumb, Bayes factors between 3 and 10 indicate moderate evidence, and
Bayes factors larger than 10 indicate strong evidence.
Author affiliations:
a
Department of Experimental Psychology, University College London,
London WC1H 0AP, United Kingdom;
b
Department of Psychological Methods, University
of Amsterdam, Amsterdam, 1018 WS, The Netherlands;
c
Deakin Laboratory for the Meta-
Analysis of Research, Deakin University, Burwood, VIC 3125, Australia; and
d
Department
of Economics, Deakin University, Burwood, VIC 3125, Australia
Author contributions: M.M. and F.B. designed research; M.M. and F.B. performed
research; M.M. and F.B. analyzed data; and M.M., F.B., T.D.S., D.R.S., A.J.L.H., and E.-J.W.
wrote the paper.
The authors declare no competing interest.
Copyright © 2022 the Author(s). Published by PNAS. This article is distributed under
Creative Commons Attribution License 4.0 (CC BY).
1
M.M. and F.B. contributed equally to this work.
2
To whom correspondence may be addressed. Email: m.maier@ucl.ac.uk.
Published July 19, 2022.
*Our analysis is based on the corrected dataset in ref. 12.
†
We also reanalyzed the data by including only models of selection for statistical signifi-
cance, confirming our results.
PNAS 2022 Vol. 119 No. 31 e2200300119 https://doi.org/10.1073/pnas.2200300119 1of2
LETTE
R
Downloaded from https://www.pnas.org by Universiteit van Amsterdam on July 25, 2022 from IP address 145.109.101.128.
mean effect. Finally, we find strong evidence for publication
bias across all subdomains (BF
pb
>10), apart from food,
when using only the most precise estimates (BF
pb
=2.49).
We conclude that the “nudge”literature analyzed in
ref. 3 is characterized by severe publication bias. Contrary
to Mertens et al. (3), our Bayesian analysis indicates that,
after correcting for this bias, no evidence remains that
nudges are effective as tools for behaviour change.
Data Availability. Data and analysis script are available in ref. 11.
ACKNOWLEDGMENTS. We thank Mertens et al. for sharing well-documented
data and code.
1. R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions about Health, Wealth, and Happiness (Yale University Press, 2008).
2. Y. Lin, M. Osman, R. Ashcroft, Nudge: Concept, effectiveness, and ethics. Basic Appl. Soc. Psych. 39, 293–306 (2017).
3. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proc. Natl. Acad. Sci. U.S.A. 119, e2107346118
(2022).
4. J. L. Vevea, C. M. Woods, Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychol. Methods 10, 428–443 (2005).
5. C. Doucouliagos, T. D. Stanley, Theory competition and selectivity: Are all economic facts greatly exaggerated? J. Econ. Surv. 27, 316–339 (2013).
6. M. Maier, F. Barto
s, E. J. Wagenmakers, Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychol. Methods, 10.1037/met0000405 (2022).
7. J. L. Vevea, L. V. Hedges, A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995).
8. T. D. Stanley, H. Doucouliagos, Meta-regression approximations to reduce publication selection bias. Res. Synth. Methods 5,60–78 (2014).
9. F. Barto
s, M. Maier, E.-J. Wagenmakers, H. Doucouliagos, T. D. Stanley, No need to choose: Model-averaging across complementary publication bias adjustment methods. Evidence Synthesis Methods, in press.
10. J. A. Hoeting, D. Madigan, A. E. Raftery, C. T. Volinsky, Bayesian model averaging: A tutorial. Stat. Sci. 14, 382–417 (1999).
11. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, mhhb_nma_data_corrected.csv. Open Science Framework. https://osf.io/ubt9a/. Accessed 3 May 2022.
12. M. Maier et al., Code and data for analyses in “No evidence for nudging after adjusting for publication bias.”Open Science Framework. https://osf.io/svz6e/. Deposited 6 January 2022.
Table 1. Comparison of unadjusted and adjusted effect size estimates for all studies and for subsets of
studies based on different categories or domains
Random effects RoBMA
PSMA
RoBMA
PSMA
(precise)
Combined 0.43 [0.38, 0.48] 0.04 [0.00, 0.14] 0.11 [0.00, 0.24]
t(333) =16.51 BF
01
=0.95 BF
01
=0.31
Intervention category
Information 0.25 [0.19, 0.30] 0.00 [0.00, 0.00] 0.00 [0.00, 0.07]
t(88) =8.79 BF
01
=33.84 BF
01
=10.57
Structure 0.58 [0.50, 0.66] 0.12 [0.00, 0.43] 0.23 [0.00, 0.49]
t(186) =13.93 BF
01
=1.12 BF
01
=0.33
Assistance 0.22 [0.15, 0.29] 0.01 [0.00, 0.07] 0.01 [0.00, 0.12]
t(65) =6.42 BF
01
=9.05 BF
01
=8.00
Domain
Health 0.31 [0.22, 0.39] 0.01 [0.00, 0.10] 0.02 [0.00, 0.19]
t(64) =7.03 BF
01
=8.98 BF
01
=3.53
Food 0.66 [0.52, 0.81] 0.02 [0.09, 0.32] 0.27 [0.00, 0.64]
t(81) =9.01 BF
01
=5.16 BF
01
=0.55
Environment 0.48 [0.37, 0.58] 0.01 [0.18, 0.25] 0.00 [0.44, 0.34]
t(56) =9.16 BF
01
=4.41 BF
01
=3.05
Finance 0.23 [0.15, 0.31] 0.00 [0.00, 0.00] 0.00 [0.00, 0.00]
t(34) =6.08 BF
01
=41.23 BF
01
=30.95
Prosocial 0.32 [0.22, 0.42] 0.00 [0.00, 0.05] 0.05 [0.00, 0.27]
t(38) =6.36 BF
01
=11.93 BF
01
=1.89
Other 0.40 [0.29, 0.50] 0.08 [0.00, 0.33] 0.04 [0.22, 0.40]
t(55) =7.66 BF
01
=1.38 BF
01
=2.45
First column: Random effects metaanalysis estimates with 95% CI based on clustered SEs, all Pvalues <0.001. Second and
third columns: RoBMA
PSMA
model-averaged posterior mean effect size estimates with 95% credible intervals and Bayes
factor for the presence of the effect ignoring the clustering of SEs or using the most precise estimates (precise). Results
differ slightly from the moderator analysis presented in the article because we analyzed each subfield separately to
allow 1) testing for the presence of the effect in each category/domain in the Bayesian framework, and 2) publication
bias to operate differently in different subdomains.
2of2 https://doi.org/10.1073/pnas.2200300119 pnas.org
Downloaded from https://www.pnas.org by Universiteit van Amsterdam on July 25, 2022 from IP address 145.109.101.128.