ArticlePDF Available
No evidence for nudging after adjusting for publication bias
Maximilian Maier
a,1,2
, Franti
sek Barto
s
b,1
, T. D. Stanley
c,d
, David R. Shanks
a
, Adam J. L. Harris
a
, and
Eric-Jan Wagenmakers
b
Thaler and Sunsteinsnudge(1) has spawned a revolution
in behavioral science research. Despite its popularity, the
nudge approachhas been criticized for having a limited
evidence base(e.g., ref. 2). Mertens et al. (3) seek to
address that limitation with a timely and comprehensive
metaanalysis. Mertens et al.s headline nding is that
choice architecture [nudging] is an effective and widely
applicable behavior change tool(p. 8). We propose their
nding of moderate publication bias(p. 1) is the real
headline; when this publication bias is appropriately cor-
rected for, no evidence for the effectiveness of nudges
remains (Fig. 1).
Mertens et al. (3) nd signicant publication bias,
through Egger regression. Their sensitivity analysis (4)
indicates that the true effect size could be as low as
d=0.08 (if publication bias is severe). Mertens et al.
argue that severe publication bias is only partially sup-
ported by the funnel plot and proceed largely without
taking publication bias into account in their subsequent
analyses. However, the reported Egger coefcient (b=2.10) is
severe(5).
A newly proposed bias correction technique, robust Bayes-
ian metaanalysis (RoBMA) (6), avoids an all-or-none debate
over whether or not publication bias is severe.RoBMA
simultaneously applies 1) selection models that estimate rela-
tive publication probabilities (7) and 2) models of the relation-
ship between effect sizes and SEs [i.e., Precision Effect Test
and Precision Effect Estimate with Standard Error (6, 8, 9)].
Multimodel inference is then guided mostly by those models
that predict the observed data best (6, 9, 10). RoBMA makes
multimodel inferences about the presence or absence of an
effect, heterogeneity, and publication bias (6, 9).
Table 1 compares the unadjusted results to the publica-
tion biasadjusted results.* Since publication biascorrected
three-level selection models are computationally intracta-
ble, we analyzed the data in two ways: 1) ignoring the three-
level structure (column 2) and 2) using only the most
precise estimate from studies with multiple results (column
3). Strikingly, there is an absence of evidence for an overall
effect and evidence against an effect in the information
and assistanceintervention categories, whereas the evi-
dence is undecided for structureinterventions. When
using only the most precise estimates, we further nd evi-
dence against an effect in most of the domains, apart from
other,”“food,and prosocial(the evidence is indecisive)
and weak evidence for the overall effect.
However, all inter-
vention categories and domains apart from nanceshow
evidence for heterogeneity, which implies that some nudges
might be effective, even when there is evidence against the
−0.4 −0.2 0.0 0.2 0.4
Cohen's d
Combined 0.04 [0.00, 0.14]
BF01 =0.95
Intervention
category:
Information
Structure
Assistance
0.00 [0.00, 0.00]
BF01 =33.84
0.12 [0.00, 0.43]
BF01 =1.12
0.01 [0.00, 0.07]
BF01 =9.05
Domain:
Health
Food
Environment
Finance
Pro−social
Other
0.01 [0.00, 0.10]
BF01 =8.98
0.02 [−0.09, 0.32]
BF01 =5.16
0.01 [−0.18, 0.25]
BF01 =4.41
0.00 [0.00, 0.00]
BF01 =41.23
0.00 [0.00, 0.05]
BF01 =11.93
0.08 [0.00, 0.33]
BF01 =1.38
Fig. 1. RoBMA
PSMA
model-averaged posterior mean effect size estimates
with 95%credible intervals and Bayes factors for the absence of the effect
for the combined sample or split by either the domain or intervention cate-
gory (ignoring the clustering of SEs). BF
01
quanties evidence for the null
hypothesis. BF
01
larger than one corresponds to evidence in favor of the
null hypothesis, and BF
01
lower than one corresponds to evidence in favor
of the alternative hypothesis (evidence for the alternative hypothesis can
be obtained by reciprocating the Bayes factor; BF
10
=1/BF
01
). As a rule of
thumb, Bayes factors between 3 and 10 indicate moderate evidence, and
Bayes factors larger than 10 indicate strong evidence.
Author afliations:
a
Department of Experimental Psychology, University College London,
London WC1H 0AP, United Kingdom;
b
Department of Psychological Methods, University
of Amsterdam, Amsterdam, 1018 WS, The Netherlands;
c
Deakin Laboratory for the Meta-
Analysis of Research, Deakin University, Burwood, VIC 3125, Australia; and
d
Department
of Economics, Deakin University, Burwood, VIC 3125, Australia
Author contributions: M.M. and F.B. designed research; M.M. and F.B. performed
research; M.M. and F.B. analyzed data; and M.M., F.B., T.D.S., D.R.S., A.J.L.H., and E.-J.W.
wrote the paper.
The authors declare no competing interest.
Copyright © 2022 the Author(s). Published by PNAS. This article is distributed under
Creative Commons Attribution License 4.0 (CC BY).
1
M.M. and F.B. contributed equally to this work.
2
To whom correspondence may be addressed. Email: m.maier@ucl.ac.uk.
Published July 19, 2022.
*Our analysis is based on the corrected dataset in ref. 12.
We also reanalyzed the data by including only models of selection for statistical signifi-
cance, confirming our results.
PNAS 2022 Vol. 119 No. 31 e2200300119 https://doi.org/10.1073/pnas.2200300119 1of2
LETTE
R
Downloaded from https://www.pnas.org by Universiteit van Amsterdam on July 25, 2022 from IP address 145.109.101.128.
mean effect. Finally, we nd strong evidence for publication
bias across all subdomains (BF
pb
>10), apart from food,
when using only the most precise estimates (BF
pb
=2.49).
We conclude that the nudgeliterature analyzed in
ref. 3 is characterized by severe publication bias. Contrary
to Mertens et al. (3), our Bayesian analysis indicates that,
after correcting for this bias, no evidence remains that
nudges are effective as tools for behaviour change.
Data Availability. Data and analysis script are available in ref. 11.
ACKNOWLEDGMENTS. We thank Mertens et al. for sharing well-documented
data and code.
1. R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions about Health, Wealth, and Happiness (Yale University Press, 2008).
2. Y. Lin, M. Osman, R. Ashcroft, Nudge: Concept, effectiveness, and ethics. Basic Appl. Soc. Psych. 39, 293306 (2017).
3. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proc. Natl. Acad. Sci. U.S.A. 119, e2107346118
(2022).
4. J. L. Vevea, C. M. Woods, Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychol. Methods 10, 428443 (2005).
5. C. Doucouliagos, T. D. Stanley, Theory competition and selectivity: Are all economic facts greatly exaggerated? J. Econ. Surv. 27, 316339 (2013).
6. M. Maier, F. Barto
s, E. J. Wagenmakers, Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychol. Methods, 10.1037/met0000405 (2022).
7. J. L. Vevea, L. V. Hedges, A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419435 (1995).
8. T. D. Stanley, H. Doucouliagos, Meta-regression approximations to reduce publication selection bias. Res. Synth. Methods 5,6078 (2014).
9. F. Barto
s, M. Maier, E.-J. Wagenmakers, H. Doucouliagos, T. D. Stanley, No need to choose: Model-averaging across complementary publication bias adjustment methods. Evidence Synthesis Methods, in press.
10. J. A. Hoeting, D. Madigan, A. E. Raftery, C. T. Volinsky, Bayesian model averaging: A tutorial. Stat. Sci. 14, 382417 (1999).
11. S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, mhhb_nma_data_corrected.csv. Open Science Framework. https://osf.io/ubt9a/. Accessed 3 May 2022.
12. M. Maier et al., Code and data for analyses in No evidence for nudging after adjusting for publication bias.Open Science Framework. https://osf.io/svz6e/. Deposited 6 January 2022.
Table 1. Comparison of unadjusted and adjusted effect size estimates for all studies and for subsets of
studies based on different categories or domains
Random effects RoBMA
PSMA
RoBMA
PSMA
(precise)
Combined 0.43 [0.38, 0.48] 0.04 [0.00, 0.14] 0.11 [0.00, 0.24]
t(333) =16.51 BF
01
=0.95 BF
01
=0.31
Intervention category
Information 0.25 [0.19, 0.30] 0.00 [0.00, 0.00] 0.00 [0.00, 0.07]
t(88) =8.79 BF
01
=33.84 BF
01
=10.57
Structure 0.58 [0.50, 0.66] 0.12 [0.00, 0.43] 0.23 [0.00, 0.49]
t(186) =13.93 BF
01
=1.12 BF
01
=0.33
Assistance 0.22 [0.15, 0.29] 0.01 [0.00, 0.07] 0.01 [0.00, 0.12]
t(65) =6.42 BF
01
=9.05 BF
01
=8.00
Domain
Health 0.31 [0.22, 0.39] 0.01 [0.00, 0.10] 0.02 [0.00, 0.19]
t(64) =7.03 BF
01
=8.98 BF
01
=3.53
Food 0.66 [0.52, 0.81] 0.02 [0.09, 0.32] 0.27 [0.00, 0.64]
t(81) =9.01 BF
01
=5.16 BF
01
=0.55
Environment 0.48 [0.37, 0.58] 0.01 [0.18, 0.25] 0.00 [0.44, 0.34]
t(56) =9.16 BF
01
=4.41 BF
01
=3.05
Finance 0.23 [0.15, 0.31] 0.00 [0.00, 0.00] 0.00 [0.00, 0.00]
t(34) =6.08 BF
01
=41.23 BF
01
=30.95
Prosocial 0.32 [0.22, 0.42] 0.00 [0.00, 0.05] 0.05 [0.00, 0.27]
t(38) =6.36 BF
01
=11.93 BF
01
=1.89
Other 0.40 [0.29, 0.50] 0.08 [0.00, 0.33] 0.04 [0.22, 0.40]
t(55) =7.66 BF
01
=1.38 BF
01
=2.45
First column: Random effects metaanalysis estimates with 95% CI based on clustered SEs, all Pvalues <0.001. Second and
third columns: RoBMA
PSMA
model-averaged posterior mean effect size estimates with 95% credible intervals and Bayes
factor for the presence of the effect ignoring the clustering of SEs or using the most precise estimates (precise). Results
differ slightly from the moderator analysis presented in the article because we analyzed each subeld separately to
allow 1) testing for the presence of the effect in each category/domain in the Bayesian framework, and 2) publication
bias to operate differently in different subdomains.
2of2 https://doi.org/10.1073/pnas.2200300119 pnas.org
Downloaded from https://www.pnas.org by Universiteit van Amsterdam on July 25, 2022 from IP address 145.109.101.128.
... Mertens et al.'s (2022) meta-analysis on nudges across application domains shows that nudges usually produce modest effects. Yet, concerns about publication bias-the tendency for studies with significant or positive results to be published more often than those with null or negative results-suggest that even these results may be exaggerated (Bakdash & Marusich, 2022;Maier et al., 2022;Szaszi et al., 2022). Maier et al.'s (2024) inspection of studies conducted by two prominent "Nudge Units" reveals that, despite preregistration protocols, the protocols did not safeguard against analytic flexibility. ...
... Using a meta-analysis of 212 studies with 447 effect sizes, Mertens et al. (2022) conclude that the nudges' mean effect sizes are small to medium (d = 0.43). However, re-analyses of the authors' publicly available data lead other scholars to suggest that this effect could be overstated because of publication bias (Bakdash & Marusich, 2022;Maier et al., 2022;Szaszi et al., 2022). Publication bias is a systemic issue identified in the fields of psychology, medicine, economics, environmental sciences, and IS (Bartoš et al., 2023a;Fanelli et al., 2017;Kepes & Thomas, 2018). ...
... However, the apparent efficacy of digital nudges reported in the literature may be subject to biases inherent in scholarly publication (Bakdash & Marusich, 2022;Maier et al., 2022;Szaszi et al., 2022). Publication outlets often tend to publish positive results, potentially leading to overestimating the effectiveness of nudges. ...
Conference Paper
Digital nudges influence people's decisions while preserving their freedom of choice. This publication bias-adjusted, robust Bayesian meta-analysis evaluates the impact of digital nudges on pro-environmental behavior. We analyzed 159 effect estimates from more than a million observations across 67 studies of 58 publications. The findings indicate no significant average effect of nudge interventions on behavioral outcomes while highlighting a pronounced publication bias. Despite a mean indistinguishable from zero, the substantial variability among the studies suggests that some nudges may be effective. However, our examination of potential moderators, such as the nudge category and nudge personalization, shows that these factors do not significantly account for the observed variability. This study offers a cautionary note about the limitations of the current green nudging literature. It underscores the necessity for more rigorous research methodologies, including registered reports and high-powered replications, to determine which nudges influence sustainable behaviors in what contexts.
... This draws on insights from behavioural economics that small changes can have big effects. [32][33][34] A large and expensive intervention may not be effective or financially feasible. Instead, it is usually more effective, and cost-effective, to begin with a smaller intervention, such as one drawing on a local quality improvement initiative. ...
Article
Full-text available
Surgeons, anaesthetists, wider surgical teams and hospital managers are a large global group that has the capacity and power to play a leadership role to contribute to change. Hospitals are a good target for improvement since they are centres of communities, linking together surrounding healthcare facilities and influencing wider determinants of the environment. District and rural hospitals are good sites to start since they serve large populations, have the least sustained energy and clean water supplies and will benefit most from quality improvement. Within hospitals, surgeons and surgical pathways are the ideal places to start decarbonising healthcare. Surgery is a high-resource activity, but it focuses on one patient at a time, allowing measures to be introduced, and their effects closely monitored. Through a mass movement, surgical teams should be able to influence policy-makers for healthcare and industry supply chains, amplifying their effect. This article describes how we can make personal, professional and organisational changes to start creating impact. Change can be hard, especially in healthcare, so this new community needs to blend carbon literacy and behavioural change techniques for success. The article is focused on the front-line team and written by clinician experts in behavioural change and sustainable practice. As such, it will not tackle the technicalities of sustainability and carbon accounting. It intends to challenge individual readers to start making changes now, and to challenge systems leaders to start making larger-scale changes urgently.
... In a comprehensive comparison between 'nudge units' maintained by governments to implement nudges and academic research that evaluates nudges, DellaVigna and Linos (2022) find that effect sizes observed in studies conducted by the former are typically dwarfed by those found in studies conducted by the latter. The larger effects promised by academic research have not materialized when put into practice by governments, with the difference potentially attributable to publication bias (DellaVigna and Linos, 2022;Maier et al., 2022). To be sure, nudges surely sometimes generate improved outcomes, such as increasing vaccination rates (Milkman et al., 2022) and participation in retirement plans (Madrian and Shea, 2001). ...
Article
Full-text available
To what extent can the harms of misinformation be mitigated by relying on nudges? Prior research has demonstrated that non-intrusive ‘accuracy nudges’ can reduce the sharing of misinformation. We investigate an alternative approach. Rather than subtly reminding people about accuracy, our intervention, indebted to research on the bystander effect, explicitly appeals to individuals' capacity to help solve the misinformation challenge. Our results are mixed. On the one hand, our intervention reduces the willingness to share and believe in misinformation fact-checked as false. On the other hand, it also reduces participants' willingness to share information that has been fact-checked as true and innocuous, as well as non-fact-checked information. Experiment 1 offers proof of concept; Experiment 2 tests our intervention with a more realistic mix of true and false social media posts; Experiment 3 tests our interventions alongside an accuracy nudge. The effectiveness of our intervention at reducing willingness to share misinformation remains consistent across experiments; meta-analysis reveals that our treatment reduced willingness to share false content across experiments by 20% of a scale point on a six-point scale. We do not observe the accuracy nudge reducing willingness to share false content. Taken together, these results highlight the advantages and disadvantages of accuracy nudges and our more confrontational approach.
... Finally, it should be specified that not finding statistically significant results is not a limitation. In recent decades, authors have increasingly recognized the detrimental ramifications of academic outlets favoring studies that produce significant results (Brodeur et al., 2020;Maier et al., 2022). This practice causes spurious effects to be more likely to be published. ...
Article
Full-text available
Limbal rings are dark bands in the eyes that circle the iris. Previous research suggests that the presence of limbal rings augments perceptions of male faces as healthy and attractive, particularly among female perceivers. Nonetheless, a degree of heterogeneity exists in attractiveness effects across different stimulus sets. One possibility of this discrepancy could be previously undetected differences in attractiveness across different social targets that impede a clearer understanding for the signal value of limbal rings. Namely, it could be possible that the attractiveness advantage of faces with limbal rings is most apparent among highly attractive stimuli. This study sought to reconcile this discrepancy by developing a novel stimulus set that systematically varies the attractiveness of social targets in addition to the presence of limbal rings in male faces. Women evaluated these targets based on perceptions of their health and attractiveness. Although limbal rings continued to augment perceptions of health in male faces, no differences emerged in the attractiveness of faces with and without limbal rings.
Article
Choice architecture refers to features of the psychological, social, and physical environment that can affect the process and outcome of human decisions. With roots in cognitive psychology and behavioral economics, it has become a fertile topic of inquiry in a wide range of behavioral domains. Thaler and Sunstein (2008) popularized choice architecture and its concept of nudges, which are small environmental adjustments that can facilitate behavioral choices without limiting personal autonomy. Nudge interventions have been studied intensely in relation to health, financial, ecological, and consumer outcomes. In this article, I consider how the strategic use of digital nudges might be used in career interventions, particularly in the context of a proactive, large-scale service delivery model. After overviewing the efficacy of nudge interventions in other literatures (e.g., educational decision-making), I suggest an agenda for designing and testing career development nudges. Though they often yield modest effect sizes, nudges may have potential as a cost-effective addition to the career intervention repertoire, providing both a means of assisting those who have been underserved by traditional career services and an outreach bridge between technology- and counselor-based interventions.
Article
Objective To review and synthesise research on technological debiasing strategies across domains, present a novel distributed cognition-based classification system, and discuss theoretical implications for the field. Background Distributed cognition theory is valuable for understanding and mitigating cognitive biases in high-stakes settings where sensemaking and problem-solving are contingent upon information representations and flows in the decision environment. Shifting the focus of debiasing from individuals to systems, technological debiasing strategies involve designing system components to minimise the negative impacts of cognitive bias on performance. To integrate these strategies into real-world practices effectively, it is imperative to clarify the current state of evidence and types of strategies utilised. Methods We conducted systematic searches across six databases. Following screening and data charting, identified strategies were classified into (i) group composition and structure, (ii) information design and (iii) procedural debiasing, based on distributed cognition principles, and cognitive biases, classified into eight categories. Results Eighty articles met the inclusion criteria, addressing 100 debiasing investigations and 91 cognitive biases. A majority (80%) of the identified debiasing strategies were reportedly effective, whereas fourteen were ineffective and six were partially effective. Information design strategies were studied most, followed by procedural debiasing, and group structure and composition. Gaps and directions for future work are discussed. Conclusion Through the lens of distributed cognition theory, technological debiasing represents a reconceptualisation of cognitive bias mitigation, showing promise for real-world application. Application The study results and debiasing classification presented can inform the design of high-stakes work systems to support cognition and minimise judgement errors.
Article
The UK Behavioural Insights Team transformed nudging and behavioural economics from nascent ideas to key policy tools for the UK Coalition Government. This article argues that political economic circumstances significantly contributed to the success of this ‘nudge’ programme. The Global Financial Crisis (GFC) created a ‘contest of authority’ over dominant policy approaches. By framing the crisis as a crisis of rationality, behavioural perspectives gained political support. The GFC also saw that the UK Government (from 2010) adopt a programme of fiscal austerity. Nudging complemented this programme by suggesting effective policy could be made cheaply. Using various accounts of nudging in the UK from those involved in its development, we demonstrate the role of the country’s political economy in the behavioural turn. We conclude by reflecting on the role of behavioural insights today, given a political–economic landscape much changed since 2010.
Article
Full-text available
Publication bias is a ubiquitous threat to the validity of meta‐analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition‐dependent, all‐or‐none choice between competing methods and conflicting results, we extend robust Bayesian meta‐analysis and model‐average across two prominent approaches of adjusting for publication bias: (1) selection models of p‐values and (2) models adjusting for small‐study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi‐lab replications demonstrate the benefits of Bayesian model‐averaging of complementary publication bias adjustment methods. This article is protected by copyright. All rights reserved.
Article
Full-text available
Meta-analysis is an essential tool to synthesize information from a series of primary studies. However, the application of meta-analysis is often frustrated by publication bias -- the fact that statistically significant results are published more often than nonsignificant results. To alleviate the problem of publication bias we developed a robust Bayesian meta-analysis (RoBMA). RoBMA applies a series of meta-analytic models to the data simultaneously and estimates the effect size by taking all models into account. RoBMA can quantify evidence for the presence as well as the absence of publication bias, RoBMA can correct for publication bias in cases where the true effect effect size differs between studies, and RoBMA does not require all-or-none decisions. We illustrate RoBMA with a meta-analysis on violent video games and aggressive behavior, and apply it to the ManyLabs 2 data for which we know publication bias to be absent. Simulations suggest that RoBMA provides a valuable complement to current methods for meta-analysis.
Article
Full-text available
Significance Changing individuals’ behavior is key to tackling some of today’s most pressing societal challenges such as the COVID-19 pandemic or climate change. Choice architecture interventions aim to nudge people toward personally and socially desirable behavior through the design of choice environments. Although increasingly popular, little is known about the overall effectiveness of choice architecture interventions and the conditions under which they facilitate behavior change. Here we quantitatively review over a decade of research, showing that choice architecture interventions successfully promote behavior change across key behavioral domains, populations, and locations. Our findings offer insights into the effects of choice architecture and provide guidelines for behaviorally informed policy making.
Article
Full-text available
Nudges are psychologically informed tools designed to promote behavioral change in order to improve health and well-being. In this review, we focus on three areas of concern: theory, evidence base, ethics. We begin by discussing the problems arising from the theoretical framework that nudges are based on and propose an alternative framework that helps to classify nudges into two types (Type 1 and Type 2). We then evaluate the evidence for nudges in the health domain, drawing attention to critical empirical issues (internal and external reliability) that explain the limited evidence base for their effectiveness. The review ends with an examination of the implications of the theoretical and empirical issues we discussed with respect to current debates regarding the ethics of nudge.
Book
Full-text available
Every day, we make decisions on topics ranging from personal investments to schools for our children to the meals we eat to the causes we champion. Unfortunately, we often choose poorly. The reason, the authors explain, is that, being human, we all are susceptible to various biases that can lead us to blunder. Our mistakes make us poorer and less healthy; we often make bad decisions involving education, personal finance, health care, mortgages and credit cards, the family, and even the planet itself. Thaler and Sunstein invite us to enter an alternative world, one that takes our humanness as a given. They show that by knowing how people think, we can design choice environments that make it easier for people to choose what is best for themselves, their families, and their society. Using colorful examples from the most important aspects of life, Thaler and Sunstein demonstrate how thoughtful "choice architecture" can be established to nudge us in beneficial directions without restricting freedom of choice. Nudge offers a unique new take-from neither the left nor the right-on many hot-button issues, for individuals and governments alike. This is one of the most engaging and provocative books to come along in many years. © 2008 by Richard H. Thaler and Cass R. Sunstein. All rights reserved.
Article
Full-text available
Publication bias, sometimes known as the "file-drawer problem" or "funnel-plot asymmetry," is common in empirical research. The authors review the implications of publication bias for quantitative research synthesis (meta-analysis) and describe existing techniques for detecting and correcting it. A new approach is proposed that is suitable for application to meta-analytic data sets that are too small for the application of existing methods. The model estimates parameters relevant to fixed-effects, mixed-effects or random-effects meta-analysis contingent on a hypothetical pattern of bias that is fixed independently of the data. The authors illustrate this approach for sensitivity analysis using 3 data sets adapted from a commonly cited reference work on research synthesis (H. M. Cooper & L. V. Hedges, 1994).
Article
Abstract There is growing concern and mounting evidence of selectivity in empirical economics. Most empirical economic literatures have a truncated distribution of results. The aim of this paper is to explore the link between publication selectivity and theory contests. This link is confirmed through the analysis of 87 distinct empirical economics literatures, involving more than three and a half thousand separate empirical studies, using objective measures of both selectivity and contests. Our meta–meta-analysis shows that publication selection is widespread, but not universal. It distorts scientific inference with potentially adverse effects on policy making, but competition and debate between rival theories reduces this selectivity and thereby improves economic inference.
Article
Publication selection bias is a serious challenge to the integrity of all empirical sciences. We derive meta-regression approximations to reduce this bias. Our approach employs Taylor polynomial approximations to the conditional mean of a truncated distribution. A quadratic approximation without a linear term, precision-effect estimate with standard error (PEESE), is shown to have the smallest bias and mean squared error in most cases and to outperform conventional meta-analysis estimators, often by a great deal. Monte Carlo simulations also demonstrate how a new hybrid estimator that conditionally combines PEESE and the Egger regression intercept can provide a practical solution to publication selection bias. PEESE is easily expanded to accommodate systematic heterogeneity along with complex and differential publication selection bias that is related to moderator variables. By providing an intuitive reason for these approximations, we can also explain why the Egger regression works so well and when it does not. These meta-regression methods are applied to several policy-relevant areas of research including antidepressant effectiveness, the value of a statistical life, the minimum wage, and nicotine replacement therapy. Copyright © 2013 John Wiley & Sons, Ltd. Copyright © 2013 John Wiley & Sons, Ltd.
Article
Standard statistical practice ignores model uncertainty. Data analysts typically select a model from some class of models and then proceed as if the selected model had generated the data. This approach ignores the uncertainty in model selection, leading to over-confident inferences and decisions that are more risky than one thinks they are. Bayesian model averaging (BMA)provides a coherent mechanism for accounting for this model uncertainty. Several methods for implementing BMA have recently emerged. We discuss these methods and present a number of examples.In these examples, BMA provides improved out-of-sample predictive performance. We also provide a catalogue of currently available BMA software.
Article
When the process of publication favors studies with smallp-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailedp-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.