ArticlePDF Available

Comparing Bayesian Posterior Passing with Meta-analysis

Authors:

Abstract

Brand, von der Post, Ounsley, and Morgan (2019) introduced Bayesian posterior passing as an alternative to traditional meta-analyses. In this commentary I relate their procedure to traditional meta-analysis, showing that posterior passing is equivalent to fixed effects meta-analysis. To overcome the limitations of simple posterior passing, I introduce improved posterior passing methods to account for heterogeneity and publication bias. Additionally, practical limitations of posterior passing and the role that it can play in future research are discussed.
Meta-Psychology, 2021, vol 5, MP.2020.2539
https://doi.org/10.15626/MP.2020.2539
Article type: Commentary
Published under the CC-BY4.0 license
Open data: Not applicable
Open materials: Not applicable
Open and reproducible analysis: Not applicable
Open reviews and editorial process: Yes
Preregistration: No
Edited by: Rickard Carlsson
Reviewed by: Brand, C.O., Martin, S.R.
Analysis reproduced by: Not applicable
All supplementary files can be accessed at OSF:
https://doi.org/10.17605/OSF.IO/JGXK7
Comparing Bayesian Posterior Passing with
Meta-analysis
Joshua Pritsker
Purdue University
Abstract
Brand, von der Post, Ounsley, and Morgan (2019) introduced Bayesian posterior passing as an alternative to tra-
ditional meta-analyses. In this commentary I relate their procedure to traditional meta-analysis, showing that pos-
terior passing is equivalent to fixed effects meta-analysis. To overcome the limitations of simple posterior passing,
I introduce improved posterior passing methods to account for heterogeneity and publication bias. Additionally,
practical limitations of posterior passing and the role that it can play in future research are discussed.
Keywords: bayesian updating, posterior passing, meta-analysis
Introduction
The ability to accumulate evidence across studies is
often said to be a great advantage of Bayesian infer-
ence. This point was recently discussed by Brand, von
der Post, Ounsley, and Morgan (2019), who suggested
that Bayesian posterior passing alleviates the need for
traditional meta-analysis. They performed a simulation
study comparing posterior passing to non-cumulative
analysis and combined analysis of the data from all
studies. However, they lacked a formal theoretical com-
parison of posterior passing to traditional methods of
meta-analysis. This commentary relates posterior pass-
ing to traditional meta-analyses, allowing for one to de-
termine the performance of posterior passing on the ba-
sis of how traditional meta-analytic methods are known
to perform. To avoid some of the pitfalls of posterior
passing, I suggest improved procedures that account
for heterogeneity and publication bias. I address Brand
et al.’s (2019) suggestion that posterior passing avoids
some of the problems of traditional meta-analyses, and
discuss practical limitations in using it as a replacement
for traditional meta-analysis.
Posterior Passing as Meta-analysis
As Brand et al. (2019) suggest that posterior passing
may replace traditional meta-analyses, one might won-
der how posterior passing relates to traditional meta-
analyses. Given that meta-analyses are typically based
on proxy statistics and their standard errors instead of
individual data points, one might consider posterior dis-
tributions generated the same way. We can do this by
using the likelihood of a statistic instead of the likeli-
hood of the full data. For instance, supposing that we
have a parameter θthat we want to make inferences
about, we might construct the likelihood function using
the sampling distribution of an estimate of θ. In stan-
dard meta-analysis, studies are typically summarized
by their estimates and standard errors. If an estimate
was derived by maximizing a likelihood that takes the
form of gaussian function over θ, the estimate and its
standard error fully summarize this likelihood function.
Even in cases where the likelihood function is not fully
described by its maximum likelihood estimate and stan-
dard error, using the estimate and standard error may
be viewed as a second-order asymptotic approximation
to the log-likelihood.
Taking this approach, with ˆ
θibeing the estimate from
2
study iof θ, we can get the posterior after study i, de-
noted as fi(θ), by:
fi(θ)π(θ)f(xi|θ)(1a)
=fi1(θ)fˆ
θi|θ, SE2
i(1b)
Where π(·)is our prior distribution, xirefers to all
the information of study i, and SEiis the standard er-
ror for ˆ
θi. Typically, fˆ
θi|θ, SE2
i=φˆ
θi|θ, SE2
i, where
φˆ
θi|θ, SE2
iis the density at ˆ
θiunder a normal distri-
bution with a mean of θand variance of SE2
i. Now, we
may expand (1) across studies:
f(θ)π0(θ)
k
Y
i=1
φˆ
θi|θ, SE2
i(2)
Where π0(θ)is the prior distribution used by the ini-
tial study. Hence, the posterior is proportional to the
initial prior times the product of the likelihoods of the
relevant studies. How does this compare to the poste-
rior distribution given by a traditional meta-analysis? In
a fixed-effects setting (3), they are identical, provided
that the meta-analysis uses the same prior as the ini-
tial study. The standard fixed effects model is that each
ˆ
θifollows a normal distribution with a universal mean
value of θ, and a variance of SE2
i:
ˆ
θi∼ N θ, SE2
i(3)
Then, the posterior density can be constructed by tak-
ing the product of the likelihoods of each estimate:
ff ixed (θ|x)π(θ)
k
Y
i=1
φˆ
θi|θ, SE2
i(4)
This equivalence also answers some of the questions
brought up by Brand et al. (2019), such as how pos-
terior passing would perform under publication bias.
Now, it is clear that we can use existing studies on tradi-
tional fixed-effects meta-analyses to answer these ques-
tions (e.g., Simonsohn et al., 2014).
In a random effects setting, we assume that each
study samples from a slightly different population, and
these populations have their own parameter values, re-
ferred to as µi, which vary around the true θvalue:
µi∼ N θ, τ2(5a)
ˆ
θi∼ N µi,SE2
ˆ
θi(5b)
Then, the likelihood for each estimate is given by
marginalizing out µi:
frandom (θ|x)π(θ, τ)
k
Y
i=1Z
µ
φˆ
θi|µi,SE2
i(6a)
×φµi|θ, τ2dµi
=π(θ, τ)
k
Y
i=1
φˆ
θi|θ, SE2
i+τ2(6b)
Where µiis the mean of the population that study
icomes from, τ2is the variance across populations,
and the prior distribution is now over both θand τ.
Hence, in a random-effects setting (4), posterior pass-
ing, like (3), will underestimate the posterior vari-
ance. This point was previously noted by Martin (2017).
A random-effects model is typically preferable, as the
fixed-effects assumption of no between-study variance
is implausible in most settings (Borenstein et al., 2010).
The use of the full-data posterior by Brand et al. (2019)
is inconsequential to this result. Hence, although poste-
rior passing will produce consistent point estimates, the
posterior variance may be underestimated.
How can we improve posterior passing?
Incorporating Random Effects
An obvious question to ask at this point is if we can
modify posterior passing to incorporate random effects.
A Bayesian solution is to update τalong with θ, mod-
eling their joint posterior distribution. Using statistic
likelihoods as in the previous section, we can derive the
joint posterior update by marginalizing out µ, just as in
a standard random-effects meta-analysis:
fi(θ, τ)π(θ, τ)f(xi|θ, τ)(7a)
=fi1(θ, τ)f(xi|θ, τ)(7b)
=fi1(θ, τ)Z
µ
f(xi|µi)φµi|θ, τ2dµi(7c)
=fi1(θ, τ)φˆ
θi|θ, SE2
i+τ2(7d)
To get the marginal posterior distribution of θ, one
may integrate out τ, and vice versa. Notably, the pa-
rameter must be tracked across studies to gain evidence
about its value. However, even if previous studies hadn’t
used a random effects model, they can still be added
using their likelihood functions:
fi(θ, τ)π(θ, τ)φˆ
θi1|θ, SE2
i1+τ2(8)
×φˆ
θi|θ, SE2
i+τ2
3
Where study i1is the study that hadn’t used a ran-
dom effects model. In the case of the current study be-
ing the first on a topic, no evidence will be gained about
the value of τ, but incorporating it using only the infor-
mation from a subjective prior distribution will nonethe-
less give a more realistic view of the uncertainty of fi(θ).
Addressing publication bias
A second major issue with posterior passing is that
Brand et al. (2019) provide no way to address publi-
cation bias. Without adjustment, posterior passing will
perform identical to a fixed-effects meta-analysis that
completely ignores publication bias. Unadjusted meta-
analyses are known to perform poorly when publication
bias is substantiative, yielding potentially misleading re-
sults (Simonsohn et al., 2014). This problem can be
viewed as one of biased sampling, hence the likelihood
function is given by a weighted distribution as follows
(Pfeffermann et al., 1998):
fxi|publishedi=Eip|xif(xi)
Ep(9)
=Eip|xi
R
x
Ep|xif(xi)dxi
f(xi)
Where pare the probabilities of publication. Now,
we need to model Ep|xi. Considering that studies are
typically given a dichotomous interpretation (McShane
& Gal, 2017), a realistic option is a simple step function:
Ep|xi=
αsiˆ
θi=0
βsiˆ
θi=1(10)
Where si(θi)is a function that gives the standard in-
terpretation of ˆ
θiin a pass/fail manner, such as its p-
value being below 0.05. However, it is unclear if di-
chotomization exists to the same extent in Bayesian
studies as it does in frequentist studies. In such a con-
text one may replace (10) with a smoother model, such
as a logistic one:
Ep|xi=logistic α+βsiˆ
θi (11)
Where siˆ
θicould represent a Bayes factor cutoff or
similar. For a review of other models that have been sug-
gested, see Sutton, Song, Gilbody, and Abrams (2000).
In any case, the posterior update is now as follows:
fi(θ, τ, α, β)π(θ, τ, α, β)f(xi|θ, τ, α, β)(12a)
π(θ, τ, α, β)(12b)
×Ep|xi, θ, τ, α, β
Ep|θ, τ, α, βf(xi|θ, τ)
=fi1(θ, τ, α, β)(12c)
×Ep|xi, θ, τ, α, β
Ep|θ, τ, α, β
×Z
µ
f(xi|µi)φµi|θ, τ2dµi
=fi1(θ, τ, α, β)(12d)
×Ep|xi, θ, τ, α, β
Ep|θ, τ, α, β
×φˆ
θi,|θ, SE2
i+τ2
As with τ, multiple studies are needed to identify α,
and β. Hence, inference in early studies will be highly
dependent on the prior distribution for these parame-
ters, which should not be uninformative. However, this
problem dissipates as evidence for these parameters ac-
cumulates.
Including Studies Outside of the Posterior Passing
Chain
In meta-analysis, extensive literature searches are
conducted to avoid systematically excluding any stud-
ies. However, it may not be obvious how one can in-
corporate studies outside of a posterior passing ‘chain,’
a sequence of studies where each study’s prior is equal
to the previous study’s posterior, into our prior distribu-
tion. The same problem occurs if two studies are done
simultaneously, creating a fork in the chain. Brand et
al. (2019) suggest that a normal prior with variance
representing our uncertainty and with a mean at the
estimate given by the study may be used. However, it
can be difficult to determine such a variance, and this
procedure only makes sense if we are the first study in
a chain aiming to get information from an unchained
study. A better answer is to include the study’s likeli-
hood function in our posterior derivation. When there
are simultaneous studies, simply take one’s posterior as
the prior and use the likelihood from the other as if it
were outside of the chain altogether. Switching back to
the simple fixed effects procedure for conciseness, this
gives us:
fi(θ)π(θ)f(xi1|θ)f(xi|θ)(13)
To include more studies, one simply needs to add
more f(xn|θ)functions. As a side note, the nature of
4
this function yields an obvious option for Bayesian-style
updating in a non-Bayesian framework:
Li(θ)f(x1,...,xi1|θ)f(xi|θ)(14)
This may simply be interpreted as the likelihood func-
tion for θacross all the included studies. Inferences can
then be made using standard frequentist sequential trial
methods (cf. Wetterslev et al., 2017).
Further comments and discussion
Is Posterior Passing a Practical Replacement for
Meta-analysis?
Brand et al. (2019) suggest that posterior passing can
replace traditional meta-analyses. Indeed, the improved
posterior passing procedures introduced in the previous
section can compete with traditional meta-analyses, but
is it practical? To match the quality of traditional meta-
analyses, one would have to meet the same conditions,
including extensive literature search and the inclusion
of all available studies. Furthermore, most studies are
currently done in a frequentist manner, and splits can
occur in a chain due to simultaneous studies, so in prac-
tice each study in the chain will have to do its own mini
meta-analysis. At that point, it might be preferable to
just do traditional meta-analyses.
Brand et al. (2019) also suggest that posterior pass-
ing can solve the problem of conflicting meta-analyses
by updating evidence in real time. Meta-analyses can
provide contradictory results due to a number of rea-
sons, such as differing inclusion criteria and using dif-
ferent methods. However, posterior passing only ap-
pears to be able to mitigate differences occurring by
meta-analyses occurring at different times rather than
by different methods. In this case though, one would
simply go with the most recent meta-analysis. How-
ever, when multiple meta-analyses disagree, they of-
ten have differences in statistical methodology or in-
clusion criteria. Replacing meta-analyses with posterior
passing would likely result in multiple posterior passing
chains to reflect this disagreement. Perhaps this might
be avoided if fields are sufficiently vigilant in prevent-
ing conflicting chains, but the same would be true of
traditional meta-analytic conflicts. In fact, one might
argue that for any criticism of a meta-analysis, one
could seemingly make an equivalent criticism of a poste-
rior passing chain. Instead of having disagreeing meta-
analyses, we would instead simply have a number of
individual studies in disagreement. It may be that pos-
terior passing could help mitigate disagreements of this
nature by the fact that inclusion criteria could change
with any study in the chain. However, having such fuzzy
inclusion criteria is clearly undesirable as it would lead
to results with an unclear interpretation. Hence, pos-
terior passing does not appear to avoid the problem of
conflicting meta-analyses in a desirable manner. The
same applies for all methodological limitations of stan-
dard meta-analyses, such as being impacted by publi-
cation bias, as posterior passing and meta-analysis are
mathematically equivalent.
Alternative Roles for Posterior Passing
Even if posterior passing cannot generally replace
traditional meta-analyses, it may nonetheless be use-
ful. With the improvements suggested above, posterior
passing can replace traditional meta-analysis in areas
where meta-analyses are unlikely to produce conflict-
ing results in the first place. An alternative to creating
posterior passing chains that still utilizes the posterior
passing mechanism is to use it in meta-analyses. This
doesn’t solve the issue of conflicting meta-analyses, but
has practical advantages. By using the posterior distri-
bution of the last similar meta-analysis as a prior distri-
bution, meta-analyses can be performed in chunks in-
stead of having to redo the entire meta-analysis with
each update. Similarly, instead of using posterior pass-
ing chains, studies can use posterior distributions from
meta-analyses as their priors to get accurate net effect
estimates within each study. This allows for broader
conclusions than would otherwise be warranted by the
study alone. A particularly relevant case for this is large-
scale replication projects, where the prior can be gotten
from the last meta-analysis. This yields a readily inter-
pretable lower-bound on the extent to which a field’s
view on a topic should shift as the result of the replica-
tion effort, by providing the change in posterior assum-
ing that previous studies had been conducted properly.
Hence, although posterior passing may have problems
as a replacement for meta-analysis, it can have utility
regardless.
Author Contact
The corresponding author may be contacted at
jpritsk@purdue.edu, ORCiD 0000-0001-9647-6684.
Conflict of Interest and Funding
No conflicts of interest declared.
Author Contributions
Pritsker is the sole author of this article.
Open Science Practices
This article is a commentary and had no data or ma-
terials to share, and it was not pre-registered. The en-
5
tire editorial process, including the open reviews, are
published in the online supplement.
References
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein,
H. R. (2010). A basic introduction to fixed-
effect and random-effects models for meta-
analysis. Research synthesis methods,1, 97–111.
https://doi.org/10.1002/jrsm.12
Brand, C. O., Ounsley, J. P., van der Post, D. J., & Mor-
gan, T. J. H. (2019). Cumulative science via
bayesian posterior passing. Meta-psychology,3.
https://doi.org/10.15626/MP.2017.840
Martin, S. (2017). Open peer review by Stephen Martin.
Meta-psychology: Decision Letter for Brand et al.
https://doi.org/10.17605/OSF.IO/C4WN8
McShane, B. B., & Gal, D. (2017). Statistical significance
and the dichotomization of evidence. Journal of
the American Statistical Association,112, 885–
895. https://doi.org/10.1080/01621459.2017.
1289846
Pfeffermann, D., Krieger, A. M., & Rinott, Y. (1998).
Parametric distributions of complex survey data
under informative probability sampling. Statis-
tica Sinica,8, 1087–1114.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014).
P-curve and effect size: Correcting for publica-
tion bias using only significant results. Perspec-
tives on Psychological Science,9, 666–681. https:
//doi.org/10.1177/1745691614553988
Sutton, A. J., Song, F., Gilbody, M., S., & Abrams, K. R.
(2000). Modelling publication bias in meta-
analysis: A review. Statistical Methods in Med-
ical Research,9, 421–445. https://doi.org/10.
1177/096228020000900503
Wetterslev, J., Jakobsen, J. C., & Gluud, C. (2017). Trial
sequential analysis in systematic reviews with
meta-analysis. BMC medical research methodol-
ogy,17.https://doi.org/10.1186/s12874-017-
0315-7
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Background Most meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors). Methods We developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached. ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis. Conclusions Trial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.
Article
Full-text available
The sample distribution is defined as the distribution of the sample mea-surements given the selected sample. Under informative sampling, this distribution is different from the corresponding population distribution, although for several examples the two distributions are shown to be in the same family and only differ in some or all the parameters. A general approach of approximating the marginal sample distribution for a given population distribution and first order sample se-lection probabilities is discussed and illustrated. Theoretical and simulation results indicate that under common sampling methods of selection with unequal proba-bilities, when the population measurements are independently drawn from some distribution (superpopulation), the sample measurements are asymptotically inde-pendent as the population size increases. This asymptotic independence combined with the approximation of the marginal sample distribution permits the use of stan-dard methods such as direct likelihood inference or residual analysis for inference on the population distribution.
Article
Meta-analysis is now a widely used technique for summarizing evidence from multiple studies. Publication bias, the bias induced by the fact that research with statistically significant results is potentially more likely to be submitted and published than work with null or non-significant results, poses a thereat to the validity of such analyses. The implication of this is that combining only the identified published studies uncritically may lead to an incorrect, usually over optimistic, conclusion. How publication bias should be addressed when carrying out a meta-analysis is currently a hotly debated subject. While statistical methods to test for its presence are starting be used, they do not address the problem of how to proceed if publication bias is suspected. This paper provides a review of methods, which can be employed as a sensitivity analysis to assess the likely impact of publication bias on a meta-analysis. It is hoped that this will raise awareness of such methods, and promote their use and development, as well as provide an agenda for future research.
Article
This paper introduces a statistical technique known as “posterior passing” in which the results of past studies can be used to inform the analyses carried out by subsequent studies. We first describe the technique in detail and show how it can be implemented by individual researchers on an experiment by experiment basis. We then use a simulation to explore its success in identifying true parameter values compared to current statistical norms (ANOVAs and GLMMs). We find that posterior passing allows the true effect in the population to be found with greater accuracy and consistency than the other analysis types considered. Furthermore, posterior passing performs almost identically to a data analysis in which all data from all simulated studies are combined and analysed as one dataset. On this basis, we suggest that posterior passing is a viable means of implementing cumulative science. Furthermore, because it prevents the accumulation of large bodies of conflicting literature, it alleviates the need for traditional meta-analyses. Instead, posterior passing cumulatively and collaboratively provides clarity in real time as each new study is produced and is thus a strong candidate for a new, cumulative approach to scientific analyses and publishing.
Article
In light of recent concerns about reproducibility and replicability, the ASA issued a Statement on Statistical Significance and p-values aimed at those who are not primarily statisticians. While the ASA Statement notes that statistical significance and p-values are “commonly misused and misinterpreted,” it does not discuss and document broader implications of these errors for the interpretation of evidence. In this article, we review research on how applied researchers who are not primarily statisticians misuse and misinterpret p-values in practice and how this can lead to errors in the interpretation of evidence. We also present new data showing, perhaps surprisingly, that researchers who are primarily statisticians are also prone to misuse and misinterpret p-values thus resulting in similar errors. In particular, we show that statisticians tend to interpret evidence dichotomously based on whether or not a p-value crosses the conventional 0.05 threshold for statistical significance. We discuss implications and offer recommendations.
Article
There are two popular statistical models for meta-analysis, the fixed-effect model and the random-effects model. The fact that these two models employ similar sets of formulas to compute statistics, and sometimes yield similar estimates for the various parameters, may lead people to believe that the models are interchangeable. In fact, though, the models represent fundamentally different assumptions about the data. The selection of the appropriate model is important to ensure that the various statistics are estimated correctly. Additionally, and more fundamentally, the model serves to place the analysis in context. It provides a framework for the goals of the analysis as well as for the interpretation of the statistics. In this paper we explain the key assumptions of each model, and then outline the differences between the models. We conclude with a discussion of factors to consider when choosing between the two models. Copyright © 2010 John Wiley & Sons, Ltd. Copyright © 2010 John Wiley & Sons, Ltd.
Open peer review by Stephen Martin. Meta-psychology: Decision Letter for Brand
  • S Martin
Martin, S. (2017). Open peer review by Stephen Martin. Meta-psychology: Decision Letter for Brand et al. https://doi.org/10.17605/OSF.IO/C4WN8
Correcting for publication bias using only significant results
P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666-681. https: //doi.org/10.1177/1745691614553988