Page 1
Small-study effectsExtended random effects modelApplicationSimulation studyConcluding remarksReferences
Treatment effect estimates adjusted for small-study
effects via a limit meta-analysis
Gerta R¨ ucker1, James Carpenter12, Guido Schwarzer1
1Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg
2Medical Statistics Unit, London School of Hygiene & Tropical Medicine, London, UK
DFG Forschergruppe FOR 534
ruecker@imbi.uni-freiburg.de
MAER Net Conference, Cambridge, 18 September, 2011
1
Page 2
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarksReferences
Outline
Small-study effects in meta-analysis
Extended random effects model
Application to an example
Simulation study
Concluding remarks
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 20112
Page 3
Small-study effectsExtended random effects model Application Simulation studyConcluding remarks References
Small-study effects in meta-analysis
Small trials may show larger treatment effects than big trials, potentially
caused by
? Publication bias:
Small studies tend to be published only if they show a large effect
? Selective outcome reporting bias:
Present the most significant outcome
? Clinical heterogeneity between patients in large and small trials
? For binary data, treatment effect estimate correlated with standard
error
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysis Sunday, 18 September, 20113
Page 4
Small-study effectsExtended random effects model Application Simulation studyConcluding remarksReferences
Small-study effects in meta-analysis
? Graphical representation of small-study effects: Asymmetry in funnel
plot
? Numerous tests for funnel plot asymmetry available (Sterne et al.,
2011)
? Treatment effect estimates adjusted for small-study effects
? Copas selection model (Copas and Shi, 2000)
? Trim and Fill method (Duval and Tweedie, 2000)
? Regression-based approach (Stanley, 2008; Moreno et al., 2009)
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 20114
Page 5
Small-study effectsExtended random effects model Application Simulation studyConcluding remarks References
A funnel plot showing a strong small-study effect
0.10.5 1.05.010.0 50.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 20115
Page 6
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarks References
Extended random effects model (R¨ ucker et al., 2010)
? Random effects model in meta-analysis:
xi= µ +
?
σ2
i+ τ2?i,?i
iid∼ N(0,1)
xiobserved effect in study i, µ global mean,
σ2
iwithin-study sampling variance, τ2between-study variance
? Extended random effects model, taking account of possible small
study effects by allowing the effect to depend on the standard error:
?
β replaces µ, and α represents bias introduced by small-study effects
(‘publication bias’) (Stanley, 2008; Moreno et al., 2009)
xi= β +
σ2
i+ τ2(α + ?i),?i
iid∼ N(0,1)
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 20116
Page 7
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarks References
Interpretation of α in the extended random effects model
xi= β +
?
σ2
i+ τ2(α + ?i),?i
iid∼ N(0,1)
? α interpreted as the expected shift in the standardised treatment
effect if precision is very small:
?xi− β
? α corresponds to the intercept in a radial (Galbraith) plot
? Egger test on publication bias based on H0: α = 0
E
σi
?
→ α,σi→ ∞
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 20117
Page 8
Small-study effects Extended random effects modelApplicationSimulation study Concluding remarksReferences
Interpretation of α in the extended random effects model
xi= β +
?
σ2
i+ τ2(α + ?i),?i
iid∼ N(0,1)
? β0= β + τα interpreted as the limit treatment effect if precision is
infinite:
E(xi) → β + τα,
? Interpretation of β changes as α is included in the model: In the
presence of a small-study effect, the treatment effect is represented
by β + τα instead of β alone
? β + τα corresponds to a point at the top of the funnel plot
σi→ 0
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysis Sunday, 18 September, 20118
Page 9
Small-study effectsExtended random effects model Application Simulation studyConcluding remarksReferences
ML estimation of α and β
? Use inverse variance weighting: wi= 1/(s2
? ML estimatesˆβ and ˆ α can be interpreted as slope and intercept in
linear regression on so-called generalised radial (Galbraith) plots
? α and β often estimated with large standard error, particularly if
? there are only few studies, or
? there are small studies (large random error) with extreme results
⇒ Potentially false positive finding of small-study effects
? Idea: Shrinkage by inflation of precision, based on extended model
i+ ˆ τ2)
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 20119
Page 10
Small-study effects Extended random effects modelApplication Simulation studyConcluding remarks References
Inflation of precision, based on extended model
xi= β +
?
σ2
i+ τ2(α + ?i),?i
iid∼ N(0,1)
? Imagine each study has an M-fold increased precision:
?
? Limit meta-analysis:
Let M → ∞, substitute estimates for β,τ2,σ2
xM,i= β +
σ2
i/M + τ2(α + ?i),?i
iid∼ N(0,1)
iand ?i
x∞,i=ˆβ +
?
ˆ τ2
i+ ˆ τ2(xi−ˆβ)
s2
? Limit meta-analysis compared to empirical Bayes estimation
? Takes account for bias correction
? Shrinkage factor
s2
?
ˆ τ2
i+ˆ τ2less marked than for empirical Bayes
?
ˆ τ2
i+ˆ τ2
s2
?
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201110
Page 11
Small-study effectsExtended random effects modelApplicationSimulation study Concluding remarksReferences
Application: NSAIDS example
Example
? Meta-analysis of 37 placebo-controlled randomized trials on the
effectiveness and safety of topical non-steroidal anti-inflammatory
drugs (NSAIDS) in acute pain (Moore et al., 1998)
Models compared
? Fixed and random effects model
? Three estimates based on limit meta-analysis (R¨ ucker et al., 2010)
? Expectation β0= β + τα
? Model including bias parameter
? Model without bias parameter
? Copas selection model (Copas and Shi, 2000)
? Trim and Fill method (Duval and Tweedie, 2000)
? Peters method (Moreno et al., 2009)
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 2011 11
Page 12
Small-study effects Extended random effects modelApplicationSimulation studyConcluding remarks References
NSAIDS example (Moore et al., 1998): Funnel plot
0.10.5 1.05.010.050.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 201112
Page 13
Small-study effectsExtended random effects modelApplication Simulation studyConcluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.51.0 5.0 10.0 50.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Random effects model
Fixed effect model
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201113
Page 14
Small-study effects Extended random effects modelApplicationSimulation study Concluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.51.05.0 10.050.0 100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201114
Page 15
Small-study effectsExtended random effects modelApplicationSimulation studyConcluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.1 0.51.05.010.050.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Random effects model
Fixed effect model
Trim−and−fill method
Copas selection model
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 2011 15
Page 16
Small-study effectsExtended random effects model ApplicationSimulation studyConcluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.51.05.010.050.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 201116
Page 17
Small-study effectsExtended random effects modelApplication Simulation studyConcluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.5 1.05.0 10.0 50.0100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201117
Page 18
Small-study effectsExtended random effects model ApplicationSimulation studyConcluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.51.05.010.050.0 100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Shrinkage, resulting in limit meta−analysis
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysis Sunday, 18 September, 201118
Page 19
Small-study effects Extended random effects model ApplicationSimulation studyConcluding remarks References
NSAIDS example (Moore et al., 1998)
0.10.51.05.010.050.0 100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Limit MA, expectation β + τα
Limit MA, including bias parameter
Limit MA, without bias parameter
Peters method
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201119
Page 20
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarksReferences
NSAIDS example (Moore et al., 1998)
0.10.5 1.0 5.010.050.0 100.0
1.5
1.0
0.5
0.0
Odds Ratio
Standard error
Limit MA, expectation β + τα
Limit MA, including bias parameter
Limit MA, without bias parameter
Peters method
Random effects model
Fixed effect model
Trim−and−fill method
Copas selection model
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201120
Page 21
Small-study effectsExtended random effects modelApplicationSimulation study Concluding remarksReferences
NSAIDS example (Moore et al., 1998): Effect estimates
Model
Fixed effect model
Random effects model
Odds ratio [95% CI]
2.89 [2.49; 3.35]
3.73 [2.80; 4.97]
Trim and fill (random effects estimate)
Copas selection model
2.45 [1.83; 3.28]
1.82 [1.46; 2.26]
Limit meta-analysis, expectation (β0= β + τα)
Limit meta-analysis, including bias parameter
Limit meta-analysis, without bias parameter
1.84 [1.26; 2.68]
1.52 [1.04; 2.21]
1.76 [1.52; 2.04]
Peters method 1.51 [1.03; 2.20]
Gerta R¨ ucker, Freiburg Small-study effects in meta-analysisSunday, 18 September, 2011 21
Page 22
Small-study effectsExtended random effects model Application Simulation study Concluding remarksReferences
Simulation study (R¨ ucker et al., 2011)
36 Scenarios, based on binary response data, each repeated 1000 times,
setting
? the number of trials in the meta-analysis: 10
(trial sizes drawn from a log-normal distribution)
? heterogeneity variance τ2= 0.10
? true odds ratio: 0.5, 0.75, 1
? control group event probability: 0.05, 0.10, 0.20, 0.30
? small-study effects simulated based on Copas selection model
(Copas and Shi, 2000), with selection parameter ρ2: 0, 0.36, 1
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysis Sunday, 18 September, 2011 22
Page 23
Small-study effectsExtended random effects modelApplicationSimulation studyConcluding remarksReferences
Simulation results: Mean Squared Error (MSE)
Event proportion in control group
Small-study effects in meta-analysis
0.0
0.1
0.2
0.3
0.4
0.5
5%10%20%30%
No selection, OR=0.5Weak selection, OR=0.5
5% 10%20% 30%
Strong selection, OR=0.5
No selection, OR=0.75Weak selection, OR=0.75
0.0
0.1
0.2
0.3
0.4
0.5
Strong selection, OR=0.75
0.0
0.1
0.2
0.3
0.4
0.5
No selection, OR=1
5%10%20%30%
Weak selection, OR=1 Strong selection, OR=1
Fixed effect model
Random effects model
Limit meta−analysis, allowing for an intercept (β−lim)
Limit meta−analysis, line through origin (µ−lim)
Limit meta−analysis, expectation (β + τ α)
Peters method
Copas selection model
Trim and fill method
Gerta R¨ ucker, FreiburgSunday, 18 September, 201123
Page 24
Small-study effectsExtended random effects modelApplication Simulation studyConcluding remarksReferences
Simulation study: Summary of results
? In the absence of small-study effects
? Conventional models worked best
? Copas selection model preferable to Trim and Fill
? Extended random effects model not optimal
? In the presence of strong selection
? Limit meta-analysis without bias parameter had smallest MSE
? Limit meta-analysis including bias parameter had smallest bias
? Limit meta-analysis expectation and Peters method had best coverage
? Estimates robust against varying estimators for τ2
(Diploma thesis Dominik Struck, Freiburg)
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201124
Page 25
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarksReferences
Concluding remarks
Modelling and philosophy
? Extend the random effects model by a parameter for bias caused by
potential small-study effects
? Limit meta-analysis yields shrunken estimates of individual study
effects — can also be justified from an empirical Bayesian viewpoint
? Consistent with the philosophy of random effects modelling, that
‘inference for each particular study is performed by ‘borrowing
strength’ from the other studies’ (Higgins et al., 2009)
? For adjusting it doesn’t matter where small-study effects come from
(Moreno et al., 2009)
? Large studies are more reliable than small studies
‘Could it be better to discard 90% of the data? A statistical paradox’
(Stanley et al., 2010)
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201125
Page 26
Small-study effectsExtended random effects model Application Simulation study Concluding remarksReferences
References
Copas, J. and Shi, J. Q. (2000). Meta-analysis, funnel plots and sensitivity analysis. Biostatistics,
1:247–262.
Duval, S. and Tweedie, R. (2000). Trim and Fill: a simple funnel-plot-based method of testing and
adjusting for publication bias in meta-analysis. Biometrics, 56:455–463.
Higgins, J. P., Thompson, S. G., and Spiegelhalter, D. J. (2009). A re-evaluation of random-effects
meta-analysis. Journal of the Royal Statistical Society, 172:137–159.
Moore, R. A., Tramer, M. R., Carroll, D., Wiffen, P. J., and McQuay, H. J. (1998). Quantitive systematic
review of topically applied non-steroidal anti-inflammatory drugs. British Medical Journal,
316(7128):333–338.
Moreno, S., Sutton, A., Ades, A., Stanley, T., Abrams, K., Peters, J., and Cooper, N. (2009).
Assessment of regression-based methods to adjust for publication bias through a comprehensive
simulation study. BMC Medical Research Methodology, 9:2.
R¨ ucker, G., Carpenter, J., and Schwarzer, G. (2011). Detecting and adjusting for small-study effects in
meta-analysis. Biometrical Journal, 53(2):351–368.
R¨ ucker, G., Schwarzer, G., Carpenter, J., Binder, H., and Schumacher, M. (2010). Treatment effect
estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics, 12(1):122–142.
Doi:10.1136/jme.2008.024521.
Stanley, T., Jarrell, S. B., and Doucouliagos, H. (2010). Could it be better to discard 90% of the data?
A statistical paradox. The American Statistician, 64(1):70–77.
Stanley, T. D. (2008). Meta-regression methods for detecting and estimating empirical effects in the
presence of publication selection. Oxford Bulletin of Economics and Statistics, 70(105–127).
Sterne, J. A. C. et al. (2011). Recommendations for examining and interpreting funnel plot asymmetry
in meta-analyses of randomised controlled trials. British Medical Journal, 343:d4002. doi:
10.1136/bmj.d4002.
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201126
Page 27
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarksReferences
Appendix: ML estimation of α and β
? Writing wi= 1/(s2
estimates
i+ ˆ τ2) (inverse variance weighting), obtain
ˆβ =
?k
i=1wixi−1
?k
√wi(xi−ˆβ).
k
?k
i=1
k(?k
√wi?k
i=1
i=1
√wixi
i=1wi−1
√wi)2
ˆ α =1
k
k ?
i=1
? ˆβ and ˆ α can be interpreted as slope and intercept in linear regression
on so-called generalised radial plots
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysisSunday, 18 September, 201127
Page 28
Small-study effectsExtended random effects model ApplicationSimulation study Concluding remarks References
Appendix: ML estimation of α and β – Variance estimates
? Variance estimates:
?
?
Var (ˆβ) =
1
?wi−1
?wi−1
k(? √wi)2
k(? √wi)2
Var (ˆ α) =
1
k
?wi
? Both variance estimates inversely proportional to variance of the
observed study precisions√wi= 1/si
⇒ Estimation is the more precise, the more precision (size) varies
between studies
Gerta R¨ ucker, FreiburgSmall-study effects in meta-analysis Sunday, 18 September, 2011 28
Page 29
Small-study effectsExtended random effects modelApplication Simulation studyConcluding remarks References
Appendix: Simulation results for bias log?
Limit meta−analysis, expectation (β + τ α)
Peters method
Copas selection model
Trim and fill method
OR − logOR
Event proportion in control group
Small-study effects in meta-analysis
−0.4
−0.2
0.0
0.2
0.4
5% 10%20%30%
No selection, OR=0.5Weak selection, OR=0.5
5% 10% 20%30%
Strong selection, OR=0.5
No selection, OR=0.75Weak selection, OR=0.75
−0.4
−0.2
0.0
0.2
0.4
Strong selection, OR=0.75
−0.4
−0.2
0.0
0.2
0.4
No selection, OR=1
5% 10% 20%30%
Weak selection, OR=1Strong selection, OR=1
Fixed effect model
Random effects model
Limit meta−analysis, allowing for an intercept (β−lim)
Limit meta−analysis, line through origin (µ−lim)
Gerta R¨ ucker, FreiburgSunday, 18 September, 201129
Page 30
Small-study effects Extended random effects modelApplicationSimulation studyConcluding remarks References
Appendix: Simulation results for coverage of 95% CI
Event proportion in control group
Small-study effects in meta-analysis
0.0
0.2
0.4
0.6
0.8
1.0
5% 10% 20%30%
No selection, OR=0.5Weak selection, OR=0.5
5% 10%20% 30%
Strong selection, OR=0.5
No selection, OR=0.75Weak selection, OR=0.75
0.0
0.2
0.4
0.6
0.8
1.0
Strong selection, OR=0.75
0.0
0.2
0.4
0.6
0.8
1.0
No selection, OR=1
5% 10%20%30%
Weak selection, OR=1Strong selection, OR=1
Fixed effect model
Random effects model
Limit meta−analysis, allowing for an intercept (β−lim)
Limit meta−analysis, line through origin (µ−lim)
Limit meta−analysis, expectation (β + τ α)
Peters method
Copas selection model
Trim and fill method
Gerta R¨ ucker, Freiburg Sunday, 18 September, 2011 30
Download full-text