Michael Borenstein

Biostat, Inc. · Research

In any meta‐analysis, it is critically important to report the dispersion in effects as well as the mean effect. If an intervention has a moderate clinical impact on average we also need to know if the impact is moderate for all relevant populations, or if it varies from trivial in some to major in others. Or indeed, if the intervention is benefici...

In any meta-analysis it is important to report not only the mean effect size but also how the effect size varies across studies. A treatment that has a moderate clinical impact in all studies is very different than a treatment where the impact is moderate on average, but in some studies is large and in others is trivial (or even harmful). A treatme...

Comprehensive Meta‐Analysis (CMA) is a computer program for meta‐analysis that was developed with funding from the National Institutes of Health in the United States. CMA features a spreadsheet view and a menu‐driven interface. As such, it allows a researcher to enter data and perform a simple analysis in a matter of minutes. At the same time, it o...

This chapter provides practical advice about how to think about heterogeneity. It highlights the prediction interval, the statistic that reports the range of true effects. This statistic provides the information that we need, and that many think is being provided by the other statistics. The forest plot of a meta‐analysis typically includes a line...

This chapter introduces the fixed‐effect model. It discusses the assumptions of this model and shows how these are reflected in the formulas used to compute a summary effect, and in the meaning of the summary effect. All factors that could influence the effect size are the same in all the studies, and therefore the true effect size is the same (hen...

The basic idea of meta‐analysis is to compute an effect size from each of several studies, and to calculate a weighted average of these effect size estimates. This chapter provides some examples of situations in which requirements for meta‐analysis are met and where meta‐analysis can therefore be used to combine findings across studies. It aims to...

This chapter provides information on various websites, professional societies, and journals on meta‐analysis, as well as special issues dedicated to meta‐analysis and books on systematic review methods and meta‐analysis. The Human Genome Epidemiology Network is a global collaboration committed to the assessment of the impact of human genome variati...

When the studies report means and standard deviations, the preferred effect size is usually the raw mean difference, the standardized mean difference, or the response ratio. These effect sizes are discussed in this chapter. When the outcome is reported on a meaningful scale and all studies in the analysis use the same scale, the meta‐analysis can b...

In this chapter, the authors show how they can use a prediction interval to describe the distribution of true effect sizes. They review how the prediction interval is used in primary studies, and also show how the same mechanism can be used for meta‐analysis. The summary line in a forest plot uses a diamond to depict the mean effect size and its co...

Most of the issues that one would address when reporting the results of a meta‐analysis are similar to those for reporting the results of a primary study. There are some unique issues as well, and this chapter addresses those issues. A common mistake is to use the fixed‐effect model on the basis that there is no evidence of heterogeneity. The fores...

A meta‐analysis of effect sizes addresses the magnitude of the effect. Vote counting is the process of counting the number of studies that are statistically significant and the number that are not, and then choosing the winner. A meta‐analysis of p‐values tells us only that the effect is probably not zero. This chapter describes two methods for per...

For studies that report a correlation between two continuous variables, the correlation coefficient itself can serve as the effect size index. The correlation is an intuitive measure that has been standardized to take account of different metrics in the original scales. Most meta‐analysts do not perform syntheses on the correlation coefficient itse...

Vote counting is the process of counting the number of studies that are statistically significant and comparing this with the number that are not statistically significant. In any event, the idea of vote counting is fundamentally flawed and the variants on this process are equally flawed. This chapter aims to explain why this is so, and to provide...

This chapter is adapted from the text Common Mistakes in Meta‐Analysis and How to Avoid Them. When the analysis is based on studies pulled from the literature, the random‐effects model is almost invariably the model that should be used. The random‐effects model works well if the following assumptions are met: the studies that were performed are a r...

In meta‐analysis, the confidence interval for the mean is traditionally based on the Z distribution, which yields a relatively narrow interval. When researchers use the random effects model, it would be better to use the Knapp–Hartung adjustment, which yields a wider confidence interval. The adjustment includes two components. First, it modifies th...

This chapter presents worked examples for exploring how to compute the measures of heterogeneity. It shows how to compute the effect size (the log odds ratio) and variance for each study. Further, the chapter also shows how to compute the effect size (the Fisher’s z transformation of the correlation coefficient) and variance for each study. It incl...

The first case of a complex data structure is the case where studies report data from two or more independent subgroups. The stage‐1 and stage‐2 patients represent two independent subgroups since each patient is included in one group or the other, but not both. This chapter aims to compute a summary effect for the impact of the intervention for sta...

For data from a prospective study, such as a randomized trial, that was originally reported as the number of events and non‐events in two groups (the classic 2 × 2 table), researchers typically compute a risk ratio, an odds ratio, and/or a risk difference. For risk ratios, computations are carried out on a log scale. The log risk ratio and the stan...

This chapter provides examples of how one might explain the results of a simple meta‐analysis, for example to a colleague. There is one example based on each of several effect sizes. The chapter introduces the analysis by providing some basic information such as the number of studies and the effect‐size index. It also provides the rationale for usi...

The report of a meta‐analysis will focus on the mean effect size, and then address heterogeneity as a separate matter, if at all. Castells et al. conducted a meta‐analysis of studies that assessed the impact of methylphenidate vs. placebo on the cognitive functioning of adults with attention deficit hyperactivity disorder. Katout et al. looked at t...

This chapter presents an overview of the key concepts discussed in part 7 of this book. The part discusses three cases where studies provide more than one unit of data for the analysis. These are the case of multiple independent subgroups within a study, multiple outcomes or time‐points based on the same subjects, and two or more treatment groups t...

A cumulative meta‐analysis is a meta‐analysis that is performed first with one study, then with two studies, and so on, until all relevant studies have been included in the analysis. Lau et al. used the streptokinase analysis to show the potential impact of meta‐analysis as part of the research process. They argued that if meta‐analysis had been av...

This chapter begins with an example to show how meta‐analysis and narrative review would approach the same question, and then uses this example to highlight the key differences between the two. The meta‐analysis allows us to combine the effects and evaluate the statistical significance of the summary effect. The meta‐analytic approaches allow us to...

In this chapter, the authors address a number of issues that are relevant to both subgroup analyses and to meta‐regression. The researcher must always choose between a fixed‐effect model and a random‐effects model. Researchers often ask about the practical implications of using a random‐effects model rather than a fixed‐effect model. Since the mean...

A central theme in this volume is the fact that we usually prefer to work with effect sizes, rather than p‐values. The reason reflects a fundamental issue that applies both to primary studies and to meta‐analysis, and is the subject of this chapter. Since narrative reviews typically work with p‐values while meta‐analyses typically work with effect...

The goal of a meta‐analysis is only rarely to synthesize data from a set of identical studies. Almost invariably, the goal is to broaden the base of studies in some way, expand the question, and study the pattern of answers. The question of whether it makes sense to perform a meta‐analysis, and the question of what kinds of studies to include, must...

This chapter provides an overview of software Comprehensive Meta‐Analysis (CMA) and shows how to use it to implement the ideas. The same approach could be used with any other program as well. The chapter also provides a sense for the look‐and‐feel of the program. CMA features a spreadsheet view and a menu‐driven interface. As such, it allows a rese...

In this chapter, the authors show how meta‐analysis can be used to compare the mean effect for different subgroups of studies. They present three computational models. These are fixed‐effect, random‐effects using separate estimates of 𝜏2, and random‐effects using a pooled estimate of 𝜏2. In a primary study, the t‐test can be used to compare the mea...

The effect size, a value which reflects the magnitude of the treatment effect or the strength of a relationship between two variables, is the unit of currency in a meta‐analysis. In this chapter, the effect size for each study is computed, and then the effect sizes is discussed to assess the consistency of the effect across studies and to compute a...

This chapter shows how the multiple regression used in primary studies can be applied to meta‐regression. It begins with the fixed‐effect model, which is simpler, and then moves on to the random‐effects model, which is generally more appropriate. Since the meaning of a summary effect size is different for fixed versus random effects, the null hypot...

This chapter focuses on two themes related to statistical power. The first theme is conceptual. The chapter discusses the factors that determine power and explores how the value of these factors may change as we move from a primary study to a meta‐analysis. The second theme is practical. The chapter briefly reviews the process of power analysis for...

Studies that used independent groups and studies that used matched groups were both used to yield estimates of the standardized mean difference. There is no problem in combining these estimates in a meta‐analysis since the effect size has the same meaning in all studies. The question of whether or not it is appropriate to combine effect sizes from...

This chapter aims to compute a summary effect for the intervention on Basic skills, which combines the data from reading and math. It investigates the difference in effect size for reading versus math, and explains the method used to compute this effect size and its variance. Since every study will be represented by one score in the meta‐analysis r...

This chapter discusses the reasons for publication bias and the evidence that it exists. It also outlines a series of methods that have been developed to assess the likely impact of bias in any given meta‐analysis. The chapter introduces the idea of a small‐study effect, and how this is often conflated with publication bias. In particular, it expla...

This chapter provides some context for the variance for specific effect sizes such as the standardized mean difference or a log risk ratio. The term precision is used as a general term to encompass three formal statistics, the variance, standard error, and confidence interval. The chapter outlines the relationship between the indices of precision....

Under the random‐effects model, the true effect size may vary from study to study. This chapter discusses approaches to identify and then quantify this heterogeneity. It describes the mechanism that is used to extract the true between‐studies variation from the observed variation. The chapter considers what is meant by ‘heterogeneous’ and then resp...

This chapter provides information on software used for meta‐analysis. The software Comprehensive Meta‐Analysis (CMA) was initially released in 2000 and has been updated on a regular basis since then. The next version is scheduled for release in 2021. For researchers who would prefer to use R to perform meta‐analysis, Wolfgang Viechtbauer has publis...

This chapter presents worked examples for continuous data (using the standardized mean difference), binary data (using the odds ratio) and correlational data (using the Fisher’s z transformation). It starts with the mean, standard deviation, and sample size, and uses the bias‐corrected standardized mean difference (Hedges’ g) as the effect size mea...

This chapter addresses various criticisms that have been leveled at meta‐analysis. They are one number cannot summarize a research field, the file drawer problem invalidates meta‐analysis, mixing apples and oranges, garbage in, garbage out, important studies are ignored, meta‐analysis can disagree with randomized trials, and meta‐analyses are perfo...

This chapter addresses how to proceed when we want to incorporate treatment groups in the same analysis. Specifically, it aims to compute a summary effect for the active intervention versus control and aims to investigate the difference in effect size for interventions. the chapter describes the difference between multiple outcomes and multiple com...

This chapter presents two methods, the Mantel–Haenszel method and the one‐step method (also known as the Peto method) for performing a meta‐analysis on odds ratios. For both methods we assume the data from each study are presented in the form of a 2 × 2 table. The Mantel–Haenszel method is based on the fixed‐effect model, where the weight assigned...

This chapter highlights the conceptual and practical differences between fixed‐effect and random‐effects models. Under the random‐effects model the goal is not to estimate one true effect, but to estimate the mean of a distribution of effects. Under the fixed‐effect model there is a wide range of weights whereas under the random‐effects model the w...

Researchers have developed the practice of classifying heterogeneity as being low, moderate, or high based on the value of I2. This chapter argues that the idea of classifying heterogeneity based on I2 should be strongly discouraged. In the transfusion analysis, the I2 statistic was 29%. In the off‐hours analysis, the I2 statistic was 75%. On that...

This chapter introduces the random‐effects model. It discusses the assumptions of this model, and show how these are reflected in the formulas used to compute a summary effect, and in the meaning of the summary effect. The fixed‐effect model starts with the assumption that the true effect size is the same in all studies. In a random‐effects meta‐an...

To compute the summary effect in a meta‐analysis the researchers compute an effect size for each study and then combine these effect sizes, rather than pooling the data directly. Van Howe published a review article in the International Journal of STD and AIDS that looked at the relationship between circumcision and HIV infection in Africa. The arti...

This chapter provides an overview of two issues. One is the approach to estimates of effect (known as artifact correction), which will be of interest to nearly anyone thinking about using meta‐analysis. The other is the methods that are commonly used to combine results in the field of psychometric meta‐analysis, which will be of interest primarily...

When we speak about heterogeneity in a meta-analysis, our intent is usually to understand the substantive implications of the heterogeneity. If an intervention yields a mean effect size of 50 points, we want to know if the effect size in different populations varies from 40 to 60, or from 10 to 90, because this speaks to the potential utility of th...

Research designs in which clusters are the unit of randomization are quite common in the social sciences. Given the multilevel nature of these studies, the power analyses for these studies are more complex than in a simple individually randomized trial. Tools are now available to help researchers conduct power analyses for cluster randomized trials...

The precision of estimates of treatment effects in multilevel experiments depends on the sample sizes chosen at each level. It is often desirable to choose sample sizes at each level to obtain the smallest variance for a fixed total cost, that is, to obtain optimal sample allocation. This article extends previous results on optimal allocation to fo...

Recent, large, randomized controlled trials (RCTs) showed no benefit of long-acting injectable (LAI) antipsychotics over oral antipsychotics in preventing relapse in schizophrenia, nor did a recent meta-analysis incorporating these studies. However, RCTs might enroll a disproportionate number of patients with better treatment adherence and lower il...

Subgroup analysis is the process of comparing a treatment effect for two or more variants of an intervention-to ask, for example, if an intervention's impact is affected by the setting (school versus community), by the delivery agent (outside facilitator versus regular classroom teacher), by the quality of delivery, or if the long-term effect diffe...

Background: While long-acting injectable antipsychotics (LAIs) are hoped to reduce high relapse rates in schizophrenia, recent randomized controlled trials (RCTs) challenged the benefits of LAIs over oral antipsychotics (OAPs). Methods: Systematic review/meta-analysis of RCTs that lasted ≥ 6 months comparing LAIs and OAPs. Primary outcome was st...

There are two popular statistical models for meta-analysis, the fixed-effect model and the random-effects model. The fact that these two models employ similar sets of formulas to compute statistics, and sometimes yield similar estimates for the various parameters, may lead people to believe that the models are interchangeable. In fact, though, the...

IntroductionIndividual studiesThe summary effectHeterogeneity of effect sizesSummary points

In any meta-analysis, we start with summary data from each study and use it to compute an effect size for the study. An effect size is a number that reflects the magnitude of the relationship between two variables. For example, if a study reports the mean and standard deviation for the treated and control groups, we might compute the standardized m...

KEY POINTS • Various procedures for addressing publication bias are discussed else- where in this volume. The goal of this chapter is to show how these different procedures fit into an overall strategy for addressing bias, and to discuss computer programs that can be used to implement this strategy. • To address publication bias the researcher shou...

An algorithm is presented for calculating the power for the logistic and proportional hazards models in which some of the covariates are discrete and the remainders are multivariate normal. The mean and covariance matrix of the multivariate normal covariates may depend on the discrete covariates.The algorithm, which finds the power of the Wald test...

Publication bias is the tendency to decide to publish a study based on the results of the study, rather than on the basis of its theoretical or methodological quality. It can arise from selective publication of favorable results, or of statistically significant results. This threatens the validity of conclusions drawn from reviews of published scie...

Despite the demonstrated efficacy of clozapine in severely refractory schizophrenia, questions remain regarding its efficacy for primary negative symptoms, comparison with a moderate dose of a first-generation antipsychotic, and adverse effects during a longer-term trial. This study examined its efficacy in partially responsive, community-based pat...

This paper provides the reader with an overview of several key elements in study planning and analysis. In particular, it highlights the differences between significance tests (statistical significance) and effect size estimation (clinical significance). This paper focuses on methodologic issues, and provides an overview of trends in research. PAPE...

Purpose. The New Jersey Diabetic Retino >atliy Study (NJDRS) is designed to determine the frequency and severity of diabetic retin >pathy in a randomly selected sample of type I diabetic African-American patients, listec in the New Jersey Hospital Discharge Data (HDD). The study examines relationship of severity of retinopalhy to diabetes duration,...

There is controversy over whether tardive dyskinesia (TD) is solely a consequence of antipsychotic drug treatment or in part may reflect an intrinsic aspect of the disease process. Pathophysiologic factors could, independently or in concert with drug effects, lead to the development of dyskinetic signs. We studied prospectively 118 patients in thei...

To determine whether a rigorous antiseptic hand washing of bare hands with 4% chlorhexidine and alcohol reduced fingertip microbial colonization as compared with the use of boxed, clean, nonsterile latex gloves. In addition, to investigate if aseptic donning technique and/or a prior hand washing would reduce the level of glove contamination. Prospe...

We present a comparison of the results from two studies of patients on clozapine. The American study (n = 84; Hillside Hospital, Glen Oaks, NY) and the European study (n = 63; Innsbruck University Clinics, Innsbruck, Austria) both examine efficacy and side effects in schizophrenic patients on this atypical neuroleptic. There is a very substantial d...

If the purpose of a clinical study is not only to test the null hypothesis but also to estimate the magnitude of the treatment effect, the study design should ensure not only that the study will have adequate power but also that it will enable the researcher to report the relevant parameters with an appropriate level of precision. This paper discus...

The prevalence, clinical correlates, and outcome of the deficit syndrome were determined for 70 patients ascertained in their first episode of schizophrenia and then followed through their recovery. Patients were treated in a standardized manner and underwent baseline assessments of symptoms and adverse effects that were repeated at intervals throu...

A statistical wit once remarked that researchers often pose the wrong question and then proceed to answer that question incorrectly. The question that researchers intend to ask is whether or not a treatment effect is clinically significant. The question that is typically asked, however, is whether or not the treatment effect is statistically signif...

This study examined the pathomorphology of the caudate nuclei in first-episode schizophrenic patients with minimal previous neuroleptic exposure. Magnetic resonance imaging (MRI) of the brain was used to examine longitudinally the caudate pathomorphology in 29 first-episode schizophrenic patients and 10 healthy comparison subjects. MRI scans were o...