Elizabeth Tipton

• PhD
• Professor at Northwestern University

Results of meta-analyses are of interest not only to researchers but often to policy-makers and other decision-makers (e.g., in education and medicine), and visualizations play an important role in communicating data and statistical evidence to the broader public. Therefore, the potential audience of meta-analytic visualizations is broad. However,...

If our goal in education research is to produce knowledge that is useful and used, we need to design impact evaluations that value understanding variation as much as understanding average impact. The context in which an intervention is implemented and the characteristics of participants are integral components of the intervention, and we need to st...

This article presents methods for using extant data to improve the properties of estimators of the standardized mean difference (SMD) effect size. Because samples recruited into education research studies are often more homogeneous than the populations of policy interest, the variation in educational outcomes can be smaller in these samples than is...

Literature screening is the process of identifying all relevant records from a pool of candidate paper records in systematic review, meta‐analysis, and other research synthesis tasks. This process is time consuming, expensive, and prone to human error. Screening prioritization methods attempt to help reviewers identify most relevant records while o...

Meta-analysis methodology has evolved with the development of more robust statistical techniques; however, few reviews in special education have focused specifically on methodological rigor in meta-analyses. In this study, we examined 29 meta-analyses of mathematics interventions published from 2000 to 2022 to determine the extent to which research...

Meta-analysts often ask a yes-or-no question: Is there an intervention effect or not? This traditional, all-or-nothing thinking stands in contrast with current best practice in meta-analysis, which calls for a heterogeneity-attuned approach (i.e., focused on the extent to which effects vary across procedures, participant groups, or contexts). This...

Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality an...

It is common practice in both randomized and quasi‐experiments to adjust for baseline characteristics when estimating the average effect of an intervention. The inclusion of a pre‐test, for example, can reduce both the standard error of this estimate and—in non‐randomized designs—its bias. At the same time, it is also standard to report the effect...

Regular exercise at the intensity matching maximal fat oxidation (FATmax) has been proposed as a key element in both athletes and clinical populations when aiming to enhance the body's ability to oxidize fat. In order to allow a more standardized and tailored training approach, the connection between FATmax and the individual aerobic thresholds (Ae...

Meta-analysts often ask a yes-or-no question: Is there an intervention effect or not? This traditional, all-or-nothing thinking stands in contrast with current best practice in meta-analysis, which calls for a heterogeneity-attuned approach (i.e., focused on the extent to which effects vary across procedures, participant groups, or contexts). This...

Background Assessment is necessary to ensure both attainment and maintenance of competency in gastrointestinal (GI) endoscopy, and this can be accomplished through self-assessment. We conducted a systematic review with meta-analysis to evaluate the accuracy of self-assessment among GI endoscopists. Methods This was an individual participant data me...

As the body of scientific evidence about effective policies and practices grows, so does the need to effectively communicate that evidence to policy-makers and practitioners. Clearinghouses have emerged to facilitate the evidence-based decision-making process for education practitioners. While the results and methods for developing and analyzing th...

In this comment, we highlight a difference of opinion with "Mertens, S., Herberz, M., Hahnel, U. J., & Brosch, T. (2022). The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proceedings of the National Academy of Sciences, 119(1)."

Knowing who to target with certain messages is the prerequisite of efficient public health campaigns during pandemics. Using the COVID-19 pandemic situation, we explored which facets of the society—defined by age, gender, income, and education levels—are the most likely to visit social gatherings and aggravate the spread of a disease. Analyzing the...

Practitioners and policymakers often want estimates of the effect of an intervention for their local community, e.g., region, state, county. In the ideal, these multiple population average treatment effect (ATE) estimates will be considered in the design of a single randomized trial. Methods for sample selection for generalizing the sample ATE to d...

BACKGROUND Although it is widely recognized that physical activity is an important determinant of health there is considerable challenge in assessing this complex behavior. Tools for the objective assessment of the frequency, intensity, and duration of physical activity in adults and children have largely been developed for short-term use within re...

Background: Although it is widely recognized that physical activity is an important determinant of health, assessing this complex behavior is a considerable challenge. Objective: The purpose of this systematic review and meta-analysis is to examine, quantify, and report the current state of evidence for the validity of energy expenditure, heart...

Wellness in the Schools (WITS) is a national non-profit organization partnering with public schools to provide healthy, scratch cooked, less processed meals (called an Alternative Menu), and active recess. This study examined the effects of WITS programming on school lunch consumption, including fruit and vegetable intake, in second and third grade...

To determine “what works, for whom, and under what conditions,” interventions need to be studied in diverse and heterogeneous samples. At an international scope, this degree of heterogeneity is unlikely in a single study and instead requires conducting multiple studies of the same intervention across the globe. In this paper, we provide an overview...

In the past decade, behavioural science has gained influence in policymaking but suffered a crisis of confidence in the replicability of its findings. Here, we describe a nascent heterogeneity revolution that we believe these twin historical trends have triggered. This revolution will be defined by the recognition that most treatment effects are he...

Knowing who to target with certain messages is the prerequisite of efficient public health campaigns during pandemics. Using the COVID-19 pandemic situation, we explored which facets of the society - defined by age, gender, income, and education levels - are the most likely to visit social gatherings and aggravate the spread of a disease. Analyzing...

In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the exact form of the dependence is unknown. RVE uses a working model of...

Zero-heat-flux thermometers provide clinicians with the ability to continuously and non-invasively monitor body temperature. These devices are increasingly being used to substitute for more invasive core temperature measurements during surgery and in critical care. The aim of this review was to determine the accuracy and precision of zero-heat-flux...

Researchers conducting randomized trials have increasingly shifted focus from the average treatment effect to understanding moderators of treatment effects. Current methods for exploring moderation focus on model selection and hypothesis tests. At the same time, recent developments in the design of randomized trials have argued for the need for pop...

As a result of the evidence-based decision-making movement, the number of randomized trials evaluating educational programs and curricula has increased dramatically over the past 20 years. Policy makers and practitioners are encouraged to use the results of these trials to inform their decision making in schools and school districts. At the same ti...

In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the nature of the dependence is unknown. RVE uses a working model of the...

Developing and maintaining a high-quality science teaching corps has become increasingly urgent with standards that require students to move beyond mastering facts to reasoning and arguing from evidence. Effective professional development (PD) for science teachers enhances teacher outcomes and, in turn, enhances primary and secondary student outcom...

Background: Seizures are a morbid complication of intracerebral hemorrhage (ICH) and increase the risk for herniation, status epilepticus, and worse patient outcomes. Prophylactic levetiracetam is administered to approximately 40% of patients with ICH. It is unclear which patients are consciously selected for treatment by physicians. We sought to...

The purpose of the proposed systematic-review and meta-analysis is to systematically examine, quantify and report the validity of energy expenditure, intensity, heart rate, and steps measured by combined-sensing Fitbits.

The original publication of this article contained the wrong image in Figure 1. The correct Figure 1 is presented at the next page.

This chapter explains a new approach that survey samplers can use when designing probability samples for survey experiments where there is a possibility of treatment heterogeneity. It begins by explaining why probability samples are preferred to nonprobability samples for estimating two quantities (or estimands): population average treatment effect...

A global priority for the behavioural sciences is to develop cost-effective, scalable interventions that could improve the academic outcomes of adolescents at a population level, but no such interventions have so far been evaluated in a population-generalizable sample. Here we show that a short (less than one hour), online growth mindset interventi...

Students from higher–socioeconomic status (SES) backgrounds show a persistent advantage in academic outcomes over lower-SES students. It is possible that students’ beliefs about academic ability, or mindsets, play some role in contributing to these disparities. Data from a recent nationally representative sample of ninth-grade students in U.S. publ...

There is substantial interest in the possibility that cognitive skills can be improved by dedicated behavioral training. Yet despite the large amount of work being conducted in this domain, there is not an explicit and widely agreed upon consensus around the best methodological practices. This document seeks to fill this gap. We start from the pers...

Randomized control trials (RCTs) have long been considered the “gold standard” for evaluating the impacts of interventions. However, in most education RCTs, the sample of schools included is recruited based on convenience, potentially compromising a study’s ability to generalize to an intended population. An alternative approach is to recruit schoo...

Having surveyed the history and methods of meta-regression in a previous paper,1 in this paper we review which and how meta-regression methods are applied in recent research syntheses. To do so, we reviewed studies published in 2016 across four leading research synthesis journals: Psychological Bulletin, the Journal of Applied Psychology, Review of...

At the beginning of the development of meta-analysis, understanding the role of moderators was given the highest priority, with meta-regression provided as a method for achieving this goal. Yet in current practice, meta-regression is not as commonly used as anticipated. This paper seeks to understand this mismatch by reviewing the history of meta-r...

Having surveyed the history and methods of meta‐regression in a previous paper,1 in this paper we review which and how meta‐regression methods are applied in recent research syntheses. To do so, we reviewed studies published in 2016 across four leading research synthesis journals: Psychological Bulletin, the Journal of Applied Psychology, Review of...

There is substantial interest in the possibility that cognitive skills can be improved by dedicated behavioral training. Yet despite the large amount of work being conducted in this domain, there is not an explicit and widely agreed upon consensus around the best methodological practices. This document seeks to fill this gap. We start from the pers...

At the beginning of the development of meta‐analysis, understanding the role of moderators was given the highest priority, with meta‐regression provided as a method for achieving this goal. Yet in current practice, meta‐regression is not as commonly used as anticipated. This paper seeks to understand this mismatch by reviewing the history of meta‐r...

The National Study of Learning Mindsets (NSLM) is a randomized trial evaluating an intervention in a national sample of schools that were selected to participate via probability sampling methods. The response rate for this study was 56%. This paper evaluates whether site-level non-response compromises the generalizability of the results from the ac...

Three previously reported experiments found that a technology-enhanced intervention increased student conceptual understanding of mathematics in Texas. To investigate generalizability to broader populations and settings, we triangulate among three methods. First, we examine interactions between demographic variables and intervention effects. We fou...

Objective Assess impact of school lunch environmental factors on fruit and vegetable (F&V) consumption in second and third grade students. Design Cross-sectional observations in 1 school year. Participants Students from 14 elementary schools in 4 New York City boroughs (n = 877 student-tray observations). Main Outcome Measure(s) Dependent variab...

Background Transcutaneous carbon dioxide (TcCO 2 ) monitoring is a non-invasive alternative to arterial blood sampling. The aim of this review was to determine the accuracy and precision of TcCO 2 measurements. Methods Medline and EMBASE (2000–2016) were searched for studies that reported on a measurement of PaCO 2 that coincided with a measuremen...

A priori power analyses allow researchers to estimate the number of participants needed to detect the effects of an intervention. However, power analyses are only as valid as the parameter estimates used. One such parameter, the expected effect size, can vary greatly depending on several study characteristics, including the nature of the interventi...

School-based evaluations of interventions are increasingly common in education research. Ideally, the results of these evaluations are used to make evidence-based policy decisions for students. However, it is difficult to make generalizations from these evaluations because the types of schools included in the studies are typically not selected rand...

The ubiquity of video games in today’s society has led to significant interest in their impact on the brain and behavior and in the possibility of harnessing games for good. The present meta-analyses focus on one specific game genre that has been of particular interest to the scientific community—action video games, and cover the period 2000–2015....

Bland-Altman method comparison studies are common in the medical sciences and are used to compare a new measure to a gold-standard (often costlier or more invasive) measure. The distribution of these differences is summarized by two statistics, the 'bias' and standard deviation, and these measures are combined to provide estimates of the limits of...

Background: Results from clinical trials have shown that sugar-sweetened beverages (SSBs) lead to increased body mass index (BMI) and obesity. This relationship has yet to be explored in observational data for nonclinical populations of adults. Objective: To compare adults who drank 4+ SSBs daily to those who drank 0 in the population of adults...

Childhood obesity is a complex, worldwide problem. Significant resources are invested in its prevention, and high-quality evaluations of these efforts are important. Conducting trials in school settings is complicated, making process evaluations useful for explaining results. Intervention fidelity has been demonstrated to influence outcomes, but ot...

Background: Large-scale randomized experiments are important for determining how policy interventions change average outcomes. Researchers have begun developing methods to improve the external validity of these experiments. One new approach is a balanced sampling method for site selection, which does not require random sampling and takes into acco...

Background: Policy makers and researchers are frequently interested in understanding how effective a particular intervention may be for a specific population. One approach is to assess the degree of similarity between the sample in an experiment and the population. Another approach is to combine information from the experiment and the population t...

Sample size and statistical power calculation should consider clustering effects when schools are the unit of randomization in intervention studies. The objective of the current study was to investigate how student outcomes are clustered within schools in an obesity prevention trial. Baseline data from the Food, Health & Choices project were used....

In this article, we review four software packages for implementing propensity score analysis in R: Matching, MatchIt, PSAgraphics, and twang. After briefly discussing essential elements for propensity score analysis, we apply each package to a data set from the Early Childhood Longitudinal Study in order to estimate the average effect of elementary...

Purpose Identifying and understanding causal risk factors for crime over the life-course is a key area of inquiry in developmental criminology. Prospective longitudinal studies provide valuable information about the relationships between risk factors and later criminal offending. Meta-analyses that synthesize findings from these studies can summari...

This paper describes the effects of an analysis-of-practice professional development (PD) program on elementary school students' (grades 4–6) science outcomes. The study design was a cluster-randomized trial with an analysis sample of 77 schools, 144 teachers and 2823 students. Forty-two schools were randomly assigned to treatment, (88.5 hours) of...

Recently, statisticians have begun developing methods to improve the generalizability of results from large-scale experiments in education. This work has included the development of methods for improved site selection when random sampling is infeasible, including the use of stratification and targeted recruitment strategies. The current paper provi...

In longitudinal panels and other regression models with unobserved effects, fixed effects estimation is often paired with cluster-robust variance estimation (CRVE) in order to account for heteroskedasticity and un-modeled dependence among the errors. CRVE is asymptotically consistent as the number of independent clusters increases, but can be biase...

In longitudinal panels and other regression models with unobserved effects, fixed effects estimation is often paired with cluster-robust variance estimation (CRVE) in order to account for heteroskedasticity and un-modeled dependence among the errors. CRVE is asymptotically consistent as the number of independent clusters increases, but can be biase...

The aim of this investigation was to examine the comparative effectiveness of the new Recline Exercise (RE) and the traditional Head Lift Exercise (Shaker Exercise) on submental muscle activity, tongue strength, and perceived exertion in 40 healthy young adults (mean age = 24.5 years, SD 2.6 years). Both groups participated in a 6-week exercise reg...

Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance es...

Meta-regression models are commonly used to synthesize and compare effect sizes. Unfortunately, traditional meta-regression methods are ill-equipped to handle the complex and often unknown correlations among non-independent effect sizes. Robust variance estimation (RVE) is a recently proposed meta-analytic method for dealing with dependent effect s...

Although a large-scale experiment can provide an estimate of the average causal impact for a program, the sample of sites included in the experiment is often not drawn randomly from the inference population of interest. In this article, we provide a generalizability index that can be used to assess the degree of similarity between the sample of uni...

Although primary studies often report multiple outcomes, the covariances between these outcomes are rarely reported. This leads to difficulties when combining studies in a meta-analysis. This problem was recently addressed with the introduction of robust variance estimation. This new method enables the estimation of meta-regression models with depe...

An important question in the design of experiments is how to ensure that the findings from the experiment are generalizable to a larger population. This concern with generalizability is particularly important when treatment effects are heterogeneous and when selecting units into the experiment using random sampling is not possible-two conditions co...

Methodologists have recently proposed robust variance estimation as one way to handle dependent effect sizes in meta-analysis. Software macros for robust variance estimation in meta-analysis are currently available for Stata (StataCorp LP, College Station, TX, USA) and spss (IBM, Armonk, NY, USA), yet there is little guidance for authors regarding...

Randomized experiments are often seen as the gold standard for causal research. Despite the fact that experiments use random assignment to treatment conditions, units are seldom selected into the experiment using probability sampling. Very little research on experimental design has focused on how to make generalizations to well-defined populations...

As a result of the use of random assignment to treatment, randomized experiments typically have high internal validity. However, units are very rarely randomly selected from a well-defined population of interest into an experiment; this results in low external validity. Under nonrandom sampling, this means that the estimate of the sample average tr...

Dependent effect size estimates are a common problem in meta-analysis. Recently, a robust variance estimation method was introduced that can be used whenever effect sizes in a meta-analysis are not independent. This problem arises, for example, when effect sizes are nested or when multiple measures are collected on the same individuals. In this pap...

Having good spatial skills strongly predicts achievement and attainment in science, technology, engineering, and mathematics fields (e.g., Shea, Lubinski, & Benbow, 2001; Wai, Lubinski, & Benbow, 2009). Improving spatial skills is therefore of both theoretical and practical importance. To determine whether and to what extent training and experience...

Mind-body interventions to manage stress-related health problems are of widespread interest. One of the best known methods is mindfulness-based stress reduction (MBSR), andMBSR courses are now offered by health services, as well as in social and welfare settings. In this systematic review, we report on the effects of MBSR interventions on health, q...

Meta-analysis involves representing the results of each study by a numerical index of effect size and statistically combining these estimates across studies. Effect sizes used in meta-analysis include risk differences, risk ratios, odds ratios, standardized mean differences, and Pearson correlation coefficients. Computation of each of these effect...

Conventional meta-analytic techniques rely on the assumption that effect size estimates from different studies are independent and have sampling distributions with known conditional variances. The independence assumption is violated when studies produce several estimates based on the same individuals or there are clusters of studies that are not in...