ArticlePDF Available

Quasi-experimental designs for community-level public health violence reduction interventions: a case study in the challenges of selecting the counterfactual

DOI:10.1007/s11292-017-9308-0
Authors:
Caterina Roman at Temple University
Caterina Roman
Hannah Klein at Lewis University
Hannah Klein
Kevin T. Wolff
Kevin T. Wolff
Kevin T. Wolff
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

Objectives We highlight the importance of documenting the step-by-step processes used for the selection of comparison areas when evaluating a community-level intervention that targets a large-scale community. Methods We demonstrate the proposed method using a propensity score matching framework for an impact analysis of the Cure Violence Public Health Model in Philadelphia. To select comparison communities, propensity score models are run using different levels of aggregation to define the intervention site. We discuss the trade-offs made. ResultsWe find wide variation in documentation and explanation in the extant literature of the methods used to select comparison communities. The size of the unit of analysis at which a community is measured complicates the decision processes, and in turn, can affect the validity of the counterfactual. Conclusions It is important to carefully consider the unit of analysis for measurement of comparison communities. Assessing the geographic clustering of matched communities to mirror that of the treated community holds conceptual appeal and represents a strategy to consider when evaluating community-level interventions taking place at a large scale. Regardless of the final decisions made in the selection of the counterfactual, the field could benefit from more systematic diagnostic tools that document and guide the steps and decisions along the way, and ask: “could there have been another way of doing each step, and what difference would this have made?” Overall, across community-level evaluations that utilize quasi-experimental designs, documentation of the counterfactual selection process will provide a more fine-grained understanding of causal inference.
Figures - uploaded by Caterina Roman
Author content
All figure content in this area was uploaded by Caterina Roman
Content may be subject to copyright.
Content uploaded by Caterina Roman
Author content
All content in this area was uploaded by Caterina Roman on May 25, 2018
Content may be subject to copyright.
A preview of the PDF is not available
  • A preview of this full-text is provided by Springer Nature.
  • Learn more
Preview content only
Content available from Journal of Experimental Criminology
This content is subject to copyright. Terms and conditions apply.
... The core evaluation question is whether GVI reduced group-involved violence (Piehl, Kennedy, & Braga, 2000;Roman, Klein, & Wolff, 2018). 3 A key consideration when evaluating a GVI intervention is whether the intervention meaningfully differs "from standard practice or from practice that would have been generated in the absence of the process." ...
... As a practical matter, GVI implementation in most cities do not involve randomly assigned treatment and control groups; therefore, randomized controlled trials are often not a feasible evaluation option (Braga, Kennedy, Waring, & Piehl, 2001;Saunders, Lundberg, Braga, Ridgeway, & Miles, 2015;Braga, Weisburd, & Turchan, 2018;Braga & Weisburd, 2014;Weiner, et al., 2007). For example, absent a tool such as propensity score matching, a raw comparison of treated groups with untreated groups would create selection bias (Roman, Klein, & Wolff, 2018). For example, a group that receives GVI treatment is inherently different from a group that has never received GVI treatment on a range of discernable and non-discernable factors, including recent involvement in violence. ...
... Instead, internally valid quasi-experiments will utilize some criterion other than random assignment to isolate the effect of a treatment (Welsh & Farrington, 2001;Mark & Reichardt, 2009;Roman, Klein, & Wolff, 2018;. For example, where the study units -such as census tracts -do not receive the GVI treatment at the same time, a quasi-experiment may utilize this variation to determine the effect of the GVI treatment. ...
View
... Evaluations must compare the effects of interventions with estimates of the "counterfactual," or what would have happened to key outcomes if an intervention had not been implemented (Roman, Klein and Wolff 2018). Simply measuring outcomes before and after an intervention, or what researchers call a pre-post design, is not reliable evidence. ...
Community Violence Intervention at the Roots (CVI-R): Building Evidence for Grassroots Community Violence Prevention
Technical Report
Full-text available
  • Jun 2022
The crime and justice field recently started to label a wide array of violence prevention strategies as Community Violence Interventions (or CVI). Many of these strategies depend on law enforcement and social services, but the most innovative approaches are community-centered and community-sourced. They are grassroots efforts that rely on the resources of neighborhoods and residents themselves, operating separately from law enforcement and traditional human services. These strategies could be called Community Violence Interventions at the Roots (or CVI-R). The most established CVI-R programs are Cure Violence and Advance Peace. They offer highly localized and potentially cost-effective approaches to public safety, but do they work? Evaluation evidence is recent and not yet consistent, but the grassroots approach to community violence prevention is highly promising. To build sustainable CVI-R models, communities and researchers must collaborate in designing rigorous evaluations to produce reliable and actionable evidence.
View
... 30 However, studies of Cure Violence's impact on gun violence outcomes, while generally positive, have produced varied estimates of program effect across sites-sometimes within the same city-with some studies reporting null or negative results. 31,32,33,34 The Operation Peacemaker Fellowship in Richmond, CA, from which the Advance Peace model was created, was associated with statistically significant reductions in firearm violence, but the evaluators attributed a smaller yet nontrivial increase in non-firearm violence to the model as well. 35 An evaluation of Advance Peace in Sacramento found that the program was associated with mean gun violence incident decreases ranging from -8% to -29% in the four communities where the program was implemented, compared with an overall +9% increase in gun violence incidents in non-Advance Peace zones, but interrupted time series analyses did not find those reductions to be statistically significant. ...
View
... Researchers have employed a range of statistical methods and models to select the "counterfactual" or comparison areas for evaluation, attempting to minimize as much bias as possible that arises from potentially omitted factors. Criminal justice scholars have been vocal about the difficulties inherent in rigorously evaluating place-based intervention programs and strategies (Farrell et al., 2016;National Research Council, 2005;Roman et al., 2018) when randomization is not possible. ...
Process and Impact Evaluation of the Kensington Initiative: A Collaborative Gun Violence and Opioid Reduction Strategy in Philadelphia Process and Impact Evaluation of the Kensington Initiative Authors
Technical Report
Full-text available
  • Jun 2022
The Kensington Initiative (KI) is a collaborative, law enforcement-driven strategy in the greater Kensington neighborhood of Philadelphia. This initiative was developed in early 2018 to bring agencies together in a multi-pronged approach to combat gun violence and reduce overdose incidents in an area with numerous high-volume outdoor drug markets. The effort is led by the Pennsylvania Office of the Attorney General’s (OAG) Bureau of Narcotics Investigation (BNI) and Drug Strike Force Section and coordinated with the Philadelphia Police Department (PPD), with follow-up city services on targeted blocks delivered by agencies operating under the City’s Managing Director’s Office (MDO). In 2019, when the KI was already underway, the OAG was awarded a federal grant through the U.S. Bureau of Justice Assistance (BJA) to further develop the KI strategy and add a research partner (Temple University, Department of Criminal Justice) to conduct a process and impact evaluation of the initiative. Using a logic model approach, the research team also developed a system of performance tracking and reporting to provide feedback to the OAG around effective elements of the initiative. This report details the results of the process and impact evaluation, including departures from the original logic model, successes and challenges experienced, and whether the initiative led to a reduction in shootings on the blocks targeted by an enforcement. For some assessments, additional outcomes were analyzed (e.g., drug-related computer-aided dispatch (CAD) events, overdose deaths, Philly311 requests for city services).
View
... In our view, meaningful gun violence reduction requires an evidence-based, dual-pronged approach that leverages upstream strategies like street outreach to prevent shootings from occurring in conjunction with midstream tactics that interrupt cycles of retaliation and provide services to victims via hospital-based violence intervention programs (HVIPs). Both types of programs currently operate in cities throughout the U.S. yet often suffer from barriers to sustained success including insufficient funding, decentralized administration, outreach worker retention, and lack of rigorous evaluation (Decker et al., 2008;Roman et al., 2018;Whitehill et al., 2014). ...
Community gun violence and functional disability: An ecological analysis among men in four U.S. cities
Article
This study uses data on neighborhoods in four U.S. cities over five years to examine the relationship between fatal and non-fatal gun violence and rates of functional disability among men. Descriptive analyses indicate significant disparities in shooting rates across neighborhoods and heightened associated disability in high shooting communities. Multivariate results show that rates of non-fatal shootings correspond to greater functional disability among young men, but not older men. Fatal gun violence is not associated with increased community disability. The findings suggest that improvements in local gun violence prevention may serve to address broader community disparities in health and well-being.
View
... [5][6][7][8] However, studies examining the program's impact on homicides and nonfatal shootings, while generally positive, have produced varied program effect estimates across sites and cities, with some studies reporting no beneficial effects. [4][5][6][7][8][9][10] The Cure Violence model was first implemented in Baltimore, Maryland, in 2007. ...
Using synthetic control methodology to estimate effects of a Cure Violence intervention in Baltimore, Maryland
Article
Full-text available
  • Feb 2021
Objective To estimate the long-term impact of Safe Streets Baltimore, which is based on the Cure Violence outreach and violence interruption model, on firearm violence. Methods We used synthetic control methods to estimate programme effects on homicides and incidents of non-fatal penetrating firearm injury (non-fatal shootings) in neighbourhoods that had Safe Streets’ sites and model-generated counterfactuals. Synthetic control analyses were conducted for each firearm violence outcome in each of the seven areas where Safe Streets was implemented. The study also investigated variation in programme impact over time by generating effect estimates of varying durations for the longest-running programme sites. Results Synthetic control models reduced prediction error relative to regression analyses. Estimates of Safe Streets’ effects on firearm violence varied across intervention sites: some positive, some negative and no effect. Beneficial programme effects on firearm violence reported in prior research were found to have attenuated over time. Conclusions For highly targeted interventions, synthetic control methods may provide more valid estimates of programme impact than panel regression with data from all city neighbourhoods. This research offers new understanding about the effectiveness of the Cure Violence intervention over extended periods of time in seven neighbourhoods. Combined with existing Cure Violence evaluation literature, it also raises questions about contextual and implementation factors that might influence programme outcomes.
View
View
Predicting and Preventing Gun Violence: An Experimental Evaluation of READI Chicago
Article
Gun violence is the most pressing public safety problem in American cities. We report results from a randomized controlled trial (N = 2,456) of a community-researcher partnership called the Rapid Employment and Development Initiative (READI) Chicago. The program offered an 18-month job alongside cognitive behavioral therapy and other social support. Both algorithmic and human referral methods identified men with strikingly high scope for gun violence reduction: for every 100 people in the control group, there were 11 shooting and homicide victimizations during the 20-month outcome period. Fifty-five percent of the treatment group started programming, comparable to take-up rates in programs for people facing far lower mortality risk. After 20 months, there is no statistically significant change in an index combining three measures of serious violence, the study’s primary outcome. Yet there are signs that this program model has promise. One of the three measures, shooting and homicide arrests, declines 65 percent (p = .13 after multiple-testing adjustment). Because shootings are so costly, READI generates estimated social savings between $182,000 and $916,000 per participant (p = .03), implying a benefit-cost ratio between 4:1 and 20:1. Moreover, participants referred by outreach workers—a prespecified subgroup—show enormous declines in both arrests and victimizations for shootings and homicides (79 and 43 percent, respectively) which remain statistically significant even after multiple-testing adjustments. These declines are concentrated among outreach referrals with higher predicted risk, suggesting that human and algorithmic targeting may work better together.
View
View
An evaluator’s reflections and lessons learned about gang intervention strategies: an agenda for research
Article
Full-text available
  • Jun 2021
Purpose This paper is designed to critically review and analyze the body of research on a popular gang reduction strategy, implemented widely in the United States and a number of other countries, to: (1) assess whether researchers designed their evaluations to align with the theorized causal mechanisms that bring about reductions in violence; and (2) discuss how evidence on gang programs is generated and consumed. That review and assessment is then used to frame a research agenda for studying gang interventions. Design/methodology/approach A case study design is used to generate a multi-faceted understanding of the possible avenues for evaluation research on the law enforcement-based strategy known as the Group Violence Intervention. The paper discusses questions that remain to be answered about the strategy, such as “what type of deterrence is operating?” and if the model actually works by the threat of deterrence, and not by removing high-risk offenders and shootings from the street, what activities are needed to maintain the effect? Findings Across roughly two dozen impact evaluations of GVI, none have examined the likely cause and effect components of this multi-partner strategy in reducing the violence. Furthermore, there are many issues related to the production and generation of criminal justice evaluation research that have adversely pushed the balance of evidence on what works in gang reduction toward law enforcement programming. However, there are many strategies that researchers can use to think broadly about appropriate and holistic research and evaluation on gangs and gang programming. Practical implications The recommendations for research, if implemented, can help build a body of knowledge to move toward community-based and restorative models of gang violence reduction. Originality/value This original piece is one of the first essays to contextualize and discuss how aspects of the production of social science research on gangs may directly impact what programs and strategies are implemented on the ground.
View
Community-Based Violence Prevention: An Assessment of Pittsburgh’s One Vision One Life Program
Book
Full-text available
  • May 2010
In 2006, more than 6 million individuals were victimized by violent crimes in the United States. Although violence is below levels of the early 1990s, it remains high. The extent of violence and its impact highlight a critical need to develop and implement effective programs to reduce violence and victimization. Communities have initiated a wide range of such programs, and scholars have conducted numerous evaluations of varying quality of them. Reviews have found certain types of strategies and specific programs to be promising, but additional critical evaluations are needed to plan violence-reduction programs. This monograph assesses the implementation and impact of the One Vision One Life violence-prevention strategy in Pittsburgh, Pennsylvania. In 2003, Pittsburgh witnessed a 49-percent increase in homicides, prompting a “grassroots” creation and implementation of the One Vision One Life antiviolence strategy. This initiative used a problem-solving, data-driven model, including street-level intelligence, to intervene in escalating disputes, and seeks to place youth in appropriate social programs. Analysis of the program, which is modeled on similar efforts elsewhere, can help inform other efforts to address urban violence.
View
View
Communities are not all created equal: Strategies to prevent violence affecting youth in the United States
Article
Full-text available
We describe violence in the United States (US) and solutions the Urban Networks to Increase Thriving Youth (UNITY) Initiative has developed, led by Prevention Institute, a US non-governmental organization (NGO) and authors of this article, with initial funding from the US Centers for Disease Control and Prevention (CDC). Safety distribution across populations is unequal, while public health research has identified aspects of community environments that affect the likelihood of violence, or risk and resilience factors. An overwhelming number of risk factors have accumulated in some US communities, disproportionately impacting young people of color. US policies, systems, and institutions powerfully shape how and where these factors manifest. Violence is preventable, not inevitable. We argue that comprehensive strategies for improving community environments can reduce violence and promote health equity. We present lessons, tools, and frameworks that UNITY cities use to adapt for international application, including multi-sector collaboration, strategies for influencing policy and legislation, and strengthening local violence prevention efforts.
View
Implementing and Evaluating Comprehensive Evidence-Based Approaches to Prevent Youth Violence: Partnering to Create Communities Where Youth Are Safe From Violence
Article
Full-text available
Violence, including its occurrence among youth, results in considerable physical, emotional, social, and economic consequences in the U.S. Youth violence prevention work at the Division of Violence Prevention (DVP) at the Centers for Disease Control and Prevention (CDC) emphasizes preventing youth violence-related behaviors, injuries, and deaths by collaborating with academic and community partners and stakeholders. Since 2000, DVP has funded three rounds of CDC's National Centers of Excellence in Youth Violence Prevention (YVPCs) in 5-year cycles, with the goal of supporting university-community partnerships so that the best science can be utilized in order to prevent youth violence. The current YVPCs focus on: (a) partnering with communities to identify community needs; (b) selecting and implementing the best comprehensive evidence-based programs to meet those needs; and (c) rigorously evaluating whether those efforts have a community-level impact on youth violence rates. The introduction to this special issue on the current YVPCs provides a brief historical overview on the YVPC Program; outlines the YVPCs' accomplishments to date; and describes the current YVPCs, their community partners, and their activities. The introduction concludes with an overview of the special issue.
View
Designs for Evaluating the Community-Level Impact of Comprehensive Prevention Programs: Examples from the CDC Centers of Excellence in Youth Violence Prevention
Article
Full-text available
This article discusses the opportunities and challenges of developing research designs to evaluate the impact of community-level prevention efforts. To illustrate examples of evaluation designs, we describe six projects funded by the Centers for Disease Control and Prevention to evaluate multifaceted approaches to reduce youth violence in high-risk communities. Each of these projects was designed to evaluate the community-level impact of multiple intervention strategies to address individual and contextual factors that place youth at risk for violent behavior. Communities differed across projects in their setting, size, and how their boundaries were defined. Each project is using multiple approaches to compare outcomes in one or more intervention communities to those in comparison communities. Five of the projects are using comparative interrupted time-series designs to compare outcomes in an intervention community to matched comparison communities. A sixth project is using a multiple baseline design in which the order and timing of intervention activities is randomized across three communities. All six projects are also using regression point displacement designs to compare outcomes within intervention communities to those within broader sets of similar communities. Projects are using a variety of approaches to assess outcomes including archival records, surveys, and direct observations. We discuss the strengths and weaknesses of the designs of these projects and illustrate the challenges of designing high-quality evaluations of comprehensive prevention approaches implemented at the community level.
View
Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores
Article
Full-text available
Objectives Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.
View
Firearms and Violence: A Critical Review
Book
  • Jan 2005
For years proposals for gun control and the ownership of firearms have been among the most contentious issues in American politics. For public authorities to make reasonable decisions on these matters, they must take into account facts about the relationship between guns and violence as well as conflicting constitutional claims and divided public opinion. In performing these tasks, legislators need adequate data and research to judge both the effects of firearms on violence and the effects of different violence control policies. Readers of the research literature on firearms may sometimes find themselves unable to distinguish scholarship from advocacy. Given the importance of this issue, there is a pressing need for a clear and unbiased assessment of the existing portfolio of data and research. Firearms and Violence uses conventional standards of science to examine three major themes - firearms and violence, the quality of research, and the quality of data available. The book assesses the strengths and limitations of current databases, examining current research studies on firearm use and the efforts to reduce unjustified firearm use and suggests ways in which they can be improved. © 2005 by the National Academy of Sciences. All rights reserved.
View
View
Natural and Quasi-Experiments in Economics
Article
Using research designs patterned after randomized experiments, many recent economic studies examine outcome measures for treatment groups and comparison groups that are not randomly assigned. By using variation in explanatory variables generated by changes in state laws, government draft mechanisms, or other means, these studies obtain variation that is readily examined and is plausibly exogenous. This article describes the advantages of these studies and suggests how they can be improved. It also provides aids in judging the validity of inferences that they draw. Design complications such as multiple treatment and comparison groups and multiple preintervention or postintervention observations are advocated.
View
Assessing Macro-Level Predictors and Theories of Crime: A Meta-Analysis
Article
The macro-level approach reemerged as a salient criminological paradigm in the late 1970s and early 1980s. Prompted by new theories and reformulations of existing ones, over 200 empirical studies explored ecological correlates of crime. Few efforts have been made, however, to "make sense" of this literature. A "meta-analysis" was undertaken to determine the relative effects of macro-level predictors of crime. Indicators of "concentrated disadvantage" (e.g., racial heterogeneity, poverty, and family disruption) are among the strongest and most stable predictors. Except for incarceration, variables indicating increased use of the criminal justice system (e.g., policing and gettough policy effects) are among the weakest. Across all studies, social disorganization and resource/economic deprivation theories receive strong empirical support; anomie/strain, social support/social altruism, and routine activity theories receive moderate support; and deterrence/rational choice and subcultural theories receive weak support.
View