ArticlePDF Available

Quasi-experimental designs for community-level public health violence reduction interventions: a case study in the challenges of selecting the counterfactual

Authors:
Caterina Roman at Temple University
Caterina Roman
Hannah Klein at Lewis University
Hannah Klein
Kevin Wolff at John Jay College of Criminal Justice
Kevin Wolff
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

Objectives We highlight the importance of documenting the step-by-step processes used for the selection of comparison areas when evaluating a community-level intervention that targets a large-scale community. Methods We demonstrate the proposed method using a propensity score matching framework for an impact analysis of the Cure Violence Public Health Model in Philadelphia. To select comparison communities, propensity score models are run using different levels of aggregation to define the intervention site. We discuss the trade-offs made. ResultsWe find wide variation in documentation and explanation in the extant literature of the methods used to select comparison communities. The size of the unit of analysis at which a community is measured complicates the decision processes, and in turn, can affect the validity of the counterfactual. Conclusions It is important to carefully consider the unit of analysis for measurement of comparison communities. Assessing the geographic clustering of matched communities to mirror that of the treated community holds conceptual appeal and represents a strategy to consider when evaluating community-level interventions taking place at a large scale. Regardless of the final decisions made in the selection of the counterfactual, the field could benefit from more systematic diagnostic tools that document and guide the steps and decisions along the way, and ask: “could there have been another way of doing each step, and what difference would this have made?” Overall, across community-level evaluations that utilize quasi-experimental designs, documentation of the counterfactual selection process will provide a more fine-grained understanding of causal inference.
Figures - uploaded by Caterina Roman
Author content
All figure content in this area was uploaded by Caterina Roman
Content may be subject to copyright.
Content uploaded by Caterina Roman
Author content
All content in this area was uploaded by Caterina Roman on May 25, 2018
Content may be subject to copyright.
A preview of the PDF is not available
A preview of this full-text is provided by Springer Nature.
Learn more
Content available from Journal of Experimental Criminology
This content is subject to copyright. Terms and conditions apply.
... The evaluation of FD also included an assessment of the impact of the intervention directly on the targeted groups. Details on the designs of the evaluations can be found in Roman, Link, et al. (2019) for FD and Roman et al. (2018) for CV. With regard to the implementation of Philadelphia FD, during the evaluation period, the lead agencies held four notification meetings and 16 enforcements. ...
... The research team then re-validated the geocoding process 4 and created four dichotomous 'locational' variables: The shooting occurred (1) in the CV target area; (2) a propensity scorematched CV 'control' neighborhood; (3) the FD target area; and (4) a propensity score-matched FD 'control' area. The two sets of matched control areas were calculated for the original evaluations; details about how the counterfactuals were created can be found in Roman et al. (2018) and Roman, Link, et al. (2019). ...
... The data were derived from the 2007-2011 American Community Survey and other local data sources. More detailed information on the operationalization of these variables can be found in Roman et al. (2018). ...
View
... Evaluating complex interventions such as GVI inherently present methodological challenges (Brantingham et al., 2021;Corsaro, 2018). As a practical matter, GVI implementations in most cities do not involve randomly assigned treatment and control groups; therefore, randomized controlled trials are often not a feasible evaluation option (Braga et al., 2001(Braga et al., , 2018Roman et al., 2018;Saunders et al., 2015;Weiner et al., 2007). ...
... This study had several limitations. As is typical for GVI implementations, the current implementation in Philadelphia did not entail randomly assigned treatment and control groups; therefore, randomized controlled trials were not a feasible evaluation option (Braga et al., 2001(Braga et al., , 2018Roman et al., 2018;Saunders et al., 2015;Weiner et al., 2007). Instead, the effect of the GVI implementation on firearm violence was quasi-experimentally assessed through variation in treatment start times for group-units as well as for census tracts (Braga et al., 2019;Mark & Reichardt, 2009;Roman et al., 2019;Welsh & Farrington, 2001). ...
Assessing the effects of the Group Violence Intervention on firearm violence in Philadelphia
Article
Full-text available
  • Dec 2023
  • J Exp Criminol
Objectives This study assesses the effects of a Group Violence Intervention (GVI) implementation in Philadelphia on group member-involved (GMI) firearm violence. Because the implementation began in August 2020 during the COVID-19 pandemic, public health restrictions necessitated relying on individualized Mobile Call-In Team (MCIT) custom notifications, rather than large-scale call-in meetings, as the primary implementation method. Methods During the January 2020–May 2022 study period, not all at-risk group-units received GVI treatment at the same time. Likewise, not all census tracts received GVI treatment at the same time. Given this variation in treatment initiation, a quasi-experimental stepped wedge design assessed the effect of GVI treatment on GMI shootings on the dimensions of both group and place. Estimates were calculated using Poisson regression. The effects of treatment dosage were also assessed. Results A group-unit, post-treatment relative to pre-treatment, experienced, on average, a significant 38.6% reduction in shootings per week. Where a census tract received between 4 and 7 doses relative to 0 doses (pre-treatment), there was a significant 51.0% reduction in GMI shootings per week. Conclusions A GVI implementation through custom notifications appears to maintain the effectiveness of GVI. Future research should assess the role of GVI components, including both enforcement actions and social services, as mechanisms for GVI effectiveness in a custom notification-based implementation.
View
... The three areas selected for the CV intervention were selected because they had the highest levels of firearm violence in the 3-year period leading up to program implementation. Following prior research (Roman et al., 2017(Roman et al., , 2018, we employed an iterative process for selecting comparison sites. We focused on five selection criteria, which included size, location, levels of firearm violence, population demographics, and proximity to the CV interventions. ...
Short-term evaluation of Cure Violence St. Louis: Challenges, triumphs, and lessons learned
Article
Full-text available
  • Oct 2024
  • Criminol Publ Pol
Research Summary Firearm violence continues to be a leading cause of death in the United States. As alternatives to law enforcement intervention, community‐based violence prevention programs, such as Cure Violence, have become increasingly popular across U.S. cities. This article documents the results of a multiyear, mixed‐methods, quasi‐experimental study of the implementation and impact of Cure Violence in St. Louis, Missouri, from 2020 to 2023. We analyzed data from semistructured interviews, a two‐wave community survey, and police records of neighborhood violence, finding limited evidence of program effectiveness. Although some interview participants stated that the program had a positive impact, we observed no change in community norms/perceptions of violence, and our analysis of police data suggests that program implementation was associated with declining violence in only one of the three intervention sites. Policy Implications This article highlights challenges of implementing and evaluating community‐based violence prevention. More work is needed to unpack the mechanisms responsible for turning program activities into measurable impacts in both the short and long term. Implications for planning and evaluating community‐based violence prevention programs are discussed.
View
... Contemporary evaluations of streetwork initiatives have universally measured program impact at the area level. Here, we advance streetwork program evaluation methodology by assessing the programmatic impact of SSB at the gang level; among other benefits, this approach yields direct measures of program impact closely bound to program theory and activities, while avoiding the many practical and methodological challenges involved in developing area-level counterfactuals (see Roman et al., 2018). ...
Streetwork at the crossroads: An evaluation of a street gang outreach intervention and holistic appraisal of the research evidence
Article
  • Oct 2023
Spurred by the success of public health violence interventions, and accelerated by policy pressure to reduce violence without exacerbating overpolicing and mass incarceration, streetwork programs—those that provide anti‐violence services by neighborhood‐based workers who perform their work beyond the walls of parochial institutions—have positioned themselves as the most important non–law‐enforcement violence prevention option available to urban policy makers. Yet despite their importance, the state of the field seems difficult to interpret for academics and practitioners alike. In this article, we make several contributions that bring forth new findings and deliver new perspectives on streetwork as a violence reduction strategy. First, we offer an extended analytic review of the streetwork evaluation literature that connects the study of contemporary public health violence interventions to a preceding tradition of criminologically inspired streetwork studies. Second, we present the results of an impact evaluation of StreetSafe Boston (SSB)—a multiyear streetwork intervention that served 20 Boston gangs. We find that the SSB intervention had no detectable effect on violence among the gangs that it served. We conclude by offering a framework for understanding a field at multiple crossroads: past and present, proclaimed successes and failures, help and harm.
View
... Evaluations must compare the effects of interventions with estimates of the "counterfactual," or what would have happened to key outcomes if an intervention had not been implemented (Roman, Klein and Wolff 2018). Simply measuring outcomes before and after an intervention, or what researchers call a pre-post design, is not reliable evidence. ...
Community Violence Intervention at the Roots (CVI-R): Building Evidence for Grassroots Community Violence Prevention
Technical Report
Full-text available
  • Jun 2022
The crime and justice field recently started to label a wide array of violence prevention strategies as Community Violence Interventions (or CVI). Many of these strategies depend on law enforcement and social services, but the most innovative approaches are community-centered and community-sourced. They are grassroots efforts that rely on the resources of neighborhoods and residents themselves, operating separately from law enforcement and traditional human services. These strategies could be called Community Violence Interventions at the Roots (or CVI-R). The most established CVI-R programs are Cure Violence and Advance Peace. They offer highly localized and potentially cost-effective approaches to public safety, but do they work? Evaluation evidence is recent and not yet consistent, but the grassroots approach to community violence prevention is highly promising. To build sustainable CVI-R models, communities and researchers must collaborate in designing rigorous evaluations to produce reliable and actionable evidence.
View
... 30 However, studies of Cure Violence's impact on gun violence outcomes, while generally positive, have produced varied estimates of program effect across sites-sometimes within the same city-with some studies reporting null or negative results. 31,32,33,34 The Operation Peacemaker Fellowship in Richmond, CA, from which the Advance Peace model was created, was associated with statistically significant reductions in firearm violence, but the evaluators attributed a smaller yet nontrivial increase in non-firearm violence to the model as well. 35 An evaluation of Advance Peace in Sacramento found that the program was associated with mean gun violence incident decreases ranging from -8% to -29% in the four communities where the program was implemented, compared with an overall +9% increase in gun violence incidents in non-Advance Peace zones, but interrupted time series analyses did not find those reductions to be statistically significant. ...
View
Perceptions of Violent Crime Among a Sample of Bronx College Students: Results From Focus Groups
Article
  • Sep 2024
Purpose: This study sought to understand how a convenient sample of college students between the ages of 20 and 35 years of age in the Bronx perceive and experience violent crime (VC). The focus of the study is on perceptions of risk and an exploration of crime prevention strategies. Methods: This project conducted two online focus groups with a sample of 24 young adults residing in the Bronx, NY, enrolled at the higher education institution in Spring 2021 and Fall 2022. The main research questions are the following: (a) How do young adults in the Bronx perceive and better understand VC risk? and (b) What are the young people’s proposals or ideas to develop prevention strategies to address VC in the Bronx? The focus groups were held online using Zoom and Mentimeter. Results: Study participants expressed concerns about feeling unheard and feeling of a constant threat to their safety. They expressed a lack of trust for the existing political and social programs and did not express strong community ties and community engagement. This research suggests a promise of using spatial research to engage young adults in the Bronx in VC prevention and advocacy programs. Conclusions: Overall, this study found an active voice in young adults when discussing VC. Participants expressed a need for community dialogue and action to address the increasing trend of VC in the Bronx.
View
View
Co-Developing Theories of Change for Improved Community-Based Violence Intervention Evaluation
Article
  • Mar 2024
Background Community-based violence intervention (CVI) programs are considered important strategies for preventing community violence and promoting health and safety. Mixed and inconclusive results from some prior CVI evaluations—and our general lack of understanding about the reasons for such varied findings—may be explained in part by misalignment of program theories of change and evaluation measures. Further, most prior evaluations have focused solely on deficit-based outcomes; this narrow focus is inconsistent with the premise of CVI and may fail to capture improvements in health and wellbeing that are on the hypothesized pathway from intervention to violence reduction. Methods This paper describes the process and results of co-developing a theory of change for community-based youth firearm violence intervention and prevention programs in Washington state through a community-researcher partnership. We followed a multi-step iterative process, involving 1) CVI program documentation review, 2) individual meetings, and 3) a day-long workshop. Results The theory of change included 6 key domains: 1) root causes, 2) promotive factors, 3) activities, 4) inter-mediate outcomes, 5) longer-term outcomes, and 6) multi-level context (youth/family, staff/organizational, community, and societal). Root causes were social and structural drivers of community violence. Promotive factors were assets and resources among the community, youth/their families, and community organizations that promote health and safety. Activities were supports and services the program provided to youth and their families, staff, and potentially the broader community. Inter-mediate and longer-term outcomes were the changes among youth, their families, staff, and the community that resulted from program activities. Inter-mediate outcomes may be felt within 6 months to 1 year and longer-term outcomes may be felt after 1-2 years and beyond. Conclusions The theory of change we co-developed provides a common lens to conceptualize, compare, and evaluate CVI programs in Washington state and may support more rigorous and equity-centered evaluations. Study type: original investigation Level of evidence N/A
View
View
Community-Based Violence Prevention: An Assessment of Pittsburgh’s One Vision One Life Program
Book
Full-text available
  • May 2010
In 2006, more than 6 million individuals were victimized by violent crimes in the United States. Although violence is below levels of the early 1990s, it remains high. The extent of violence and its impact highlight a critical need to develop and implement effective programs to reduce violence and victimization. Communities have initiated a wide range of such programs, and scholars have conducted numerous evaluations of varying quality of them. Reviews have found certain types of strategies and specific programs to be promising, but additional critical evaluations are needed to plan violence-reduction programs. This monograph assesses the implementation and impact of the One Vision One Life violence-prevention strategy in Pittsburgh, Pennsylvania. In 2003, Pittsburgh witnessed a 49-percent increase in homicides, prompting a “grassroots” creation and implementation of the One Vision One Life antiviolence strategy. This initiative used a problem-solving, data-driven model, including street-level intelligence, to intervene in escalating disputes, and seeks to place youth in appropriate social programs. Analysis of the program, which is modeled on similar efforts elsewhere, can help inform other efforts to address urban violence.
View
View
Communities are not all created equal: Strategies to prevent violence affecting youth in the United States
Article
Full-text available
  • Sep 2016
  • J PUBLIC HEALTH POL
We describe violence in the United States (US) and solutions the Urban Networks to Increase Thriving Youth (UNITY) Initiative has developed, led by Prevention Institute, a US non-governmental organization (NGO) and authors of this article, with initial funding from the US Centers for Disease Control and Prevention (CDC). Safety distribution across populations is unequal, while public health research has identified aspects of community environments that affect the likelihood of violence, or risk and resilience factors. An overwhelming number of risk factors have accumulated in some US communities, disproportionately impacting young people of color. US policies, systems, and institutions powerfully shape how and where these factors manifest. Violence is preventable, not inevitable. We argue that comprehensive strategies for improving community environments can reduce violence and promote health equity. We present lessons, tools, and frameworks that UNITY cities use to adapt for international application, including multi-sector collaboration, strategies for influencing policy and legislation, and strengthening local violence prevention efforts.
View
Implementing and Evaluating Comprehensive Evidence-Based Approaches to Prevent Youth Violence: Partnering to Create Communities Where Youth Are Safe From Violence
Article
Full-text available
  • Apr 2016
  • J Prim Prev
Violence, including its occurrence among youth, results in considerable physical, emotional, social, and economic consequences in the U.S. Youth violence prevention work at the Division of Violence Prevention (DVP) at the Centers for Disease Control and Prevention (CDC) emphasizes preventing youth violence-related behaviors, injuries, and deaths by collaborating with academic and community partners and stakeholders. Since 2000, DVP has funded three rounds of CDC's National Centers of Excellence in Youth Violence Prevention (YVPCs) in 5-year cycles, with the goal of supporting university-community partnerships so that the best science can be utilized in order to prevent youth violence. The current YVPCs focus on: (a) partnering with communities to identify community needs; (b) selecting and implementing the best comprehensive evidence-based programs to meet those needs; and (c) rigorously evaluating whether those efforts have a community-level impact on youth violence rates. The introduction to this special issue on the current YVPCs provides a brief historical overview on the YVPC Program; outlines the YVPCs' accomplishments to date; and describes the current YVPCs, their community partners, and their activities. The introduction concludes with an overview of the special issue.
View
Designs for Evaluating the Community-Level Impact of Comprehensive Prevention Programs: Examples from the CDC Centers of Excellence in Youth Violence Prevention
Article
Full-text available
  • Apr 2016
  • J Prim Prev
This article discusses the opportunities and challenges of developing research designs to evaluate the impact of community-level prevention efforts. To illustrate examples of evaluation designs, we describe six projects funded by the Centers for Disease Control and Prevention to evaluate multifaceted approaches to reduce youth violence in high-risk communities. Each of these projects was designed to evaluate the community-level impact of multiple intervention strategies to address individual and contextual factors that place youth at risk for violent behavior. Communities differed across projects in their setting, size, and how their boundaries were defined. Each project is using multiple approaches to compare outcomes in one or more intervention communities to those in comparison communities. Five of the projects are using comparative interrupted time-series designs to compare outcomes in an intervention community to matched comparison communities. A sixth project is using a multiple baseline design in which the order and timing of intervention activities is randomized across three communities. All six projects are also using regression point displacement designs to compare outcomes within intervention communities to those within broader sets of similar communities. Projects are using a variety of approaches to assess outcomes including archival records, surveys, and direct observations. We discuss the strengths and weaknesses of the designs of these projects and illustrate the challenges of designing high-quality evaluations of comprehensive prevention approaches implemented at the community level.
View
Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores
Article
Full-text available
  • Dec 2015
  • J Exp Criminol
Objectives Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.
View
Firearms and Violence: A Critical Review
Book
  • Jan 2005
For years proposals for gun control and the ownership of firearms have been among the most contentious issues in American politics. For public authorities to make reasonable decisions on these matters, they must take into account facts about the relationship between guns and violence as well as conflicting constitutional claims and divided public opinion. In performing these tasks, legislators need adequate data and research to judge both the effects of firearms on violence and the effects of different violence control policies. Readers of the research literature on firearms may sometimes find themselves unable to distinguish scholarship from advocacy. Given the importance of this issue, there is a pressing need for a clear and unbiased assessment of the existing portfolio of data and research. Firearms and Violence uses conventional standards of science to examine three major themes - firearms and violence, the quality of research, and the quality of data available. The book assesses the strengths and limitations of current databases, examining current research studies on firearm use and the efforts to reduce unjustified firearm use and suggests ways in which they can be improved. © 2005 by the National Academy of Sciences. All rights reserved.
View
View
Natural and Quasi-Experiments in Economics
Article
Using research designs patterned after randomized experiments, many recent economic studies examine outcome measures for treatment groups and comparison groups that are not randomly assigned. By using variation in explanatory variables generated by changes in state laws, government draft mechanisms, or other means, these studies obtain variation that is readily examined and is plausibly exogenous. This article describes the advantages of these studies and suggests how they can be improved. It also provides aids in judging the validity of inferences that they draw. Design complications such as multiple treatment and comparison groups and multiple preintervention or postintervention observations are advocated.
View
Assessing Macro-Level Predictors and Theories of Crime: A Meta-Analysis
Article
  • Jan 2005
  • CRIME JUSTICE
The macro-level approach reemerged as a salient criminological paradigm in the late 1970s and early 1980s. Prompted by new theories and reformulations of existing ones, over 200 empirical studies explored ecological correlates of crime. Few efforts have been made, however, to "make sense" of this literature. A "meta-analysis" was undertaken to determine the relative effects of macro-level predictors of crime. Indicators of "concentrated disadvantage" (e.g., racial heterogeneity, poverty, and family disruption) are among the strongest and most stable predictors. Except for incarceration, variables indicating increased use of the criminal justice system (e.g., policing and gettough policy effects) are among the weakest. Across all studies, social disorganization and resource/economic deprivation theories receive strong empirical support; anomie/strain, social support/social altruism, and routine activity theories receive moderate support; and deterrence/rational choice and subcultural theories receive weak support.
View