Efficacy vs Effectiveness Trial Results of an Indicated “Model” Substance Abuse Program: Implications for Public Health

Pacific Institute for Research and Evaluation, Chapel Hill, NC, USA.
American Journal of Public Health (Impact Factor: 4.55). 01/2007; 96(12):2254-9. DOI: 10.2105/AJPH.2005.067462
Source: PubMed


The US Department of Education requires schools to choose substance abuse and violence prevention programs that meet standards of effectiveness. The Substance Abuse and Mental Health Services Agency certifies "model" programs that meet this standard. We compared findings from a large, multisite effectiveness trial of 1 model program to its efficacy trial findings, upon which the certification was based.
1370 high-risk youths were randomized to experimental or control groups across 9 high schools in 2 large urban school districts. We used intent-to-treat and on-treatment approaches to examine baseline equivalence, attrition, and group differences in outcomes at the end of the program and at a 6-month follow-up.
Positive efficacy trial findings were not replicated in the effectiveness trial. All main effects were either null or worse for the experimental than for the control group.
These findings suggest that small efficacy trials conducted by developers provide insufficient evidence of effectiveness. Federal agencies and public health scientists must work together to raise the standards of evidence and ensure that data from new trials are incorporated into ongoing assessments of program effects.

Download full-text


Available from: Hyunsan Cho, Jun 27, 2014
  • Source
    • "); Reconnecting Youth (Hallfors et al. 2006); Strengthening Families Program (Gottfredson et al. 2006; Gutman et al. 2004); Toward no Drug Abuse (Rohrbach et al. 2010); and Triple P (Malti et al. 2011). Note that for some of these interventions, independent evaluations have also found positive effects. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151-175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR's Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of "the next generation" of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee's deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses "The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation."
    Full-text · Article · Apr 2015 · Prevention Science
    • "Hedges's g effect sizes (with small sample size bias adjustment) were calculated for studies reporting standardized mean differences between treatment and control conditions while odds ratio effect sizes were calculated for dichotomous outcomes and then converted to Hedges's g effect sizes (Cooper & Hedges, 1994). One study (Hallfors et al., 2006) reported nonsignificant results without providing any detailed statistical information ; therefore, an effect size of zero was substituted for nonsignificant outcomes as recommended by Perry (1997). This provides a more conservative pooled point estimate of the effect size. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Schools and school-based mental health professionals (SBMHP), such as school social workers, have become a primary service provider of mental health services for youth. Policy and research over the past 20 years have helped to evolve an intervention model known as Response to Intervention (RTI) to address various levels of identified social and behavioral student problems. Utilizing a systematic review approach, this paper sought to understand the extent to which teachers have become involved in the delivery of RTI school-based mental health services. Therefore, the purpose of this systematic review is: 1) to assess how often and at what level of intervention teachers may be involved in delivering mental health interventions using the framework of RTI, and 2) examine the efficacy of school mental health interventions implemented by teachers in conjunction with and comparison to school social workers and other professionals. Method Databases (ERIC, PsycINFO, MEDLINE, CINHAL, & Social Science Abstracts) were searched using the following keywords: school*, children or adolescent* or youth, mental health or mental health service*, and outcome study or effective or efficacy. Study inclusion criteria included: (a) experimental and quasi-experimental designs (b) published between January 1999 and September 2010; (c) conducted in a school setting; (d) involved a mental health and behavioral services such as prevention, risk reduction, and intervention; (e) conducted in the U.S.; and (f) contained enough statistical information to calculate effect sizes. Two of the authors independently coded the final sample of studies (N=49) and compared their results for agreement. Effect sizes (Hedges’ g) were calculated for all outcome measures using Comprehensive Meta-Analysis software when not reported in the articles. Results Results found that teachers were involved in 41% of mental health interventions evaluated and were the sole providers of interventions in 18% of the studies. Additionally, 41% of the school mental health interventions were tier 1 and occurred in the classrooms which also partly explain the active involvement of teachers in a relatively high percentage of these interventions. Effect size results for the studies reviewed suggest that there is not wide variation in outcomes between different school mental health studies, regardless of which personnel may be implementing the interventions. In this regard, studies involving teachers as sole providers of mental health services, and those studies that involved teachers as collaborators on mental health teams, had fairly equivocal results with most studies having small effect sizes, and only a few studies offered medium and large effect sizes. The results from studies where teachers were involved in school mental health interventions were fairly equivocal to those delivered SBMHP. Conclusions Findings suggest that teachers have been actively involved in school mental health interventions, and they are more likely to work as a team with other professionals than as sole providers of these interventions. Therefore, it is important for SBMHP to prepare to empower teachers to participate in the delivery of efficacious mental health interventions. Additional research is needed to understand the most effective role teachers can play in mental health service delivery.
    No preview · Conference Paper · Jan 2013
  • Source
    • "When teachers were the sole providers of interventions (n = 8), study effect sizes ranged from a zero (Hallfors et al., 2006), which indicates no treatment effect, to a large effect size of 1.214 (Duncan, Duncan, Beauchamp, Wells, & Ary, 2000). More specifically, three studies (Cho, Hallfors, & Sánchez, 2005; Hallfors et al., 2006; Melnyk et al., 2009) had small effect sizes, three studies (Botvin, Griffin, Diaz, & Ifill-Williams, 2001; Hennessey, 2007; Robinson, Smith, & Miller, 2002) had medium effect sizes, and one study (Duncan et al., 2000) that offered a large effect size. For those studies where teachers were involved in the delivery of the intervention along with others (n = 12), effect sizes ranged from a small effect size of 0.100 (McWhirter & Page, 1999) to a large effect size of 0.706 (Franklin, Moore, & Hopson, 2008). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Schools are large providers of mental health services for children and adolescents. Recent educational policy initiatives at the federal level have ushered in Response to Intervention and school-wide behavior supports that have potential to involve teachers in school mental health interventions. Little research exists investigating the involvement of teachers in school mental health services or the level of efficacy associated with teachers providing these services. This paper conducts a systematic review to investigate the extent to which teachers a) are the primary school-based service providers, b) collaboratively work with other professionals to provide services, and c) what levels of interventions within the Response to Intervention (RTI) framework apply to these interventions. This paper further evaluates how efficacious teachers and school mental health professionals are in impacting outcomes in previous studies. Results indicated that out of the 49 school mental health studies analyzed, teachers were actively involved in 40.8% of mental health interventions evaluated, and were the sole providers of interventions in 18.4% of the studies. It was also found that many of these school mental health interventions were universal (Tier 1) and took place in the classrooms. Further examination of findings suggested that different personnel, such as school mental health professionals and teachers, achieved similar outcomes across the studies.
    Full-text · Article · May 2012 · Children and Youth Services Review
Show more