Therapist Self-Report of Evidence-Based Practices in Usual Care for Adolescent Behavior Problems: Factor and Construct Validity

Treatment Research Division, National Center on Addiction and Substance Abuse (CASA) at Columbia University, 633 Third Avenue, 19th floor, New York, NY, 10017, USA, .
Administration and Policy in Mental Health and Mental Health Services Research (Impact Factor: 3.44). 11/2012; 41(1). DOI: 10.1007/s10488-012-0442-8
Source: PubMed


This study introduces a therapist-report measure of evidence-based practices for adolescent conduct and substance use problems. The Inventory of Therapy Techniques-Adolescent Behavior Problems (ITT-ABP) is a post-session measure of 27 techniques representing four approaches: cognitive-behavioral therapy (CBT), family therapy (FT), motivational interviewing (MI), and drug counseling (DC). A total of 822 protocols were collected from 32 therapists treating 71 adolescents in six usual care sites. Factor analyses identified three clinically coherent scales with strong internal consistency across the full sample: FT (8 items; α = .79), MI/CBT (8 items; α = .87), and DC (9 items, α = .90). The scales discriminated between therapists working in a family-oriented site versus other sites and showed moderate convergent validity with therapist reports of allegiance and skill in each approach. The ITT-ABP holds promise as a cost-efficient quality assurance tool for supporting high-fidelity delivery of evidence-based practices in usual care.

9 Reads
  • [Show abstract] [Hide abstract]
    ABSTRACT: Developing therapist-report fidelity tools to support quality delivery of evidence-based practices in usual care is a top priority for implementation science. This study tested the reliability and accuracy of two groups of community therapists who reported on their use of family therapy (FT) and motivational interviewing/cognitive-behavioral therapy (MI/CBT) interventions during routine treatment of inner-city adolescents with conduct and substance use problems. Study cases (n = 45) were randomized into two conditions: (a) Routine Family Therapy (RFT), consisting of a single site that featured family therapy as its standard of care for behavioral treatment; or (b) Treatment As Usual (TAU), consisting of five sites that featured non-family approaches. Therapists and trained observational raters provided FT and MI/CBT adherence ratings on 157 sessions (104 RFT, 53 TAU). Overall therapist reliability was adequate for averaged FT ratings (ICC = .66) but almost non-existent for MI/CBT (ICC = .06); moreover, both RFT and TAU therapists were more reliable in reporting on FT than on MI/CBT. Both groups of therapists overestimated the extent to which they implemented FT and MI/CBT interventions. Results offer support for the feasibility of using existing therapist-report methods to anchor quality assurance procedures for FT interventions in real-world settings, though not for MI/CBT.
    Administration and Policy in Mental Health and Mental Health Services Research 04/2014; 42(2). DOI:10.1007/s10488-014-0548-2 · 3.44 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Observational measures to assess implementation integrity (the extent to which components of an evidence-based treatment are delivered as intended) are needed. The authors evaluated the reliability of the scores and the validity of the score interpretations for the Therapy Process Observational Coding System for Child Psychotherapy-Revised Strategies scale (TPOCS-RS; McLeod, 2010) and assessed the potential of the TPOCS-RS to assess treatment differentiation, a component of implementation integrity. The TPOCS-RS includes 5 theory-based subscales (Cognitive, Behavioral, Psychodynamic, Client-Centered, and Family). Using the TPOCS-RS, coders independently rated 954 sessions conducted with 89 children (M age = 10.56, SD = 2.00; age 7-15 years; 65.20% White) diagnosed with a primary anxiety disorder who received different treatments (manual-based vs. nonmanualized) across settings (research vs. practice). Coders produced reliable ratings at the item level (M intraclass correlation coefficient = .76, SD = .18). Analyses support the construct validity of the Cognitive and Behavioral subscale scores and, to a lesser extent, the Psychodynamic, Family, and Client-Centered subscale scores. Correlations among the TPOCS-RS subscale scores and between the TPOCS-RS subscale scores and observational ratings of the alliance and client involvement were moderate suggesting independence of the subscale scores. Moreover, the TPOCS-RS showed promise for assessing implementation integrity as the TPOCS-RS subscale scores, as hypothesized, discriminated between manual-guided treatment delivered across research and practice settings and nonmanualized usual care. The findings support the potential of the TPOCS-RS Cognitive and Behavioral subscales to assess treatment differentiation in implementation research. Results for the remaining subscales are promising, although further research is needed. (PsycINFO Database Record (c) 2014 APA, all rights reserved).
    Psychological Assessment 10/2014; 27(1). DOI:10.1037/pas0000037 · 2.99 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: A major focus of implementation science is discovering whether evidence-based approaches can be delivered with fidelity and potency in routine practice. This randomized trial compared usual care family therapy (UC-FT), implemented without a treatment manual or extramural support as the standard-of-care approach in a community clinic, to nonfamily treatment (UC-Other) for adolescent conduct and substance use disorders. The study recruited 205 adolescents (M age = 15.7 years; 52% male; 59% Hispanic American, 21% African American) from a community referral network, enrolling 63% for primary mental health problems and 37% for primary substance use problems. Clients were randomly assigned to either the UC-FT site or one of five UC-Other sites. Implementation data confirmed that UC-FT showed adherence to the family therapy approach and differentiation from UC-Other. Follow-ups were completed at 3, 6, and 12 months postbaseline. There was no between-group difference in treatment attendance. Both conditions demonstrated improvements in externalizing, internalizing, and delinquency symptoms. However, UC-FT produced greater reductions in youth-reported externalizing and internalizing among the whole sample, in delinquency among substance-using youth, and in alcohol and drug use among substance-using youth. The degree to which UC-FT outperformed UC-Other was consistent with effect sizes from controlled trials of manualized family therapy models. Nonmanualized family therapy can be effective for adolescent behavior problems within diverse populations in usual care, and it may be superior to nonfamily alternatives.
    Journal of Clinical Child & Adolescent Psychology 12/2014; 44(6):1-16. DOI:10.1080/15374416.2014.963857 · 1.92 Impact Factor