No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Assessing Parents’ Treatment Fidelity: A Survey of Practitioners in Home Settings
Abstract
Children with autism spectrum disorder (ASD) and other developmental disabilities may benefit from home-based interventions to promote positive, prosocial behavior. Frequently, a practitioner (e.g., behavior analyst) provides training and support to parents to achieve behavior change. When this occurs, progress-monitoring data pertaining to both treatment fidelity and child outcomes are important to deciding if supports delivered are effective. Yet, little is known about treatment fidelity assessment in this setting. In the current study, behavior analysts working across the United States in homes ( n = 314) were surveyed. Results indicate that nearly all participants reported having received some or extensive training on the importance of treatment fidelity. Most respondents also indicated that treatment fidelity is assessed in at least 30% of sessions in homes when working with parents, primarily by means of direct observation versus indirect methods. Implications for future consultation research and the practice are provided.
... To our knowledge, only two surveys have specifically queried ABA practitioners about procedural fidelity in practice. Fallon et al. (2020) asked BCBAs who supervised caregiver-implemented ABA services with their children with ASD about their training in procedural fidelity, how often they collected fidelity data during supervision, and how they collected fidelity data. Fallon et al. reported that nearly all of the 314 respondents (i.e., 97.1%) received some training related to the importance of collecting procedural-fidelity data and data-collection methods, and 77.7% of the participants indicated that they thought their training was sufficient. ...
... With regard to how procedural-fidelity data were collected, an overwhelming majority of participants (i.e., 89.8%) said that they used direct observation (i.e., observing the behaviorchange agent implementing the procedures live or from a recording) to assess procedural fidelity. Approximately 70% of respondents said they used an indirect assessment (i.e., self-report, interview, or permanent products) to assess fidelity for at least 10% of sessions (Fallon et al., 2020). A greater reliance on direct observation is promising, given that indirect assessments can be inaccurate and unreliable, especially without additional training Gresham et al., 2017). ...
... The data collected by Fallon et al. (2020) represented an advancement in the knowledge of practice, but the sample was limited to BCBAs supervising in-home ABA intervention implemented by caregivers with their children with ASD. Thus, recruiting participation from additional supervising BCBAs would permit an assessment of other practice areas and settings. ...
Evaluating the extent to which applied-behavior-analytic interventions are carried out accurately (i.e., procedural fidelity) is important for quality control, data-based decision making, and facilitating optimal consumer outcomes. This study explored several questions related to procedural fidelity in practice by distributing a survey to behavior analysts who currently supervise or provide applied-behavior-analytic services in any setting and with any population. Specifically, we were interested in learning more about who, how, how often, for what, why, and where behavior analysts assess procedural fidelity in practice. The results from 203 behavior analysts who completed the survey revealed that behavior analysts and their supervisees were most likely to collect fidelity data using checklists while directly observing behavior-analytic services in various settings. The most common barriers to collecting fidelity data in practice were a lack of resources, no requirement to do so by employers, and limited supervision time.
... Next, the supervisor or observer creates a checklist or data collection system using those steps and conducts a direct 3 or indirect assessment of procedural fidelity. Although this process may seem straightforward, multiple reviews of behavior-analytic literature suggest that assessing procedural fidelity is not a wellestablished practice in research (e.g., Cymbal et al., 2022;Falakfarsa et al., 2022;Han et al., 2022;McIntyre et al., 2007;Peterson et al., 1982), and too little is known about procedural fidelity data collection in practice to draw conclusions (Fallon et al., 2020). Direct assessments of procedural fidelity involve trained observers collecting data while observing the behavior-change agent implementing the procedures live or via recording. ...
... To conclude, we acknowledge that proceduralfidelity data in behavior-analytic research remains underreported (e.g., Cymbal et al., 2022;Falakfarsa et al., 2022;Han et al., 2022) and that little is known about procedural-fidelity assessment in practice (Fallon et al., 2020). Therefore, it is likely that any assessment of procedural fidelity in research and practice is better than no assessment. ...
Unlabelled:
Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.
Supplementary information:
The online version contains supplementary material available at 10.1007/s43494-023-00094-w.
... Sr+ = positive reinforcer questions about assessing procedural integrity. In a survey of board certified behavior analysts who supervised behaviorbased interventions for autistic children/children with autism spectrum disorder implemented by caregivers in their homes, Fallon et al. (2020) found that most of the participants agreed that procedural integrity was a key component to intervention success and that they had the necessary training and skills to assess procedural integrity in practice. Nevertheless, data reported by Fallon et al. suggested that there is still room for improvement in monitoring integrity in practice. ...
Procedural integrity refers to the extent to which an independent variable is implemented as described. Measuring procedural integrity is one important factor when considering internal and external validity of experiments. Experimental articles in behavior-analytic journals have rarely reported procedural-integrity data. The purpose of this study was to update previous reviews of whether articles published in the Journal of Applied Behavior Analysis reported procedural integrity, spanning a period from 1980 to 2020, and compare reporting in JABA to recent reviews of studies published in Behavior Analysis in Practice (2008–2019) and the Journal of Organizational Behavior Management (2000–2020). Procedural integrity continues to be underreported across all three journals, but an increasing trend in reporting procedural integrity is evident in the Journal of Applied Behavior Analysis and Behavior Analysis in Practice. In addition to our recommendations and implications for research and practice, we provide examples and resources to assist researchers and practitioners with recording and reporting integrity data.
... To ensure internal validity in research experiments and promote treatment adherence in practice, behavior analysts must measure procedural fidelity, or the extent to which procedures are implemented as designed. While procedural fidelity has been increasingly investigated and discussed in the behavioral analysis literature over the past several decades (e.g., DiGenarro Reed & Codding, 2014;Falakfarsa et al., 2022;Fallon et al., 2020), the topic has received little attention in applied animal behavior research and practice. Kodak et al. (this issue) review the behavior-analytic procedural fidelity literature, including basic and applied studies with human participants, and draw clear connections to applied animal behavior research and practice. ...
Though operant learning has been applied to socially significant animal behavior for many years, connections between these practical applications and the basic science that supports them have weakened over time. There is a need for replications and extensions of technologies derived from basic research to applied animal settings, and for practical questions to be taken back to the lab where they can be modeled and studied under controlled conditions before incorporating the results in applied behavior‐change research and practice. This special issue highlights ways that behavior analysis can contribute to and support the development of evidence‐based applications with animals. Articles in this issue provide context for the relationship between basic research and practice in animal behavior, apply basic principles to animal behavior practice, and investigate practical problems using basic research techniques. Each of these is important for a robust interchange between basic science and practice. Here we comment on the contributions of each article to the literature and identify directions for future research.
... This study has the following limitations. First, the review discussed parent treatment fidelity, and higher levels of parent treatment fidelity may lead to more positive treatment outcomes [66]. However, two of the studies lacked specific scores for fidelity, which could limit the overall scope of the studies for this review. ...
The purpose of this review is to provide an overview of findings from selected research published between 2012 and 2022 in English-language peer-reviewed journals to evaluate the effectiveness of the parent-implemented Early Start Denver Model (P-ESDM). Thirteen studies used quantitative methods to examine the relationship between variables. We considered four categories of variables in the analysis: child characteristics, intervention intensity and duration, child outcome measures, and parent-related outcome measures and parental fidelity. The findings revealed positive child–parent-related outcomes. In addition, the quality of implementation, relating to parent fidelity, should be considered when evaluating the efficacy of the intervention. However, only half the studies revealed that the standard benchmark for acceptable fidelity was being achieved. Implications for future research and practice are discussed.
... Treatment fidelity (category 11) plays an important role in behavior analytic research (42). There is limited research on the topic within neurobehavioral rehabilitation. ...
Rationale:
As acquired brain injury rates continue to increase, the ongoing need for efficient and effective treatment within neurobehavioral rehabilitation settings is clear. Some evidence suggests certain treatment components may be very important to incorporate into service delivery models (e.g., multidisciplinary). However, program evaluation literature and the uptake of complementary intervention strategies, like applied behavior analysis (ABA), in existing neurobehavioral settings remains largely unexplored.
Primary objective:
The purpose of this project was to: (1) develop and implement a simple, systematic program evaluation informed by best-practices (i.e., research) to assess service delivery models of several neurobehavioral rehabilitation settings, and (2) survey the current use of ABA by participating neurobehavioral agencies.
Methodology:
The program evaluation tool was applied to the charts of randomly selected past and current clients (referred to as participants). A secondary research assistant independently reviewed 29% of the charts to conduct interobserver agreement, which s. was 80% (range, 53%-100%).
Results:
Average program evaluation total percentage score was 33% (range, 4% - 63%), and program evaluation items describing ABA-uptake suggested the incorporation of ABA was low.
Discussion:
We discuss service model areas of strengths and areas for improvement as specified by tool outcomes, as well as in relation to quality improvement implications.
... Treatment fidelity (category 11) plays an important role in behavior analytic research (42). There is limited research on the topic within neurobehavioral rehabilitation. ...
Ensuring high levels of procedural fidelity during behavior-analytic interventions is a crucial component of providing effective behavior-analytic services. However, few resources are available to help guide practitioners through measuring procedural fidelity. In fact, most published behavior-analytic research on procedural fidelity analyzes a single treatment procedure, which might not completely reflect the process of monitoring and addressing the procedural fidelity of a robust treatment package that might be necessary in clinical settings. The purpose of this article is to guide behavior analysts through the process of creating and using procedural fidelity measurement systems, with a focus on direct observation of implementation as a means of fidelity data collection. This process consists of six steps: (1) task analyze treatment procedures into measurable units; (2) assign measures to each treatment component; (3) plan the direct observation; (4) collect procedural fidelity data; (5) analyze and interpret procedural fidelity data; and (6) take action to improve procedural fidelity. Each step is described and discussed in the article.
A recent study described a synthesized treatment where a functional analysis was based on an open-ended interview and combined with functional communication and delay-tolerance training. The treatment resulted in a reduction in problem behaviors and an increase in appropriate requests. Most of the analysis and intervention were done in a clinic setting, and required weekly visits by the family. This present study is a systematic replication, where we conducted the same synthesized treatments, with a young child with Autism Spectrum Disorder (ASD) enrolled in a home-based Early Intensive Behavior Intervention program, EIBI. Outcomes were similar with a marked reduction in problem behaviors and an increase in appropriate requests. These findings suggest that it is possible to conduct this intervention in a home setting, with weekly consultations with parents. Our study show the utility of the synthesized treatment in an EIBI program in a home setting and how this can contribute to client time and costs.
In consultation, school psychologists may offer educators direct training to support the implementation of classroom interventions aimed to improve student outcomes. The purpose of this study was to conduct a systematic literature review of single-case design research studies during which educators received direct training to implement a classroom intervention, specifically instructions, modeling, practice, and feedback. Two doctoral students in school psychology screened 228 articles and evaluated 33 studies to determine if direct training is effective and an evidence-based practice per single-case design standards proposed by the What Works Clearinghouse. Results of the review indicate that there is support for the practice to be deemed evidence-based and associated with better intervention implementation than before its application. Implications include direct training being considered for intensive, complex interventions to promote educator success with implementation. (PsycINFO Database Record
Access to behavior-analytic professionals who can implement interventions like discrete-trial instruction (DTI) may be limited for families living in rural communities. We trained four rural parents of children with autism to implement DTI accurately with a confederate using in vivo behavioral skills training. The DTI skills that parents demonstrated with a confederate generalized to working with their children. We then assessed whether periodic supervision during videoconferencing maintained parents’ DTI skills following training. Parents accurately implemented untrained DTI programs with their children and maintained accurate implementation over several months of remote supervision occurring twice a week, weekly, and biweekly. Videoconferencing may be an effective way for individuals responsible for direct implementation to maintain contact with behavior analysts who are not locally accessible.
Today is an exciting time for women in behavior analysis. Over the years, multiple articles have documented increases in women’s participation in behavior analysis (e.g., McSweeney and Swindell 1998; McSweeney et al. 2000; Myers 1993; Poling et al. 1983; Simon et al. 2007). Contemporary data, however, depict an even more striking degree of participation. For example, 82.2 % of Behavior Analyst Certification Board® (BACB®) certificants are female,1 including 68.3 % of those who are certified at the doctoral level (i.e., BCBA-D™). These data represent a 148 % increase in female certificants over the last 15 years. The Association for Behavior Analysis International (ABAI) reported that 52 % of their full members in 2014 were women (personal communication ABAI, April 7, 2015).2 Female authors accounted for 55.5 % of authors who published in the Journal of Applied Behavior Analysis (JABA) in 2014. These data represent a 142 % increase since the first volume of JABA was published in 1968. In addition, 27.1 % of authors who published in the Journal of the Experimental Analysis of Behavior (JEAB) in 2014 were female. These data represent a 115 % increase since the average from 1978 to 1982. Given the representation of women in behavior analysis, a demographic population that is typically a minority in other scientific disciplines (National Science Foundation 2015), has increased substantially over a relatively short period of time (~20 years), we thought it an important time to provide accounts from the perspectives of prominent women who experienced behavior-analytic academic training and professional environments over this period of time and get advice based on what they learned.
Behavioral interventions delivered across home and school settings can promote positive outcomes for youth with autism spectrum disorders (ASD). Yet, stakeholders who deliver these interventions may struggle to implement interventions as intended. Low levels of treatment integrity can undermine potentially positive intervention outcomes. One way to promote implementers' treatment integrity is Implementation Planning, a logistical planning and barrier reduction strategy that is supported by emerging school-based research. The current study extended the research on Implementation Planning and evaluated the effectiveness of the strategy with parents implementing a behavioral intervention at home within a Conjoint Behavioral Consultation model. The behavioral intervention aimed to increase compliance and decrease aggression for two children with ASD at home. Initially, parents struggled to deliver the intervention consistently; however, after Implementation Planning, parents' treatment integrity increased and, subsequently, child outcomes improved. Implications for future research and consultation are presented.
Optimal levels of treatment fidelity, a critical moderator of intervention effectiveness, are often difficult to sustain in applied settings. It is unknown whether performance feedback, a widely researched method for increasing educators' treatment fidelity, is an evidence-based practice. The purpose of this review was to evaluate the current research on performance feedback as a strategy to promote the implementation of school-based practices. Studies were evaluated according to What Works Clearinghouse (WWC; Kratochwill et al., 2010) technical guidelines for single-case design, utilizing both the design and evidence standards to determine whether studies provided sufficient evidence for the effectiveness of performance feedback. Results indicate that performance feedback can be termed an evidence-based intervention based on criteria set by the WWC. Implications for future research are described.
Performance feedback (PF) is an empirically supported method of increasing teachers' treatment adherence. In the evaluations of PF to date, however, PF was delivered by someone external to the school. The primary purpose of this case study was to provide an exploratory evaluation of the effectiveness and feasibility of PF when delivered by a school-based consultant to teachers implementing a classwide behavioral intervention. Teacher treatment adherence and student outcomes were also evaluated. Results indicate that overall, (a) teachers demonstrated moderate-to-high, but variable, levels of treatment adherence; (b) when teachers demonstrated lower levels of treatment adherence, the school-based consultant implemented PF with high adherence for 5 weeks, after which adherence varied; and (c) student outcomes improved.
ABSTRACT
We argue that the field of behavior analysis would be best served if behavior analysts worked to extend the reach of behavioral services into a more diverse range of settings and with more varied populations, with an emphasis on the establishment of new career opportunities for graduating students. This is not a new proposal, but it is a tall order; it is not difficult to see why many would choose a surer route to gainful employment. Currently, the most fruitful career path for behavior analysts in practice is in the area of autism and developmental disabilities. For the continued growth of the field of behavior analysis, however, it is important to foster new career opportunities for those trained as behavior analysts. Toward this end, we identify several fields that seem well suited to behavior analysts and summarize the training requirements and likely professional outcomes for behavior analysts who pursue education and certification in these fields. These fields require relatively little additional formal training in the hopes of minimizing the response effort necessary for individuals who have already completed a rigorous program of graduate study in behavior analysis.
The current review evaluates the use of treatment fidelity strategies in evidence-based parent training programs for treating externalizing disorders. We used a broad framework for evaluating treatment fidelity developed by the National Institutes of Health Treatment Fidelity Workgroup that includes the aspects of treatment design, treatment delivery, training providers, and assessment of participant receipt of treatment and enactment of treatment skills. Sixty-five articles reporting outcome trials of evidence-based parent training programs met inclusion criteria and were coded for treatment fidelity strategies. The mean adherence to fidelity strategies was .73, which was higher than two previous review studies employing this framework in the health literature. Strategies related to treatment design showed the highest mean adherence (.83), whereas training of providers and enactment of treatment skills had the lowest (.58). In light of an increasing emphasis on effectiveness and dissemination trials, the broader treatment fidelity framework as applied in this review focuses needed attention on areas often overlooked in fidelity practices, such as training providers and generalization of treatment skills. We discuss the strengths and limitations of fidelity practices in parent training studies, implications of these findings, and areas for future research.
The current study examined the extent to which treatment integrity was increased and maintained for 4 teachers in their regular classroom settings as a result of performance feedback and negative reinforcement. Teachers received daily written feedback about their accuracy in implementing an intervention and were able to avoid meeting with a consultant to practice missed steps by implementing the procedure with 100% integrity. Treatment integrity increased for all 4 teachers and gains were maintained over time. Decreases in off-task behavior were observed for 3 of the 4 student participants. Results suggest that an intervention package of performance feedback and negative reinforcement may be a viable, time-efficient technique for increasing the integrity of plan implementation by teachers in the classroom.
Treatment integrity data are essential to drawing valid conclusions in treatment outcome studies. Such data, however, are not always included in peer-reviewed research articles in school psychology or related fields. To gain a better understanding of why treatment integrity data are lacking in the school psychology research, we surveyed the authors of the 210 treatment outcome articles published in four school psychology journals from 1995 through 2008 regarding their perceptions of barriers to implementing treatment integrity procedures. Results indicated that (a) lack of theory and specific guidelines on treatment integrity procedures; (b) lack of general knowledge about treatment integrity; (c) time, cost, and labor demands; and (d) lack of editorial requirement were broadly perceived as barriers by school psychology researchers to implementing treatment integrity procedures. Implications for future research are discussed.
This study examined teachers' implementation of treatment plans fol- lowing consultation. Interventions were implemented for 45 elementary school students referred for consultation and intervention due to academic concerns, chal- lenging behavior, or a combination of the two. The consultation follow-up proce- dures examined were brief weekly interviews, weekly interviews combined with an emphasis on the commitment to implement the treatment, and performance feedback. Performance feedback was associated with superior treatment imple- mentation and child behavioral outcomes when compared to the two other condi- tions. Treatment implementation did not differ for the weekly follow-up meeting and the commitment emphasis conditions at a statistically significant level. Teacher ratings of consultants and treatment acceptability were similar across conditions. A moderate statistically significant correlation between treatment integrity and child behavioral outcome was obtained. The correlation between treatment ac- ceptability and implementation was quite small and was not statistically signifi- cant. The implications of these findings for consultation and intervention are dis- cussed.
The purpose of this study Was to assess the degree to Which behavioral intervention studies conducted With children diagnosed With autism operationally defined the independent variables (IVs) and evaluated treatment integrity (TI). The study replicated the criteria from Gresham, Gansle, and Noell (1993) and focused exclusively on research studies conducted With children diagnosed With autism Who Were age 18 years or younger. Results indicated that of the 60 studies that Were critically revieWed, 11 operationally defined the IVs and assessed TI. Conversely, the majority of the studies (n = 41) operationally defined the IVs yet did not assess TI. Last, 3 studies operationally defined the IVs and referred to TI Within the paper but provided no data, and 5 studies did not define IVs or assess TI.
Behavioral consultation has been shown to be an effective way for school psychologists to work with teachers in implementing interventions for student problem behavior. Some teachers are resistant to the behavioral consultation process and thereby fail to implement agreed upon interventions with integrity, which is problematic considering the research linking treatment integrity to student behavior change. The purpose of this study was to evaluate the effects of a test-drive procedure to improve teachers' treatment integrity and student outcomes. The test-drive procedure was evaluated using a nonconcurrent multiple baseline design. Four elementary school teachers who initially demonstrated low adherence to intervention protocols tried several interventions and then proceeded implementing the one they found most acceptable. Results indicated that teachers who were resistant to the traditional behavioral consultation process implemented interventions with higher rates of treatment integrity once they were able to test drive several interventions and select the one they found most acceptable. Higher levels of treatment integrity were also associated with increased student academic engaged time. The implications and future directions of the findings are also discussed.
A majority of evidence-based interventions in schools are delivered through consultation models and are implemented by a mediator, such as a teacher. Research indicates that mediators do not always adequately implement adopted evidence-based interventions, limiting their effectiveness in transforming student outcomes. We propose that to transform student outcomes through evidence-based practice, conceptualization of mediators' intervention implementation must move beyond quantification of discrete intervention steps implemented. Intervention implementation requires behavior change and thus can be conceptualized as an adult behavior change process. The purpose of this article is to illustrate how adult behavior change theory may inform how intervention implementation is conceptualized, facilitated, and supported. An empirically supported theory of adult behavior change from health psychology, the Health Action Process Approach, and how it has informed development of PRIME (Planning Realistic Intervention Implementation and Maintenance by Educators), a system of supports to facilitate mediators' implementation of school-based interventions, are introduced. A case study demonstrating the application of PRIME with a public school teacher implementing a behavior support plan is presented. Implications for future research are discussed. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
The Treatment Integrity Planning Protocol (TIPP) provides a structured process for collaboratively creating a treatment integrity assessment within a consultation framework. The authors evaluated the effect of the TIPP on the implementation of an intervention designed to improve the consistency of students' mathematics performance. Treatment integrity was assessed via permanent products and teacher self-report assessments resulting from TIPP completion. Results showed that (a) 3 teachers' treatment integrity declined within the first week of implementation and increased subsequent to completion of the TIPP, (b) teachers accurately reported their level of treatment integrity, and (c) the relationship between treatment integrity and student outcomes was unclear. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Eight hundred school psychologists who were members of the National Association of School Psychologists were mailed a survey designed to assess their roles, types of referrals, consultation practices, and crisis team involvement. Three hundred seventy (49%) completed surveys were analyzed. Assessment was the most common role followed distantly by consultation. Academic problems were the most frequent type of referral with reading being the most common concern. Of the behavioral referrals, externalizing problems were more frequent than internalizing concerns. Behavioral consultation was the most common model used but less than half of respondents follow all of the stages. Only forty percent use an evaluative component in consultation. A majority of the informants had some involvement with their schools' crisis team. Implications of these findings for training, professional development, and future roles are discussed. © 2002 Wiley Periodicals, Inc.
This special series focused on treatment integrity in the child mental health and education field is timely. The articles do a laudable job of reviewing (a) the current status of treatment in-tegrity research and measurement, (b) existing conceptual models of treatment integrity, and (c) the limitations of prior research. The multidi-mensional model described by Schulte, Eaton, and Parker (2009) provides an incisive summary of current conceptual work. Thus, important conceptual ground is covered in the series, and the articles are likely to serve as an important starting point for future work in the area. The empirical studies in the series raise a host of significant issues relevant to the field. First, the work of Sheridan, Swanger-Gagné, Welch, Kwon, and Garbacz (2009) emphasizes the critical importance of establishing psycho-metric properties for integrity measures. Sheri-dan and colleagues are to be commended for focusing so clearly on psychometrics and em-phasizing the importance of making this a cen-tral issue for future research efforts. Their work also highlights two other important issues in measuring treatment integrity: (a) use of multi-ple methods and (b) the cost of measurement. We return to these issues shortly. The article by McKenna, Rosenfield, and Gravois (2009), which describes another strong psychometric study, evaluates the psy-chometric properties of a measure designed to assess adherence to a consultation model; this is a level of abstraction higher than one typi-cally sees in integrity research. Their work is akin to that of Henggeler, Schoenwald, Liao, Letourneau, and Edwards (2002), who have demonstrated the relevance of adherence at the therapist and supervisor levels in their work Preparation of this article was supported in part by grants from the National Institute of Mental Health (K23 MH69421, Southam-Gerow; R01 MH68806, Weisz) the John D. and Catherine T. MacArthur Foundation (Weisz), and the Norlien Foundation (Weisz).
Attaining and demonstrating treatment fidelity is critical in the development and testing of evidence-based interventions. Treatment fidelity refers to the extent to which an intervention was implemented in clinical testing as it was conceptualized and is clearly differentiable from control or standard-of-care interventions. In clinical research, treatment fidelity is typically attained through intensive training and supervision techniques and demonstrated by measuring therapist adherence and competence to the protocol using external raters. However, in occupational and physical therapy outcomes research, treatment fidelity methods have not been used, which, in our view, is a serious gap that impedes novel treatment development and testing in these rehabilitation fields. In this article, we describe the development of methods to train and supervise therapists to attain adequate treatment fidelity in a treatment development project involving a novel occupational and physical therapy-based intervention. We also present a data-driven model for demonstrating therapist adherence and competence in the new treatment and its differentiation from standard of care. In doing so, we provide an approach that rehabilitation researchers can use to address treatment fidelity in occupational and physical therapy-based interventions. We recommend that all treatment researchers in rehabilitation disciplines use these or similar methods as a vital step in the development and testing of evidence-based rehabilitation interventions.
There have been impressive, recent advances in the development of efficacious treatments for child and adolescent behavior problems. However, specific methods for delivering these treatments in a way that amplifies their efficacy have not been well articulated. Although many factors may be involved, attendance and adherence to treatment are arguably the most basic necessities for effective treatment delivery. We provide a conceptual and empirical review of past research on attendance and adherence to child and adolescent therapy, with a special focus on the importance of parents/guardians in managing treatment participation. Our review demonstrates that attendance and adherence are associated with a range of significant methodological, clinical, and financial outcomes. Several pretreatment predictors of attendance and adherence have been identified; however, to date only 12 controlled, clinical trials have evaluated strategies for enhancing attendance and adherence to child therapy. We conclude with an agenda for advancing research on the prediction and enhancement of attendance and adherence to child therapy as a means of improving the efficiency and effectiveness of child treatments.
The validity of selecting treatment contingencies on the basis of the results obtained through functional analysis is well documented. However, a number of second-generation questions have emerged: For example, what are the parameters required to achieve desired treatment outcomes? More specifically, what is the degree of treatment integrity needed for the successful reduction of problem behavior? This study had two purposes: to describe the relationship between treatment integrity levels and treatment effectiveness and to highlight the importance of reporting the treatment integrity in outcome-based research. The results indicate that a high level of treatment integrity is required for treatment success. Furthermore, the authors found that very low levels of integrity may be required for behavioral reduction procedures (i.e., extinction) if high levels of reinforcement are provided.
This study examined the extent to which treatment integrity of 4 special education teachers was affected by goal setting, performance feedback regarding student or teacher performance, and a meeting cancellation contingency. Teachers were trained to implement function-based treatment packages to address student problem behavior. In one condition, teachers set a goal for student behavior and received daily written feedback about student performance. In a second condition, teachers received daily written feedback about student performance as well as their own accuracy in implementing the intervention and would be able to avoid meeting with a consultant to practice missed steps by implementing the intervention with 100% integrity. This latter package increased treatment integrity the most above baseline levels. Higher levels of treatment integrity were significantly correlated with lower levels of student problem behavior for 3 of the 4 teacher-student dyads. Three of the 4 teachers also rated both feedback procedures as highly acceptable. Implications for increasing and maintaining treatment integrity by teachers via a consultation model are discussed.
A random sample of 1,000 Nationally Certified School Psychologists were mailed a 17‐item survey about treatment integrity. A total of 132 responded and answered questions about the importance of treatment integrity data for tiered interventions and the frequency and methods used to measure treatment integrity in one‐to‐one and problem‐solving team consultation. Ninety‐eight percent agreed that treatment integrity data were critical for tiered interventions. However, only 7% in one‐to‐one consultation and 0% in team consultation reported it was “always” collected. Respondents identified barriers of time, workload, system factors, and staff knowledge as explanations for why there was a gap between belief and practice. The participant responses were compared to those from a 2008 survey and found to be almost unchanged. Issues associated with decision‐making when treatment integrity data are lacking and recommendations for improving the assessment and reporting of treatment integrity are discussed.
This study examined whether outcomes in early intensive behavioral intervention (EIBI) during a university-led multisite project could be replicated by the same community agency independently of the project. Participants, age 18 to 75 months at onset of intervention, were 48 children with autism spectrum disorder (ASD) enrolled in 35 hr per week of publicly funded, community-based EIBI for 3 years and 46 children who were matched on baseline characteristics and received early childhood services as usual (SAU) through local early childhood special education providers. Linear mixed models indicated that EIBI participants improved significantly more than SAU participants on standardized tests of IQ, nonverbal IQ, adaptive behavior, and academic achievement, administered by independent evaluators. Although limited by the use of a matched comparison group rather than random assignment, the study provides evidence for the sustainability of effective EIBI in community settings for children with ASD who start intervention at varying ages throughout early childhood.
To evaluate students’ responsiveness to an intervention, both student outcome and implementer treatment integrity data are needed. Teachers are often asked to self-report treatment integrity data. However, when self-report responses are compared with those from a direct observer, it is apparent that teachers commonly overestimate the extent to which an intervention was implemented as planned. As such, more research related to teacher self-report to assess treatment integrity is needed. The objective of this preliminary single-case multiple-baseline design study was to improve interrater agreement between observers’ and teachers’ self-report ratings of treatment integrity by providing teachers with comprehensive, direct training (including an intervention description, modeling, practice, and feedback). Results indicate that after this training, agreement between observers’ and teachers’ ratings of treatment integrity improved.
Individuals with autism spectrum disorder (ASD) often have difficulty generalizing newly acquired communication skills to different contexts. In this study, a multiple baseline across settings (clinic, school, and home) design was used to determine whether an 8-year-old boy with ASD could learn to approach communication partners to request preferred items using an iPad®-based speech-generating device (SGD). Results indicated that the participant learned to approach communication partners to request across all settings. Acquisition of this skill was fastest at school and slowest in the clinic. The participant’s learning generalized to a novel room within the same setting, and he was also able to retrieve the SGD prior to approaching the communication partner without additional teaching. Results suggest that individuals with ASD may benefit from being systematically taught to use an SGD across different settings and communication partners.
Research has demonstrated the efficacy of a behavioral parent training program for skills acquisition; however, few studies have examined the extent to which those skills generalize to the home or the effect that they have on child behavior. A multiple baseline across participants design was used to assess (a) caregiver accuracy with implementation of three parenting skills, and (b) the effects of the parenting skills on child behavior. Results demonstrated that three caregiver participants successfully generalized parenting skills taught during behavioral skills training (BST) to naturally occurring routines, and the behavior of each caregiver's child improved following BST.
For multi‐tiered systems of support, such as Response‐to‐Intervention and Positive Behavior Interventions and Supports, to effectively impact student outcomes, interventions delivered across the tiers must be implemented as planned (i.e., with adequate treatment integrity). However, research suggests that most school personnel struggle to deliver interventions with treatment integrity, which negatively impacts the potential effectiveness of these interventions. Numerous strategies to support treatment integrity have been developed, but no guidance has been provided regarding how to efficiently and effectively use them. The purpose of this study was to conduct a pilot evaluation of these strategies delivered through a Multi‐Tiered Implementation Supports framework; that is, proactive, feasible treatment integrity strategies were initially delivered to all implementers and based on their responsiveness, increasingly intensive implementation supports were provided as needed. Results suggest that (a) all teachers responded to these supports, but response magnitude was different across teachers and supports; (b) higher levels of treatment integrity generally were associated with fewer disruptive behaviors; and (c) the duration of these support strategies increased across tiers. Future directions for research and implications for the feasible provision of implementation support in schools are described.
Teachers typically enter the field with limited training in classroom management, and research demonstrates that training alone does not result in improved practice. Typically, researchers have relied on time-intensive training packages that include performance feedback to improve teachers’ use of classroom management practices; however, initial evidence suggests that self-management may be an effective and efficient alternative. In this study, the authors directly compared the effects of three different self-monitoring conditions (tally, count, and rate) and no self-monitoring on five middle school teachers’ rate of specific praise using an alternating treatments design. The authors also included baseline and follow-up phases to descriptively explore the effects of self-monitoring across time. Results indicate that noting each instance of specific praise by either tallying or using a counter resulted in optimal performance, and teachers preferred using a counter. Additional study results, limitations, and implications are discussed.
Fidelity of treatment in outcome research refers to confirmation that the manipulation of the independent variable occurred as planned. Verification of fidelity is needed to ensure that fair, powerful, and valid comparisons of replicable treatments can be made. The central purpose of this article is the evaluation of outcome studies from 1980 to 1988 to determine the extent to which investigators are attending adequately to the issue of treatment fidelity. The evaluation covered 359 treatment outcome studies from major journals in four domains: clinical psychology, behavior therapy, psychiatry, and marital and family therapy. The studies were evaluated with respect to: (a) the training of treatment implementors, (b) the procedures used to promote fidelity, (c) the aspects of treatment verified, (d) the methods for assessing fidelity, and (e) the utilization of fidelity assessment in the interpretation of results. Although there were significant increases over the decade in the percentage of studies that checked adherence to treatment and used supervision to promote fidelity, the majority (55%) of the studies essentially ignored the issue of treatment fidelity. Furthermore, only one out of eight studies in the most recent period (1986–1988) combined the use of treatment manuals, supervision of treatment agents, and checking of adherence to protocol. Finally, attention given to the aforementioned aspects of fidelity did not differ significantly across journal domains. The overall implication of the review of existing practices is that investigators, reviewers, and journal editors need to give even greater consideration to the issues associated with fidelity. Specific recommendations for improving the promotion and verification of treatment fidelity in outcome studies were offered.
An international trend in school psychology services is a shift from an emphasis on assessment-based activities to a paradigm of consultation problem-solving and behavioural intervention. As the profession experiences an expansion of roles and functions, school psychologists should have an understanding of a critical aspect of behaviour change: treatment integrity (Gresham, 1989). Treatment integrity (or fidelity) refers to the extent to which an intervention is implemented as intended (or planned). This article describes the construct of treatment integrity as it relates to the implementation of consultation-derived behaviour intervention plans. Practical approaches for assessing and monitoring the integrity of treatments are presented. A treatment-monitoring interview (TMI) is proposed as an initial step towards developing a technology to assess and facilitate treatment integrity in school-based behavioural consultation practice. Limitations, implications for practice and future research directions are discussed.
Describes the concept of treatment integrity (TI) as it relates to the design and implementation of consultation plans and prereferral interventions with children in classrooms. Factors related to TI in school settings include complexity of treatments, time required to implement, materials/resources, number of treatment agents, perceived and actual effectiveness, and motivation of treatment agents. Technical issues (specification of treatment components, deviations from treatment protocols and amount of behavior change, and psychometric issues in assessing TI) involved in the measurement of TI are discussed. It is suggested that the assessment of TI is identical to the systematic observation of behavior in applied settings. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Treatment integrity (the degree to which a treatment is implemented as planned) represents a key concept in school-based intervention and is considered to be a link between use and effectiveness of interventions. 181 experimental studies published between 1980 and 1990 in 7 journals known for behaviorally based interventions were reviewed. Of primary interest was whether or not integrity was assessed, the degree of treatment integrity, operational definitions of treatments, and effect sizes produced by interventions. Only 14.4% (26 studies) systematically measured and reported integrity data; only 34% (65 studies) operationally defined treatments. Moderate positive correlations were found between degree of treatment integrity and level of treatment outcome. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
A survey of individuals holding the Nationally Certified School Psychologist (NCSP) credential was conducted via the Internet to gather information regarding their measurement of treatment integrity in school-based interventions and their beliefs about its importance. A sample of 806 self-selected professionals holding the NCSP credential provided data about the extent to which they measure treatment integrity and the methods they used to measure it when developing interventions via one-to-one and group/team consultation. Results showed that 97.6% agreed that measurement of treatment integrity was a key factor to consider and to include when evaluating interventions and when using intervention data for special education eligibility decisions. Few, however, reported regularly documenting it in one-to-one (only 11.3%) or group/team consultation (only 1.9%). Recommendations for how school psychologists and other school staff can increase their measurement of treatment integrity in school-based interventions are offered. © 2008 Wiley Periodicals, Inc.
To discuss methods of preservation of treatment fidelity in health behavior change trials conducted in public health contexts.
The treatment fidelity framework provided by the National Institutes of Health's Behavioral Change Consortium includes five domains of treatment fidelity (Study Design, Training, Delivery, Receipt, and Enactment). A measure of treatment fidelity was previously developed and validated using these categories.
Strategies for assessment, monitoring, and enhancing treatment fidelity within each of the five treatment fidelity domains are discussed. The previously created measure of treatment fidelity is updated to include additional items on selecting providers, additional confounders, theory testing, and multicultural considerations.
Implementation of a treatment fidelity plan may require extra staff time and costs. However, the economic and scientific costs of lack of attention to treatment fidelity are far greater than the costs of treatment fidelity implementation. Maintaining high levels of treatment fidelity with flexible adaptation according to setting, provider, and patient is the goal for public health trials.
The extent to which parents of children with intellectual or developmental disabilities are adherent to prescribed treatments has not been investigated. In this treatment adherence study, parents (n = 220) of children with autism spectrum disorders were surveyed regarding implementation of recommended treatments to manage problem behavior of their children living at home. Overall adherence to medical treatment recommendations was significantly greater than adherence to behavioral treatment recommendations (p < .002). Of the behavioral treatment recommendations, parents reported greater adherence to reinforcement (81.7%) than punishment (68.9%). Child diagnosis (p < .002) and the diagnosis x marital status interaction (p < .05) were significantly associated with reported adherence to behavioral and medical treatment, respectively. Results are discussed in light of the need to address adherence enhancement and measurement methods.
We examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. Only 39 of 162 outcome studies featured specified procedures for the documentation of fidelity. Of these, only 13 considered variations in integrity in analyzing the effects of the program. Lowered adherence to protocol was often associated with poorer outcome. There was mixed evidence of dosage effects. The omission of integrity data, particularly measures of adherence, may compromise the internal validity of outcome studies in the prevention literature. We do not view procedures for integrity verification as inconsistent with the adaptation of interventions to the needs of receiving communities.
The challenges of specifying a complex and individualized treatment model and measuring fidelity thereto are described, using multisystemic therapy (MST) as an example. Relations between therapist adherence to MST principles and instrumental and ultimate outcome variables are examined, as are relations between clinical supervision and therapist adherence. The findings provide modest support for the associations between MST adherence measures and instrumental and ultimate outcomes. Results also show that adherence can be altered when clinical supervision and adherence monitoring procedures are fortified. The modest associations between adherence measures and youth outcomes argue for further refinement and validation of the MST adherence measure, especially in light of the well-established effectiveness of MST with challenging clinical populations and the increasing dissemination of MST programs.
Increasing teachers' treatment integrity when implementing a class-wide intervention through performance feedback provided by a school-based consultant: A case study
- L M H Sanetti
- S M Chafouleas
- L M Fallon
- R Jaffery
Sanetti, L. M. H., Chafouleas, S. M., Fallon, L. M., & Jaffery, R.
(2014). Increasing teachers' treatment integrity when implementing a class-wide intervention through performance feedback
provided by a school-based consultant: A case study. Journal of
Educational and Psychological Consultation, 24, 239-260.
Barriers to implementing treatment integrity procedures in school psychology research
- L M H Sanetti
- F D Reed
Sanetti, L. M. H., & Reed, F. D. (2012). Barriers to implementing
treatment integrity procedures in school psychology research.
Assessment for Effective Intervention, 37, 195-202.
Professional and ethical compliance code for behavior analysts
Behavior Analyst Certification Board. (2019). Professional and
ethical compliance code for behavior analysts. Retrieved
from http://bacb.com/ethics-code
National standards project findings and conclusions
National Autism Center. (2009). National standards project findings and conclusions. Randolph, MA: Author. Retrieved from
https://www.nationalautismcenter.org/national-standardsproject/