No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Assessing the Implementation of the Good Behavior Game: Comparing Estimates of Adherence, Quality, and Exposure
Abstract
Treatment fidelity assessment is critical to evaluating the extent to which interventions, such as the Good Behavior Game, are implemented as intended and impact student outcomes. The assessment methods by which treatment fidelity data are collected vary, with direct observation being the most popular and widely recommended. The current study presents an exploratory analysis of the differences between fidelity estimates collected through direct observation versus permanent product across three dimensions of treatment fidelity: adherence, quality, and exposure. Findings indicate that direct observation may allow for a more complete estimate of adherence when compared with permanent product, if implementation occurs within the observation window. Quality was only captured via direct observation. Direct observation may not produce adequate estimates of exposure depending on implementation timing. Implications for treatment fidelity assessment are reviewed.
... Treatment fidelity data derived from these measurement methods have been found to be positively correlated with student outcome data, suggesting validity of this approach (Sanetti & Collier-Meek, 2014). However, initial research suggests that how the intervention is articulated (Sanetti & Collier-Meek, 2014), which dimensions are assessed (e.g., adherence, quality; Sanetti & Fallon, 2011), which measurement method is selected (Gresham et al., 2017), how intervention steps are rated (Collier-Meek, Fallon, & DeFouw, 2018), and the sampling of the intervention (Collier-Meek, Sanetti, Fallon, & Chafouleas, 2019) can impact treatment fidelity estimates. Thus, although researchers most often utilize this approach, there is substantial variability in how student interventions are conceptualized and rated. ...
... However, these data are rarely collected in practice (Cochrane et al., 2019) and only sometimes in research (Barnett et al., 2014). Although post-observation checklists are most often used in research and have been correlated with expected improvements in client outcomes (e.g., higher treatment fidelity associated with greater gains in academic engagement), studies also reveal that treatment fidelity estimates vary depending on how the assessment form is structured (Collier-Meek, Fallon, & DeFouw, 2018;Gresham et al., 2017;Sanetti & Collier-Meek, 2014). Perhaps a more straightforward approach may involve event recording, which has been used to report the frequency or rate of specified teacher behaviors (e.g., praise; Collier-Meek, Sanetti, & Fallon, 2017;, or time sampling, which has been used to document teachers positive and negative statements (Rathel et al., 2014). ...
When assessed, treatment fidelity is most often evaluated by checklists of intervention steps after an observation session, though estimates can vary depending on how intervention steps are operationalized and rated. A more straightforward approach may involve the adoption of direct observation methods such as time sampling or event recording, which are regularly used to evaluate student behavior. This study explores the utility of using time sampling and event recording in comparison to the traditional post-observation checklist method. Five paraeducator consultees’ treatment fidelity of behavior support plans for students with autism was assessed by post-observation ratings. In addition, 15-sec whole-interval recording of treatment fidelity across intervention steps and event recording of prompts (an intervention step for all plans) were collected throughout the observation session. To compare treatment fidelity data, we present visual analysis, associated quantitative metrics, and correlations. Implications for future treatment fidelity assessment research and consultation practice are described.
... • Allocate time and resources to support the fidelity of implementation; this includes training, ongoing evaluation of efforts/instruments, and robust longitudinal data collection. • Employ quantitative and qualitative data collection methods and data sources to gain a holistic understanding of fidelity [2]. • Consider more proximal measures in assessing student outcomes, including measures of student learning within semester units such as course grades, DWF (dropwithdraw-fail) rate, midterm grades, and upper division grades. ...
... That is, the supports provided are commensurate with the resources available and are scaled to the staff's level of need (Fallon, Collier-Meek, Kurtz, & DeFouw, 2018). It may be worthwhile to attempt a more resource-efficient coaching support first (e.g., providing regular reminders to implement) (Collier-Meek, Fallon, & DeFouw, 2020). As such, more intensive supports might be delivered only upon review of implementation data indicating more comprehensive support is needed (Fallon et al., 2018). ...
... Historically, observation has been considered the superior assessment method because it is the most direct (Gresham, 1989). Recent comparisons of treatment fidelity data concurrently resulting from different assessment methods may provide initial research-informed recommendations (Collier-Meek, Fallon, & DeFouw, 2018;Gresham, Dart, & Collins, 2017;Sanetti & Collier-Meek, 2014). In a comparison of treatment fidelity assessment methods of the Good Behavior Game using Generalizability Theory, Gresham and colleagues (2017) found that direct observation data produced the most reliable data with the fewest data points, compared with permanent product and self-report data. ...
Treatment fidelity data are critical to evaluate intervention effectiveness, yet there are only general guidelines regarding treatment fidelity measurement. Initial investigations have found treatment fidelity data collected via direct observation to be more reliable than data collected via permanent product or self-report. However, the comparison of assessment methods is complicated by the intervention steps accounted for, observation timing, and intervention sessions accounted for, which may impact treatment fidelity estimates. In this study, we compared direct observation and permanent product data to evaluate these varied assessment and data collection decisions on treatment fidelity data estimates in three classrooms engaged in a group contingency intervention. Findings revealed that treatment fidelity estimates, in addition to being different across assessment method, are, in fact, different depending on the intervention steps assessed, intervention sessions accounted for, and observation timing. Implications for treatment fidelity assessment research, reporting in intervention research broadly, and implementation assessment in practice are described.
Fidelity of implementation is a critical but often neglected component of any new system, practice, or intervention in educational research and practice. Fidelity is a multidimensional construct focused on providing evidence of adherence, quality, dosage, differentiation, and responsiveness following implementation. Unfortunately, fidelity has not always been prioritized, although evidence suggests that is changing, at least in published research. Further, although there are myriad methods for measuring fidelity, psychometric evaluations of fidelity tools have been limited, except in the SWPBIS literature. Calls for a science of fidelity have been made (Gresham, 2017) and are beginning to be answered. Overall, there appears to be more research focused exclusively on fidelity, including measurement approaches, psychometric evaluations, and relation to outcomes. As this research expands, we hope that the broad use and integration of fidelity in practice follows. We believe that the days of neglecting fidelity are behind us in education and see fidelity playing a central role in education moving forward. Through reliable and valid measurement of fidelity, scalable evidence-based practices can be developed and proliferated, positively impacting students’ academic and behavioral outcomes.
Assessing the degree to which interventions are implemented in school settings is critical to making decisions about student outcomes. School psychologists may not be available to regularly conduct observations of intervention implementation, however, their data may be used alongside other methods for multi‐informant assessment. Teacher self‐report is a commonly used and feasible assessment method. Students have been trained to implement interventions with their peers in instances where traditional adult interventionists were unavailable. This exploratory study investigated the accuracy with which classroom teachers and middle and high school students assessed implementation of the Good Behavior Game and the impact of performance feedback on their accuracy. Results indicated that most students and teachers were able to provide accurate assessments of treatment integrity compared to researcher direct observation; however, some required performance feedback to do so. These findings suggest that multi‐informant assessment may be a feasible and accurate way for school psychologists to collect formative treatment‐integrity data in the classroom. Limitations and future directions are discussed.
Collecting treatment integrity data is critical for (a) strengthening internal validity within a research study, (b) determining the impact of an intervention on student outcomes, and (c) assessing the need for implementation supports. Although researchers have noted the increased inclusion of treatment integrity data in published articles, there has been limited attention to how treatment integrity is assessed. The purpose of this study was to review how treatment integrity is typically assessed in single-case design performance feedback articles. Doctoral students in school psychology coded 58 performance feedback studies to determine how treatment integrity data are assessed in this literature, and common intervention characteristics linked to treatment integrity assessment. Findings indicate that, although data were collected most frequently via direct observation by a researcher targeting intervention adherence, intervention characteristics varied widely across studies. Implications of results are discussed.
Although high levels of intervention implementation are more likely to lead to improved student outcomes, educators struggle to maintain high implementation levels over time. School psychologists might provide research-supported, consequence-oriented supports (e.g., performance feedback) to promote educators' implementation, yet these are reactive and potentially time intensive. This study evaluated whether a proactive, antecedent-oriented support (i.e., daily, preprogrammed e-mailed prompts) could effectively promote educators' implementation. Findings indicate that for 3 of 4 teachers who participated in this multiple baseline single case design study, implementation of the class-wide behavior intervention improved upon receiving e-mailed prompts. In addition, increases in praise, decreases in corrective statements, and corresponding improvements in student outcomes were noted. This initial study suggests that prompts may be a feasible and effective Tier 1 implementation support that can be incorporated by school psychologists to support educators responsible for delivering interventions in the classroom. Additional implications for future research and school-based practice are discussed.
In consultation, school psychologists may offer educators direct training to support the implementation of classroom interventions aimed to improve student outcomes. The purpose of this study was to conduct a systematic literature review of single-case design research studies during which educators received direct training to implement a classroom intervention, specifically instructions, modeling, practice, and feedback. Two doctoral students in school psychology screened 228 articles and evaluated 33 studies to determine if direct training is effective and an evidence-based practice per single-case design standards proposed by the What Works Clearinghouse. Results of the review indicate that there is support for the practice to be deemed evidence-based and associated with better intervention implementation than before its application. Implications include direct training being considered for intensive, complex interventions to promote educator success with implementation. (PsycINFO Database Record
The Good Behavior Game (GBG) is a classroom management strategy that uses an interdependent group-oriented contingency to promote prosocial behavior and decrease problem behavior. This meta-analysis synthesized single-case research (SCR) on the GBG across 21 studies, representing 1,580 students in pre-kindergarten through Grade 12. The TauU effect size across 137 phase contrasts was .82 with a confidence interval 95% CI = [0.78, 0.87], indicating a substantial reduction in problem behavior and an increase in prosocial behavior for participating students. Five potential moderators were examined: emotional and behavioral disorder (EBD) risk status, reinforcement frequency, target behaviors, GBG format, and grade level. Findings suggest that the GBG is most effective in reducing disruptive and off-task behaviors, and that students with or at risk for EBD benefit most from the intervention. Implications for research and practice are discussed.
Evidence-based interventions are being disseminated broadly in schools across the USA, but the implementation levels achieved in community settings vary considerably. The current study examined the extent to which teacher and school factors were associated with implementation dosage and quality of the PAX Good Behavior Game (PAX GBG), a universal classroom-based preventive intervention designed to improve student social-emotional competence and behavior. Specifically, dosage (i.e., number of games and duration of games) across the school year and quality (i.e., how well the game is delivered) of PAX GBG implementation across four time points in a school year were examined. Hierarchical linear modeling was used to examine the association between teacher-level factors (e.g., demographics, self-reports of personal resources, attitudes toward the intervention, and workplace perceptions) and longitudinal implementation data. We also accounted for school-level factors, including demographic characteristics of the students and ratings of the schools' organizational health. Findings indicated that only a few teacher-level factors were significantly related to variation in implementation. Teacher perceptions (e.g., fit with teaching style, emotional exhaustion) were generally related to dosage, whereas demographic factors (e.g., teachers' age) were related to quality. These findings highlight the importance of school contextual and proximal teacher factors on the implementation of classroom-based programs.
Implementation, or treatment fidelity, is the link between evidence-based interventions and changes in student outcomes. Although higher levels of treatment fidelity are usually associated with greater student improvements, many educators struggle to implement intervention and practices consistently. To be responsive to this issue, school problem-solving teams that work to support students in multitiered systems should consider evaluating and promoting treatment fidelity. Through collaboration, teams can build an under- standing of treatment fidelity in their schools and incorporate treatment fidelity assessment, evaluation, and promotion into their practice.
Challenging behavior at school remains a concern for teachers and administrators. Thus classroom management practices to prevent challenging behavior are sorely needed. The Good Behavior Game (GBG) has been found to be useful to positively change student behavior. However, previous reviews of the GBG have not quantified effects, have focused solely on school and classroom behaviors, and have not examined study features that facilitate greater outcomes. Twenty-two peer-reviewed journal articles were reviewed. Study data were analyzed using effect sizes, percent of nonoverlapping data, percent of all nonoverlapping data, and hierarchical linear modeling to determine intervention effectiveness as well as study features that facilitated greater outcomes. Findings suggested that (a) moderate to large effects were found on challenging behaviors and these effects were immediate; (b) the GBG was most commonly used for disruptive behavior, off-task behavior, aggression, talking out, and out-of-seat behaviors; (c) the GBG has been implemented primarily in general education elementary school settings; and (d) correct application of reward procedures are important for intervention effectiveness. Study limitations, implications for practice, and areas for future research are presented.
Although it is widely accepted that procedural fidelity data are important for making valid decisions about intervention effectiveness, there is little empirical guidance for researchers and practitioners regarding how to assess procedural fidelity. A first step in moving procedural fidelity assessment research forward is to develop a comprehensive understanding of the relation between currently recommended assessment methods, systematic direct observation (SDO) and permanent product review (PPR). To this end, we first discuss methodological issues related to procedural fidelity assessment via SDO and PPR, and how they are evidenced in the behavioral intervention outcome literature. Second, we present a single-case design study in which teachers’ implementation of behavior support plans was assessed via SDO and PPR. Third, we analyze the SDO and PPR data with regard to (a) relation between the methods; (b) methodological issues including definitions of procedural fidelity and intervention steps, representativeness of intervention steps, and collection of an adequate sample; and (c) association with student outcomes. Finally, we discuss implications for research and practice.
This study examined teachers' implementation of treatment plans fol- lowing consultation. Interventions were implemented for 45 elementary school students referred for consultation and intervention due to academic concerns, chal- lenging behavior, or a combination of the two. The consultation follow-up proce- dures examined were brief weekly interviews, weekly interviews combined with an emphasis on the commitment to implement the treatment, and performance feedback. Performance feedback was associated with superior treatment imple- mentation and child behavioral outcomes when compared to the two other condi- tions. Treatment implementation did not differ for the weekly follow-up meeting and the commitment emphasis conditions at a statistically significant level. Teacher ratings of consultants and treatment acceptability were similar across conditions. A moderate statistically significant correlation between treatment integrity and child behavioral outcome was obtained. The correlation between treatment ac- ceptability and implementation was quite small and was not statistically signifi- cant. The implications of these findings for consultation and intervention are dis- cussed.
A majority of evidence-based interventions in schools are delivered through consultation models and are implemented by a mediator, such as a teacher. Research indicates that mediators do not always adequately implement adopted evidence-based interventions, limiting their effectiveness in transforming student outcomes. We propose that to transform student outcomes through evidence-based practice, conceptualization of mediators' intervention implementation must move beyond quantification of discrete intervention steps implemented. Intervention implementation requires behavior change and thus can be conceptualized as an adult behavior change process. The purpose of this article is to illustrate how adult behavior change theory may inform how intervention implementation is conceptualized, facilitated, and supported. An empirically supported theory of adult behavior change from health psychology, the Health Action Process Approach, and how it has informed development of PRIME (Planning Realistic Intervention Implementation and Maintenance by Educators), a system of supports to facilitate mediators' implementation of school-based interventions, are introduced. A case study demonstrating the application of PRIME with a public school teacher implementing a behavior support plan is presented. Implications for future research are discussed. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
The Treatment Integrity Planning Protocol (TIPP) provides a structured process for collaboratively creating a treatment integrity assessment within a consultation framework. The authors evaluated the effect of the TIPP on the implementation of an intervention designed to improve the consistency of students' mathematics performance. Treatment integrity was assessed via permanent products and teacher self-report assessments resulting from TIPP completion. Results showed that (a) 3 teachers' treatment integrity declined within the first week of implementation and increased subsequent to completion of the TIPP, (b) teachers accurately reported their level of treatment integrity, and (c) the relationship between treatment integrity and student outcomes was unclear. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
One of the most important aspects of treatment outcome research is establishing treatment integrity. Integrity of the treatment refers to the degree to which treatment is implemented as intended. Research examining the relationship between treatment integrity and therapeutic change has produced conflicting results. However, assessment, design strategies, and the possible confound of integrity with other variables may explain the inconsistency in findings. This paper elaborates the limitations of existing strategies for evaluating the relationship between treatment integrity and outcome. Recommendations for future research include controlling possible confounding variables, experimentally manipulating treatment integrity, and using novel assessment and evaluation strategies.
Disruptive behavior can waste a great deal of teaching time in the classroom, leading to feelings of frustration in teachers and an increase in academic failure among pupils. Prior research indicates that intervening in these kinds of behaviors improves the classroom atmosphere and facilitates the learning process. With this in mind, the aims of this article are to (a) reduce the incidence of disruptive behaviors such as standing up without the teacher's permission, shouting, fighting, and interrupting the teacher or a fellow classmate, using a combination of the Good Behavior Game (GBG) and Say-Do-Report (S-D-R) Correspondence training; (b) achieve long-term maintenance of results following the gradual withdrawal of the intervention; and (c) introduce the GBG in a different educational context than those discussed so far in the empirical literature. The intervention took place with the 15 children of a standard primary classroom (Cycle 1) at a state-run school in Andalusia (Spain). Using a multiple baseline design across situations, the GBG and Say not-not Do-Request not (Sn-nD-Rn) Correspondence training were introduced. A significant reduction in the incidence of disruptive behavior was observed, contingent on the respective application of the intervention in each baseline. The combined application of the GBG and the S-D-R Correspondence proved to be an effective way of decreasing disruptive behaviors (shouting, interrupting, etc.) in the classroom, and the results were maintained for 1 year following the gradual withdrawal of the treatment. © 2010 Wiley Periodicals, Inc.
We investigated the effects of systematic changes in levels of treatment integrity by altering errors of commission during error-correction procedures as part of discrete-trial training. We taught 3 students with autism receptive nonsense shapes under 3 treatment integrity conditions (0%, 50%, or 100% errors of commission). Participants exhibited higher levels of performance during perfect implementation (0% errors). For 2 of the 3 participants, performance was low and showed no differentiation in the remaining conditions. Findings suggest that 50% commission errors may be as detrimental as 100% commission errors on teaching outcomes.
Students' inattention is predictive of reading problems and of non-response to effective reading intervention. In this randomized study, 58 first-grade classrooms located in 30 schools were assigned to a control condition or to one of two intervention conditions. In these last two conditions, peer-tutoring activities were conducted to improve classroom reading instruction. In one of the intervention conditions, the Good Behavior Game was also implemented to maximize students' attention during reading lessons. Both interventions were effective: peer-tutoring activities helped students improve their reading skills and attention was generally higher when the Good Behavior Game was implemented. Contrary to expectations however, students identified as inattentive at pretest did not become better readers when the two interventions were implemented.
This paper is on the influences of the classroom context on the course and malleability of aggressive behavior from entrance into first grade through the transition into middle school. Nineteen public elementary schools participated in developmental epidemiologically based preventive trials in first and second grades, one of which was directed at reducing aggressive, disruptive behavior. At the start of first grade, schools and teachers were randomly assigned to intervention or control conditions. Children within each school were assigned sequentially to classrooms from alphabetized lists, followed by checking to insure balanced assignment based on kindergarten behavior. Despite these procedures, by the end of first quarter, classrooms within schools differed markedly in levels of aggressive behavior. Children were followed through sixth grade, where their aggressive behavior was rated by middle school teachers. Strong interactive effects were found on the risk of being highly aggressive in middle school between the level of aggressive behavior in the first grade classrooms and each boy's own level of aggressive, disruptive behavior in first grade. The more aggressive first grade boys who were in higher aggressive first grade classrooms were at markedly increased risk, compared both to the median first grade boys, and compared to aggressive males in lower aggressive first grade classrooms. Boys were already behaving more aggressively than girls in first grade; and no similar classroom aggression effect was found among girls, although girls' own aggressive behavior did place them at increased risk. The preventive intervention effect, already reported elsewhere to reduce aggressive behavior among the more aggressive males, appeared to do so by reducing high levels of classroom aggression. First grade males' own poverty level was associated with higher risk of being more aggressive, disruptive in first grade, and thereby increased their vulnerability to classroom level of aggression. Both boys and girls in schools in poor communities were at increased risk of being highly aggressive in middle school regardless of their levels of aggressive behavior in first grade. These results are discussed in terms of life course/social field theory as applied to the role of contextual influences on the development and etiology of severe aggressive behavior.
We assessed the immediate effects of two universal, first-grade preventive interventions on the proximal targets of poor achievement, concentration problems, aggression, and shy behaviors, known early risk behaviors for later substance use/abuse, affective disorder, and conduct disorder. The classroom-centered (CC) intervention was designed to reduce these early risk behaviors by enhancing teachers' behavior management and instructional skills, whereas the family-school partnership (FSP) intervention was aimed at improving parent-teacher communication and parental teaching and child behavior management strategies. Over the course of first and second grades, the CC intervention yielded the greatest degree of impact on its proximal targets, whereas the FSP's impact was somewhat less. The effects were influenced by gender and by preintervention levels of risk. Analyses of implementation measures demonstrated that greater fidelity to the intervention protocols was associated with greater impact on behavior ratings and on achievement scores, thus providing some evidence of specificity in the effect of the interventions.
A "behavioral vaccine" provides an inoculation against morbidity or mortality, impacting physical, mental, or behavior disorders. An historical example of a behavioral vaccine is antiseptic hand washing to reduce childbed fever. In current society, issues with high levels of morbidity, such as substance abuse, delinquency, youth violence, and other behavioral disorders (multiproblems), cry out for a low-cost, widespread strategy as simple as antiseptic hand washing. Congruent research findings from longitudinal studies, twin studies, and other investigations suggest that a possibility might exist for a behavioral vaccine for multiproblem behavior. A simple behavioral strategy called the Good Behavior Game (GBG), which reinforces inhibition in a group context of elementary school, has substantial previous research to consider its use as a behavioral vaccine. The GBG is not a curriculum but rather a simple behavioral procedure from applied behavior analysis. Approximately 20 independent replications of the GBG across different grade levels, different types of students, different settings, and some with long-term follow-up show strong, consistent impact on impulsive, disruptive behaviors of children and teens as well as reductions in substance use or serious antisocial behaviors. The GBG, named as a "best practice" for the prevention of substance abuse or violent behavior by a number of federal agencies, is unique because it is the only practice implemented by individual teachers that is documented to have long-term effects. Presently, the GBG is only used in a small number of settings. However, near universal use of the GBG, in major political jurisdictions during the elementary years, could substantially reduce the incidence of substance use, antisocial behavior, and other adverse developmental or social consequences at a very modest cost, with very positive cost-effectiveness ratios.
The Good Behavior Game (GBG), a type of interdependent group-oriented contingency management procedure, was first introduced in 1969 and has been used with overwhelming success in classrooms and other settings. Since its inception, the "game" has utilized team competition and peer influence combined with reinforcement procedures. It has been found to be popular, easy-to-use, time-efficient, and widely applicable and versatile. This review describes the game and its numerous variations and adaptations, as well as empirical findings specific to the variety of target behaviors and participants to which it has been applied. In addition, different types of reinforcers used, information on consumer acceptance, and issues related to implementation are considered.
We assessed the immediate effects of two universal, first‐grade preventive interventions on the proximal targets of poor achievement, concentration problems, aggression, and shy behaviors, known early risk behaviors for later substance use/abuse, affective disorder, and conduct disorder. The classroom‐centered (CC) intervention was designed to reduce these early risk behaviors by enhancing teachers' behavior management and instructional skills, whereas the family‐school partnership (FSP) intervention was aimed at improving parent‐teacher communication and parental teaching and child behavior management strategies. Over the course of first and second grades, the CC intervention yielded the greatest degree of impact on its proximal targets, whereas the FSP's impact was somewhat less. The effects were influenced by gender and by preintervention levels of risk. Analyses of implementation measures demonstrated that greater fidelity to the intervention protocols was associated with greater impact on behavior ratings and on achievement scores, thus providing some evidence of specificity in the effect of the interventions.
Numerous evidence-based classroom management strategies to prevent and respond to problem behavior have been identified, but research consistently indicates teachers rarely implement them with sufficient implementation fidelity. The purpose of this study was to evaluate the effectiveness of implementation planning, a strategy involving logistical intervention implementation planning and identification of implementation barriers, and participant modeling, a strategy involving didactic and in vivo intervention training, on teachers’ implementation of an evidence-based classroom management plan. A randomized multiple treatment embedded within a multiple baseline design across participants was used to assess (a) teachers’ adherence to the classroom management plans and quality of implementation and (b) student disruptive behavior in the classroom immediately and at follow-up. Results indicated that teachers’ adherence and quality increased with both implementation planning and participant modeling, but these improvements were not fully maintained at 1- and 2-month follow-up. A similar pattern in student disruptive behavior was also observed. These findings highlight the need for ongoing implementation support for behavioral interventions in schools. Implications for future research and practice are discussed.
The concept of treatment integrity is an essential component to data-based decision making within a response-to-intervention model. Although treatment integrity is a topic receiving increased attention in the school-based intervention literature, relatively few studies have been conducted regarding the technical adequacy of treatment integrity assessment methods. In light of recent research utilizing generalizability theory (G theory) to assess the dependability of behavioral measurement in schools, the current study used G theory to examine the dependability of direct observation, permanent products, and self-report as measures of treatment integrity when six teachers implemented the Good Behavior Game across three study sites. Results indicated that direct observation yielded the most reliable treatment integrity data, followed by permanent products and self-report. Specifically, when assessment of treatment integrity is conducted twice per week, direct observation should provide a dependable estimate (φ = .821) after only four assessments. Implications, limitations, and future directions are discussed.
The present study investigated the effects of an intensive training protocol on levels of paraeducator fidelity to a group contingency intervention used to manage the classroom behavior of students with EBD. A multiple baseline design across classrooms was used to determine whether the training was associated with initial and sustained increases in treatment fidelity. Data were also collected on the effects of paraeducator use of the group contingency program on rates of paraeducator, teacher, and student behavior. Results indicated that the training package was associated with immediate increases in paraeducator fidelity, which were subsequently sustained following the removal of systematic performance feedback on paraeducator adherence to the protocol. The implementation of the group contingency program by paraeducators also led to increases in the rates of interactions between paraeducators and students, increases in the rates of teacher instruction, and decreases in the rates of aggressive behavior by students. Findings of the study are discussed within the context of developing effective training methods for paraeducators working alongside students with EBD.
Treatment integrity (also referred to as "treatment fidelity," "intervention integrity," and "procedural reliability") is an important methodological concern in both research and practice because treatment integrity data are essential to making valid conclusions regarding treatment outcomes. Despite its relationship to validity, treatment integrity has been largely overlooked in education research and related fields. Influences over the past 5 years have increased attention to this construct, yet many questions remain in regard to how it should be addressed. The purpose of this article is twofold. First, we provide a brief overview of the current state of (a) conceptual models of treatment integrity; (b) research on assessment, promotion, and relationship to student outcomes; and (c) treatment integrity related to response to intervention models of service delivery. Second, we suggest future directions for research, practice, and policy related to treatment integrity as we move toward a better scientific understanding of this construct. The article concludes with a description of the purposes of this special series.
School psychologists commonly provide intervention services to children through consultation with teachers. Data suggest, however, that many teacher consultees struggle to implement these interventions. There are relatively few evidence-based strategies for promoting teacher consultees' intervention implementation. The purpose of this study was to evaluate Implementation Planning as a strategy for increasing the adherence and quality with which teacher consultees implemented behavior support plans. Implementation Planning involves adapting intervention steps to the implementation context, providing detailed logistical planning, as well as identifying implementation barriers and developing strategies to address them. Results indicated that teachers' implementation adherence levels increased and quality levels increased with corresponding decreases in variability, compared to standard behavioral consultation. Implications for future research on treatment integrity are discussed.
Copyright © 2015 Society for the Study of School Psychology. Published by Elsevier Ltd. All rights reserved.
Evidence-based practices within a response-to-intervention framework must be implemented with adequate treatment integrity to promote student outcomes. However, research findings indicate educators struggle to implement interventions and logistical considerations may limit the utility of performance feedback, an evidence-based treatment integrity promotion strategy. This study evaluates the effect of implementation planning, a treatment integrity promotion strategy that includes detailed logistical planning and barrier identification adapted from an adult behavior change theory from heath psychology (i.e., the Health Action Process Approach). A multiple baseline across participants design was used to evaluate teachers’ adherence to a behavior support plan as well as their quality of implementation. Results indicated that after intervention training, adherence was initially low and variable, and quality of implementation was moderate to high and variable, but both adherence and quality increased and became less variable after implementation planning. The increases in implementation were more pronounced for two teachers, whose students also had subsequent improvements in their academic engagement and disruptive behavior. These findings highlight the relationship between adequate levels of treatment integrity and student outcomes as well as provide initial support for implementation planning.
Teachers typically enter the field with limited training in classroom management, and research demonstrates that training alone does not result in improved practice. Typically, researchers have relied on time-intensive training packages that include performance feedback to improve teachers’ use of classroom management practices; however, initial evidence suggests that self-management may be an effective and efficient alternative. In this study, the authors directly compared the effects of three different self-monitoring conditions (tally, count, and rate) and no self-monitoring on five middle school teachers’ rate of specific praise using an alternating treatments design. The authors also included baseline and follow-up phases to descriptively explore the effects of self-monitoring across time. Results indicate that noting each instance of specific praise by either tallying or using a counter resulted in optimal performance, and teachers preferred using a counter. Additional study results, limitations, and implications are discussed.
The assessment of treatment integrity is important for determining the effectiveness of an intervention. Treatment integrity, a multidimensional construct, can be evaluated in two ways: by session or component. In this study, adherence, quality, and exposure data are estimated using permanent product data from implementation of the PAX Good Behavior Game to demonstrate how varied assessment of treatment integrity can influence interpretation of implementation. Implications for implementers and school-based decisions are discussed.
Three group-oriented contingency systems (interdependent, dependent, and independent) were compared in a modified reversal design to evaluate each system's effectiveness in controlling the disruptive behavior of a self-contained classroom of educable mentally retarded children. Interpendent and dependent group-contingency systems were more effective than independent contingencies in controlling disruptive behavior. Sequential ordering of contingency phases did not allow one to completely rule out carry-over effects; however, it was unlikely that the independent contingency phases were rendered ineffective due to their positioning in the treatment sequences. The roles of group cooperation, peer reinforcement, and informational feedback in the success of group-oriented contingency systems are discussed.
Out-of-seat and talking-out behaviors were studied in a regular fourth-grade class that included several “problem children”. After baseline rates of the inappropriate behaviors were obtained, the class was divided into two teams “to play a game”. Each out-of-seat and talking-out response by an individual child resulted in a mark being placed on the chalkboard, which meant a possible loss of privileges by all members of the student's team. In this manner a contingency was arranged for the inappropriate behavior of each child while the consequence (possible loss of privileges) of the child's behavior was shared by all members of this team as a group. The privileges were events which are available in almost every classroom, such as extra recess, first to line up for lunch, time for special projects, stars and name tags, as well as winning the game. The individual contingencies for the group consequences were successfully applied first during math period and then during reading period. The experimental analysis involved elements of both reversal and multiple baseline designs.
Assessing response to intervention (RTI) is a complex process that requires selecting interventions, identifying a reasonable strength of intervention, implementing the intervention, assessing progress, and applying decision rules to the resulting data. Appropriate standards and practices are critically important for all of these activities. However, the implementation of interventions has received scant attention in the RTI literature to date despite emerging empirical evidence that assuring intervention implementation may be one of the most challenging aspects of implementing RTI. This article discusses the critical role of plan implementation within RTI. Documentation that interventions were implemented is both a substantive necessity for implementing an entitlement process in which services are integrated with assessment (RTI) and a due process protection for students. The current data regarding the assessment and improvement of intervention plan implementation is also briefly reviewed.
In this article, we examine the impact of two universal, first-grade preventive interventions on the prevalence of conduct problems and disorder and mental health service need and use in early adolescence. The classroom-centered (CC) intervention was designed to reduce the risk for later conduct problems and disorder by enhancing teachers' behavior management in first grade, whereas the Family-School Partnership (FSP) intervention targeted improvement in parent-teacher communication and parents' child behavior management strategies. At Grade 6, or age 12, CC and FSP intervention children received significantly lower ratings from their teachers for conduct problems than control children. CC and FSP children were also significantly less likely than control children to meet diagnostic criteria for Conduct Disorder and to have been suspended from school in the last year. In addition, the CC intervention was associated with significantly lower rates of child mental health service need and utilization. Overall, the CC intervention appeared to be the more effective of the two in reducing the prevalence of conduct problems and disorder at age 12 and in reducing mental health service need and utilization. Nevertheless, future studies may show that the combination of CC and FSP interventions produces additive or even synergistic effects.
Teachers are oen ill-prepared to manage classrooms in urban schools. In the present study, an empirically-based behavioral management strategy, the Good Behavior Game (Game), was investigated. The effects of the Game on student behavior and teacher response statements, including praise, were examined. A teacher with 22 students in a first grade classroom of an urban elementary school participated in implementation of the Game. Using a withdrawal design, results showed that student on-task behavior increased while disruptive behavior decreased, replicating previous findings. The number of teacher praise statements remained at near zero levels across conditions. Frequency of teacher neutral and negative statements varied with the level of student disruptive behavior. Teacher praise and limitations are discussed.
The role of children's on-task behavior in the prevention of aggressive behavior was assessed among 570 Dutch speaking children followed from second- to third-grade elementary school in Flanders, Belgium. A first objective was to investigate whether individual level variation of on-task behavior moderated the impact of a universal preventive intervention, the Good Behavior Game (GBG), on aggression development, controlling for classroom levels of on-task behavior. The second goal was to study whether improved on-task behavior or reductions in peer rejection mediated intervention impact on children's aggression. Second-grade classrooms were randomly assigned to the GBG or a control condition. Results showed that intervention impact was found only among children who had low-level on-task behavior at baseline. These children showed a decrease in aggression when in the GBG condition, which was not found among control group children. The reduction in aggression among low on-task children was mediated by reductions in peer rejection. No mediation effect of on-task behavior was found. These results suggest that the effect of a universal preventive intervention may depend upon initial levels of on-task behavior and that improvements in social relations with peers may explain the reductions in aggression among these low-on task children.
In an effort to expand the pool of scientific evidence available for review, the What Works Clearinghouse (WWC) assembled a panel of national experts in single-case design (SCD) and analysis to draft SCD Standards. SCDs are adaptations of interrupted time-series designs and can provide a rigorous experimental evaluation of intervention effects. SCDs can provide a strong basis for establishing causal inference, and these designs are widely used in applied and clinical disciplines in psychology and education, such as school psychology and the field of special education. In this paper, the panel provides an overview of SCDs, specifies the types of questions that SCDs are designed to answer, and discusses the internal validity of SCDs. The panel then proposes SCD Standards to be implemented by the WWC. The Standards are bifurcated into Design and Evidence Standards. The Design Standards evaluate the internal validity of the design. Reviewers assign the categories of "Meets Standards," "Meets Standards with Reservations," and "Does not Meet Standards" to each study based on the Design Standards. Reviewers trained in visual analysis will then apply the Evidence Standards to studies that meet standards with or without reservations, resulting in the categorization of each outcome variable as demonstrating "Strong Evidence," "Moderate Evidence," or "No Evidence." Appended are: (1) Attrition Diagram; and (2) Visual Analysis. (Contains 1 table, 11 figures and 16 footnotes.)
Prevention and intervention research studies often fail to include an assessment of program integrity, and when they do, it is often examined in a limited way. Further, despite efforts to reform the intervention research process to include community stakeholders more actively in every phase of investigation, current practice generally employs a hierarchical model of integrity that fails to be responsive to community needs and priorities. In this article, we describe the traditional, hierarchical model of integrity and contrast this framework with a partnership model of treatment integrity. The limitations of the hierarchical model are illustrated through the description of two school-based prevention programs. Core features of the partnership model and steps for implementing this framework in research and practice are described. Although the partnership model has limitations when conducting efficacy research in clinical trials, it has clear advantages over the hierarchical model in conducting effectiveness research and research that is directly linked with practice. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 495–507, 2005.
Non-classroom settings are often the most violence-prone areas within a school. This study investigated the impact of an interdependent group contingency on the disruptive behaviors of students in grades K–6 in an urban school cafeteria. Nine female noontime aides and National School and Community Corps staff members implemented the Lunchroom Behavior Game (LBG), a modification of the Good Behavior Game (Barrish, Saunders, & Wolf, 1969), within a multiple-baseline design across three lunch periods. Results showed a decrease in the level of disruptive behaviors following the implementation of the LBG in each lunch period and a decreasing trend for two of the three lunch periods. Discussion focuses on the use of the LBG in preventing antisocial behavior and role expansion for school psychologists interested in promoting school-based prevention strategies.
The good behavior game (GBG) is a classwide group contingency that involves dividing the class into two teams, creating simple rules, and arranging contingencies for breaking or following those rules. Five kindergarten teachers and classrooms participated in this evaluation of the GBG. Disruptive behavior markedly decreased in all five classrooms as a result of the intervention. This study extends the GBG literature by systematically replicating the effects of the GBG with the youngest group of students to date.
The efficacy of the Good Behavior Game was examined in a multiethnic New York City public high school. Classroom rules were posted and students were divided into two teams. A reinforcement preference questionnaire was used to select daily and weekly prizes. The classroom teacher indicated that he was going to place a check on the board after every rule infraction as he named rule violators and their infractions. Students were also told that the team with the fewest marks at the end of each day would become the daily winners and receive prizes. They were also told that the team with the fewest marks for the week would be recognized as the weekly winners and receive additional prizes. The rate of disruptive behavior was charted across four treatment phases using a reversal design. The game phases were associated with marked reductions in the rate of seat leaving, talking without permission, and aggression. Teacher and student feedback supported the social validity of the procedure.
The purpose of this study was to examine the differential effects of 2 versions of the Good Behavior Game (Barrish, Saunders, & Wolf, 1969), allocating teacher attention to rule violations (GBG-response cost) and to rule following (GBG-reinforcement), on student and teacher behavior. The participants were 6 kindergarten students who were nominated as the 3 most disruptive students in each classroom. The study was conducted using single-case A/B/A/C/B/C reversal design with each teacher randomly assigned to either GBG-response cost or GBG-reinforcement condition for implementation in the first B phase. Results indicated that both versions were effective at reducing rule violations and that GBG-reinforcement consistently resulted in either comparable or lower levels of rule violations across classrooms and students. In addition, GBG-reinforcement was preferred by the teachers as a better fit to their classrooms. The implications of the findings to teachers and school psychologists in classroom settings are discussed.
We examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. Only 39 of 162 outcome studies featured specified procedures for the documentation of fidelity. Of these, only 13 considered variations in integrity in analyzing the effects of the program. Lowered adherence to protocol was often associated with poorer outcome. There was mixed evidence of dosage effects. The omission of integrity data, particularly measures of adherence, may compromise the internal validity of outcome studies in the prevention literature. We do not view procedures for integrity verification as inconsistent with the adaptation of interventions to the needs of receiving communities.
Out-of-seat and talking-out behaviors were studied in a regular fourth-grade class that included several "problem children". After baseline rates of the inappropriate behaviors were obtained, the class was divided into two teams "to play a game". Each out-of-seat and talking-out response by an individual child resulted in a mark being placed on the chalkboard, which meant a possible loss of privileges by all members of the student's team. In this manner a contingency was arranged for the inappropriate behavior of each child while the consequence (possible loss of privileges) of the child's behavior was shared by all members of this team as a group. The privileges were events which are available in almost every classroom, such as extra recess, first to line up for lunch, time for special projects, stars and name tags, as well as winning the game. The individual contingencies for the group consequences were successfully applied first during math period and then during reading period. The experimental analysis involved elements of both reversal and multiple baseline designs.
Treatment integrity for academic interventions in real-world settings
- R O Hawkins
- J Q Morrison
- S Musti-Rao
- J A Hawkins
Hawkins, R. O., Morrison, J. Q., Musti-Rao, S., & Hawkins, J.
A. (2008). Treatment integrity for academic interventions in
real-world settings. School Psychology Forum, 2, 1-15.
Using the Good Behavior Game to decrease disruptive behavior while increasing academic engagement with a Head Start population (Unpublished doctoral dissertation)
- B M Hunt
Hunt, B. M. (2012). Using the Good Behavior Game to decrease
disruptive behavior while increasing academic engagement
with a Head Start population (Unpublished doctoral dissertation). University of Southern Mississippi, Hattiesburg.
An exploratory investigation of the influences of observation timing on treatment fidelity estimates
- M A Collier-Meek
- L M H Sanetti
- L M Fallon
- S M Chafouleas
Collier-Meek, M. A., Sanetti, L. M. H., Fallon, L. M., &
Chafouleas, S. M. (2018). An exploratory investigation of the
influences of observation timing on treatment fidelity estimates. Manuscript submitted for publication.
Supporting successful interventions in schools: Tools to plan, evaluate, and sustain effective implementation
- L M H Sanetti
- M A Collier-Meek
Sanetti, L. M. H., & Collier-Meek, M. A. (2018). Supporting successful interventions in schools: Tools to plan, evaluate, and sustain
effective implementation. New York, NY: Guilford Press.