Content uploaded by James S. Kim
Author content
All content in this area was uploaded by James S. Kim on Apr 21, 2020
Content may be subject to copyright.
A preview of the PDF is not available
Effectiveness of Structured Teacher Adaptations to an Evidence‐Based Summer Literacy Program
Abstract and Figures
The authors conducted a cluster-randomized trial to examine the effectiveness of structured teacher adaptations to the implementation of an evidence-based summer literacy program that provided students with (a) books matched to their reading level and interests and (b) teacher scaffolding for summer reading in the form of end-of-year comprehension lessons and materials sent to students' homes in the summer months. In this study, 27 high-poverty elementary schools (75-100% eligibility for free or reduced-price lunch) were matched by prior reading achievement and poverty level and randomly assigned to one of two implementation conditions: a core treatment condition that directly replicated implementation procedures used in previous experiments, or a core treatment with structured teacher adaptations condition. In the adaptations condition, teachers were organized into grade-level teams around a practical improvement goal and given structured opportunities to use their knowledge, experience, and local data to extend or modify program components for their students and local contexts. Students in the adaptations condition performed 0.12 standard deviation higher on a reading comprehension posttest than students in the core treatment. An implementation analysis suggests that fidelity to core program components was high in both conditions and that teachers in the adaptations condition primarily made changes that extended or modified program procedures and activities in acceptable ways. Adaptations primarily served to increase the level of family engagement and student engagement with summer books. These results suggest that structured teacher adaptations may enhance rather than diminish the effectiveness of an evidence-based summer literacy program.
Figures - uploaded by James S. Kim
Author content
All figure content in this area was uploaded by James S. Kim
Content may be subject to copyright.
... Several school-based studies have included older students selected on the basis of poor reading comprehension scores, and assigned to programs designed to facilitate comprehension either through whole class or small group programming. Many well-designed research interventions have been associated with positive effects of differing size in controlled efficacy studies (e.g., Kim et al., 2017;Snow, Lawrence, & White, 2009;Vaughn et al., 2010Vaughn et al., , 2011Vaughn et al., , 2012, but far more variable findings in classroom-level practice (LaRusso, Donovan, & Snow, 2016). Year-long reading interventions implemented with middle school struggling readers have yielded very small effect sizes ranging from 0.09 to 0.16 (Corrin, Somers, Kemple, Nelson, & Sepanik, 2008;Kemple et al., 2008;Vaughn et al., 2019). ...
... Focusing on middle school students, Kim and colleagues (Kim et al., 2017) developed and evaluated a multiple component intervention (STARI) that addresses decoding, fluency, and comprehension aspects of reading, adapting engagement and motivation principles from Guthrie, Wigfield, and You (2012). STARI students demonstrated significantly greater improvement than BAU controls on three of six reading outcomes, with effect sizes that ranged from 0.18 to 0.21, all relatively small effects. ...
... STARI students demonstrated significantly greater improvement than BAU controls on three of six reading outcomes, with effect sizes that ranged from 0.18 to 0.21, all relatively small effects. Notably, a significant mediating effect of reading engagement was found on these three outcomes (d's from 0.32 to 0.35; Kim et al., 2017). ...
Many children and adolescents with reading comprehension difficulties demonstrate impairment in lexical and sublexical foundational skills, and require intervention for multiple levels of the reading system. This paper provides a merged analysis of results from four controlled intervention studies conducted by our research group: Children and adolescents with reading disabilities (RD) received multiple-component remediation at different ages (early elementary, middle school, high school). All participants demonstrated multi-faceted reading impairment, and all received intervention targeting decoding, word reading, text reading, and comprehension skills. Studies with younger children (Grades 1–3) demonstrated more robust comprehension effects on multiple comprehension measures, while studies with older readers demonstrated more equivocal comprehension effects despite gain on more foundational skills. A parallel moderated-mediation model, using data merged across studies, revealed that intervention-related decoding and word reading change mediated the relationship between intervention and reading comprehension gains. This mediation effect was strongly moderated by age, with younger children demonstrating the greatest mediation. The younger the child, the greater the effects of intervention and the more comprehension gains depended on decoding gains.
... Although adapting the curriculum to school, class and students is a new concept in Turkey, curriculum adaptation patterns have been determined in the Western literature (Bernard 2017;Burkhauser and Lesaux 2017;Drake and Sherin 2006;Li and Harfitt 2017;Remillard 2005;Sherin and Drake 2009) and even curriculum adaptation models have been introduced (Remillard and Heck 2014). In addition, studies have started to look into the relationships between the adaptations made by teachers and student achievement (Kim et al. 2017;Penuel, Gallagher, and Moorthy 2011). Studies have examined the quality of the adaptations and reported that teachers' curriculum reasoning and their adaptations can be improved through professional development (Davis et al. 2011;McCarthey and Woodard 2018;Westwood Taylor2013). ...
... Therefore, it could be suggested that the literature on teachers' curriculum adaptation patterns has become mature enough to develop an instrument to determine this construct. Moreover, studies recommending that teachers be supported in the process of enacting structured adaptations and related professional development (PD) be held (Debarger et al. 2017;Kim et al. 2017;Westwood Taylor 2013) usually employ observations and interviews to examine the changes in adaptations depending on teacher development. However, there is a need for instruments to be used in evaluating the effectiveness of PD activities intended for teachers on their curriculum adaptation and determining the correlation between adaptation patterns and teacher development. ...
... As stated above, studies on the process of adapting the curriculum to the class are seen to include findings based on qualitative data only, lacking an instrument that can determine adaptation patterns. In mixed-methods studies, on the other hand, researchers aim to collect large-scale data by focusing on the frequency of adaptations (Fogleman, McNeill, and Krajcik 2011;Kim et al. 2017); however, it is not clear which adaptation patterns support student learning or which patterns come out as a result of person-specific, curricular or contextual factors. Quantitative measurements of adaptation patterns could enrich both student learning and the knowledge base concerning the reasons for adaptations. ...
Curriculum adaptation portrays the way teachers ‘tweak’ the curriculum. The findings of related studies are based solely on qualitative data, and there is no instrument that can determine curriculum adaptation patterns. Based on this gap, the aim of this study is to develop the Curriculum Adaptation Patterns Scale (CAPS). After examining the qualitative studies, 113 draft items were created in eight adaptation patterns. Drawing on the experts and discussions, the dimensions and number of items were reduced. In the first pilot study (n = 322), a three-dimensional structure explaining 52% of the total variance was obtained. CAPS was re-applied on 361 teachers for confirmatory factor analysis and the fit indices corresponded to sufficient. The three emerging factors are named as ‘omitting’, ‘extending’ and ‘replacing or revising’. The reliability coefficients for these factors are also adequate (0.87, 0.72 and 0.85, respectively). The analyses indicate that the CAPS can be used for examining teachers’ curriculum adaptation patterns.
... Studies have noted that the interventions targeting at encouraging fathers' engagement in the education of their children are lacking (Kim & Quinn, 2013;Kim et al., 2016Kim et al., , 2017. Mncanca and Okeke (2016) found that children of fathers who are actively involved in their early education live in cognitively stimulated homes. ...
... Mncanca and Okeke (2016) found that children of fathers who are actively involved in their early education live in cognitively stimulated homes. Studies have shown that fathers who interact with their children are most likely to benefit from some input on knowledge and skills from teachers before or during home-based summer learning programs (Maxwell et al., 2014;Kim & Quinn, 2017;Stein, 2017). Nazneen and Chopra (2016) found that in the Philippines, a policy related to the provision of cash assistance to poor fathers helped them to be actively engaged in the education of their children. ...
... perceptions regarding policies related to fathers' engagement in children's early education. (2014) In England, interventions designed to prevent or address this are not used widely, but they show some promise in international studies (Kim & Quinn, 2013;Kim et al., 2016Kim et al., , 2017 shown that fathers and children are likely to benefit from some input on knowledge and skills from teachers before or during home-based summer learning programs (Maxwell et al., 2014;Kim & Quinn, 2017;Stein, 2017). Texting fathers over the summer may be beneficial in supporting such activity (Kraft & Monti-Nussbaum, 2017). ...
Drawing from ecological system theory, the focus of this study was on the perceptions of parents and teachers regarding policies related to fathers' engagement in children's early education. A descriptive survey research was used to obtain primary data for the study using a sample of 309 participants comprising 94 teachers, and 215 parents of children in the selected schools. A structured questionnaire instrument was used for data collection for the study. Data were analyzed using mean, multiple regression and t-test statistic. The study found that both parents and teachers had positive perceptions regarding policies related to fathers' engagement in children's early education. Additionally, it was found in the study that demographic characteristics had no significant influences on the mean perception ratings of parents and teachers on policies related to fathers' engagement in children's early education. The researchers recommended that the South African government should ensure full implementation of ECCE policies that lay out concrete commitment and guidelines for fathers' engagement in their children's education.
... The teacher support in these latter programmes may include information on general language-enhancing strategies that teachers receive as part of professional development during workshops before the intervention, and that teachers are encouraged to use in a flexible way throughout the implementation of the intervention. For some intervention programmes, a mixture of scripted and less scripted components is included (Kim et al. 2017). ...
... We identified three studies that shed light on the impact of intervention scriptedness on teachers' instructional talk. First, Kim et al. (2017) examined the effectiveness of two conditions of the programme READS, an original version with scripted components and an adapted version in which teachers were offered guidance on how to modify specific parts of the programme. Findings showed that teachers in both conditions implemented the core elements with high levels of fidelity, while teachers in the adapted condition also afforded extensions and modifications in line with the programme's aim. ...
... In the literature, adaptations of instructions in interventions have recently received increased attention, emphasising a need to consider the possibility of flexible adaptations as an active ingredient in interventions. Combining different levels of scripted instructions can also give teachers greater agency, which may enhance the quality of the implementation and students' outcomes (Kim et al. 2017). This perspective concurs with findings from Neugebauer et al. (2017), where adaptations in the form of extensions from the scripted instructions were the most beneficial component in promoting students' language growth. ...
Despite huge investments in interventions designed to support oral language skills in early childhood and beyond, many of the interventions fail to identify impacts on children’s language learning. Programmes may have limited impact because they do not sufficiently succeed in supporting teachers’ instructional talk, and thus, more efficiently promote children’s language learning. The present study examined the extent to which 15 teachers in Norway implementing a language intervention programme designed to enhance students’ second-language learning in first and second grade demonstrated changes in their instructional talk over the 8-week programme. The programme consisted of scripted parts (labelling pictures of targeted words, repeated exposures), as well as soft scripted parts (word relations and definitions) and minimally scripted parts (narratives and explanations that extended the here-and-now). Teachers received professional development that qualified them to implement the programme. Analysis of modifications in teachers’ instructional talk was based on audio-recorded small-group interactions, comparing characteristics of teacher talk at the beginning and end of the 8-week programme. Results revealed that teachers’ instructional talk developed to include more word definitions and extended discourse, talk categories aligned with the less scripted parts of the intervention. Conversely, teacher talk during the scripted parts of the programme did not change.
... In addition, there may be a tension between implementation delity and the need to adapt interventions to teacher instructional routines. Kim et al. (2017) compared a standard literacy intervention with an intervention, which allowed the teachers to make adaptations, based on their knowledge and experience. The results showed that adapted intervention led to higher gains in students' reading comprehension than the standard intervention. ...
... The nding can also be discussed with regard to the relationship between implementation delity and student outcomes. While implementation delity has traditionally been viewed as a prerequisite of intervention success (Durlak & DuPre, 2008), recent studies have pointed to the need of additional indicators of how interventions are integrated into existing teacher practices (Kim et al., 2017;Moore et al., 2019). ...
The aim of this study was to investigate the effects of intervention, combining instruction in cooperative learning and reading comprehension strategies on students’ reading comprehension in grade 5.The teachers in the experiment group implemented the intervention while the teachers in the control group received training in reading comprehension strategies and taught as usual. Students in the experiment group and control group participated in tests of reading comprehension before and after the intervention.The results showed that being a part of the experiment group did not lead to greater gains in reading comprehension above the control group. Students of teachers who did not fully implement the intervention attained higher scores on reading comprehension than students in the control group. Students of teachers who fully implemented the intervention, on the other hand, received lower scores. The results are discussed with regard to research on teachers’ integration of intervention into their instructional routines.
... In this study, students of teachers who had autonomy to make adaptations to the program outperformed their peers on standardized literacy outcomes (ES p 0.25-0.60). In a recent cluster randomized trial of a summer reading program, Kim et al. (2017) examined the effectiveness of an approach that supported teachers in making acceptable, "structured adaptations" (i.e., aligned with the program's theory of change and only within certain intervention domains) to the program compared with an approach that asked teachers to implement the program without adaptation. Students in the structured adaptations condition performed 0.12 standard deviations higher at posttest on a pilot study of unlocking understanding professional • 000 standardized test of reading comprehension than did students in the no-adaptation comparison condition. ...
... Recall that teachers in this study received initial training, regular coaching, and a substantial number of fully scripted lessons before they were asked to take on more lesson planning responsibilities. This structuring of teacher adaptations and gradual release of responsibility in lesson planning has been a component of other effective teacher PD interventions (e.g., Kim et al., 2017;McMaster et al., 2010). ...
This study piloted a reading comprehension instruction professional development intervention for teachers of upper elementary grade students (Unlocking Understanding), exploring the relative effects of two versions of the intervention that varied the degree of lesson scripting. Results indicated a statistically significant difference between the fully scripted and partially scripted (i.e., increasingly teacher-planned) groups on one of three student outcome measures. Specifically, students in classrooms with partially scripted lessons performed better on a statewide reading test (ES p 0.42) than did students in classrooms where teachers used fully scripted lessons. However, there were no significant group differences on two other reading comprehension measures. Teachers reported a high degree of satisfaction with the training, although they stated a preference for fully scripted lessons compared with partially scripted ones. Teachers in both conditions were able to deliver the lessons with acceptable fidelity. Implications for professional development are discussed.
... This study contributes to a growing literature to fill this gap by reliably and comprehensively measuring fidelity of implementation and utilizing principles from psychometrics (e.g. Snyder et al., 2013;Southam-Gerow, 2018;Kim et al., 2017). Defining validity as a unitary construct which requires evidence from an observational measure's content and scoring process as well as its coherence, we first outline how our constructs for fidelity were defined (AERA, 2014;Snyder et al., 2013), focusing on those pertaining to the quality of implementation through studentteacher interactions. ...
Türkiye’de öğretmenlerin resmî/yazılı öğretim programlarını sınıflarına uyarladığı ortaya çıkarılmış olsa da bu sürecin detayları tam olarak bilinmemektedir. Bu araştırmanın amacı, ortaokullarda görev yapan öğretmenlerin öğretim programını sınıfa uyarlama sürecinde ortaya çıkan örüntüleri, uyarlama yapma nedenlerini ve sonuçlarını belirlemektir. Karma yöntemin sıralı açıklayıcı deseni temele alınarak programı uyarlama örüntüleri ölçeğiyle farklı branşlardaki 545 ortaokul öğretmeninden veri toplanmış, analizlerden sonra 12 öğretmenle görüşmeler yapılmıştır. Bulgular ortaokul öğretmenlerinin en çok genişletme, ikinci sırada yeniden düzenleme ve son olarak da atlama yaptıklarını göstermiştir. Öğretmenlerin genişletmeler yapmasında cinsiyet, branş, çalışılan okul türü, eğitim durumu açısından; yeniden düzenlemeler ve atlamalar yapmasında da branş, çalışılan okul türü, haftalık ders saati açısından anlamlı farklılıklar olduğu belirlenmiştir. Öğretmenlerin uyarlama nedenleri ders kitaplarının yetersiz bulunması, öğrencileri liselere giriş sınavına hazırlama, dersi eğlenceli, anlaşılır ve kalıcı kılma, ön bilgi eksiklerini tamamlama, önerilen sürelerin uygun bulunmaması, öğretmenlerin yeterlikleri ve tercihleri, kaynaşıklığı sağlama çabası, öğrenme ihtiyaçlarını giderme, öğretim programı içeriklerinin yoğunluğu ve süre yetersizliği, ideolojik propaganda unsurlarını azaltma isteğidir. Yapılan uyarlamaların öğrenci motivasyonu ve başarısında artış, öğretim sürelerini dengeleme, öğrenmeleri kolaylaştırma, kafa karışıklığını önleme, zaman kazanma gibi olumlu sonuçlarının yanı sıra ekstra zaman ve iş yükü gerektirme, değişikliklerin sonuçlarını tam olarak belirleyememe ve programdan uzaklaşma gibi olumsuz sonuçları dile getirilmiştir.
Summer numeracy interventions have become standard educational practice across a range of jurisdictions, but there is a paucity of evaluations of those interventions across international settings. We employ a quasi-experimental evaluation of voluntary summer numeracy programs for 569 attendees and 2,193 comparison students in grades 1–3, conducted in 2012–13 in Ontario, Canada. Using multi-level logistic regression models, we find that students with prior academic challenges were most likely to attend the program. Estimating causal effects using entropy balancing to achieve balance, we find that attendees gained about 1.5 months more numeracy learning than did the comparison group (Cohen’s d = .19). These findings are comparable to US studies and add to an international stock of knowledge on summer interventions.
Most educational research treats teachers' deviations from prescribed or scripted programs as a nuisance for efficacy studies. This study reframes this approach to explore how kindergarten teachers' use of critical vocabulary program ingredients as well as improvised interactions contribute to instructional effectiveness. Data from 181 kindergarten teachers and their approximately 1,404 kindergarten students were analyzed using multilevel modeling. Findings demonstrate that teachers with students with large vocabulary gains had higher levels of adherence to the more structured program components and engaged in more adaptive impromptu teacher-student interactions than their less successful teacher peers. This study provides a more flexible and dynamic understanding of implementation fidelity.
Professional development programs are based on different theories of how students learn and different theories of how teachers learn. Reviewers often sort programs according to design features such as program duration, intensity, or the use of specific techniques such as coaches or online lessons, but these categories do not illuminate the programs’ underlying purpose or premises about teaching and teacher learning. This review sorts programs according to their underlying theories of action, which include (a) a main idea that teachers should learn and (b) a strategy for helping teachers enact that idea within their own ongoing systems of practice. Using rigorous research design standards, the review identifies 28 studies. Because studies differ in multiple ways, the review presents program effects graphically rather than statistically. Visual patterns suggest that many popular design features are not associated with program effectiveness. Furthermore, different main ideas are not differentially effective. However, the pedagogies used to facilitate enactment differ in their effectiveness. Finally, the review addresses the question of research design for studies of professional development and suggests that some widely favored research designs might adversely affect study outcomes.
Available for download at
http://nirn.fpg.unc.edu/resources/implementation-research-synthesis-literature
Manipulation checks should be used in psychotherapy trials to confirm that therapists followed the treatment manuals and performed the therapy competently. This article is a review of some strategies that have been used to document treatment integrity; also, their limitations are discussed here. Recommendations for improving these checks are presented. Specific guidelines are offered regarding when and how to assess both therapist adherence to treatment protocols and competence.
To improve the reading comprehension outcomes of children in high poverty schools, policymakers need to identify reading interventions that show promise of effectiveness at scale. This study evaluated the effectiveness of a low-cost and large-scale summer reading intervention that provided comprehension lessons at the end of the school year and stimulated home-based summer reading routines with narrative and informational books. We conducted a randomized controlled trial involving 59 elementary schools, 463 classrooms, and 6,383 second and third graders and examined outcomes on the North Carolina End-of-Grade (EOG) reading comprehension test administered nine months after the intervention, in the children's third- or fourth-grade year. We found that on this delayed outcome, the treatment had a statistically significant impact on children's reading comprehension, improving performance by .04 SD (standard deviation) overall and .05 SD in high poverty schools. We also found, in estimates from an instrumental variables analysis, that children's participation in home-based summer book reading routines improved reading comprehension. The cost-effectiveness ratio for the intervention compared favorably to existing compensatory education programs that target high poverty schools.
This article presents an evaluation of the first 2 years of a research-based summer learning program that provided self-selected and developmentally appropriate books to students in low-income and low-resource elementary schools by a local philanthropic organization in a large urban district. The evaluation found evidence of a positive effect of participation in the program on the state year-end standardized reading assessment but found no statistically significant effects on the proximal measures of reading achievement in the fall after summer vacation. The article also provides an analysis of implementation of the program and lessons learned that could be useful to other organizations that are interested in implementing similar programs.
Transcending the low status of educational research will require demonstrating its relevance to improvements in practice. Educational progress is most likely to emerge from approaches to research that create an equal footing for practitioners and researchers, recognizing that though these groups accumulate and curate knowledge in different ways, they both have a role in creating tools (curricula, practices, professional development approaches) that can be used to forge lasting improvements. A brief history of the ongoing shift toward practice-embedded educational research (PEER) demonstrates its increasing acceptance and popularity and suggests modifications to the future selection of research topics, funding mechanisms, and professional preparation of both practitioners and researchers.
In this chapter, we review research on students’ engagement in reading activities and how classroom instructional practices influence engagement in reading and other academic activities. We define engaged readers as motivated to read, strategic in their approaches to reading, knowledgeable in their construction of meaning from text, and socially interactive while reading. We present a conceptual model of reading engagement linking classroom practices directly and indirectly to students’ motivation to read, behavioral engagement in reading, and reading achievement. A major premise of this model is that behavioral engagement in reading mediates the effects of classroom practices on reading outcomes. We present evidence from a variety of experimental and correlational studies documenting the direct and indirect links among classroom practices, motivation, behavioral engagement, and achievement outcomes. One reading comprehension instructional program on which we focus is Concept-Oriented Reading Instruction. This program integrates strategy instruction and instructional practices to foster students’ reading motivation, and teaches reading, in particular, in content domains such as science and social studies.