Technical ReportPDF Available

Quality-outcomes study for Seattle Public Schools summer programs, 2016 program cycle

January 2017

Project: Evaluation Exemplars from QIS Data

Authors:

QTurn LLC

This paper compares outcomes for struggling students in high quality instructional settings to outcomes for similar students in low quality instructional settings.

Attendance in Seattle Public Schools Summer Programs, 2016

…

Instructional Responsiveness Profiles

…

"Low" Instructional Responsiveness Profiles for SPS and Other Cities' Programs

…

Figure B-14. State Math Assessment 2016-2017 (Z-score)

…

Figure B-16. State Reading 2016-2017 (Z-score)

…

Figures - uploaded by Charles Smith

Content may be subject to copyright.

Content uploaded by Charles Smith

Content may be subject to copyright.

301 W. Michigan Ave. | Suite 200 | Ypsilanti, Michigan 48197 | Phone: 734.961.6900 | www.cypq.org

The David P. Weikart Center for Youth Program Quality is a division of The Forum for Youth Investment.

Quality-Outcomes Study for Seattle Public

Schools Summer Programs, 2016 Program Cycle

Submitted to the Raikes Foundation on January 11, 2018

Charles Smith, PhD

Leanne Roy

Stephen C. Peck, PhD

Colin Macleod

Katharine Helegda

John Hughes

301 W. Michigan Ave. | Suite 200 | Ypsilanti, MI 48197 | 734.961.6900 | www.cypq.org

The David P. Weikart Center for Youth Program Quality is a division of the Forum for Youth Investment

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 2

The David P. Weikart Center for Youth Program Quality empowers education and human service leaders to adapt,

implement and bring to scale best-in-class, research-validated quality improvement systems to advance child and

youth development. Afterschool and other out-of-school time systems throughout the United States rely on the

Weikart Center’s intervention, performance metrics and aligned professional development to drive their continuous

improvement efforts. These include an evidence-based intervention model (Youth Program Quality Intervention, or

YPQI) and core set of instructional quality metrics (Youth Program Quality Assessment, or Youth PQA).

www.cypq.org

The Weikart Center is a division of the Forum for Youth Investment.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 3

Table of Contents

Summary ....................................................................................................................................................... 4

Introduction ................................................................................................................................................... 5

Theory of Change ..................................................................................................................................... 6

Method .......................................................................................................................................................... 7

Participants ................................................................................................................................................ 7

Measures ................................................................................................................................................... 8

Data Collection ......................................................................................................................................... 9

Results ......................................................................................................................................................... 11

Attendance .............................................................................................................................................. 11

Academic Skill Change ........................................................................................................................... 11

Profiles of Instructional Responsiveness ................................................................................................ 12

Academic Skill Gains by Profiles of Instructional Responsiveness ....................................................... 13

Academically At-Risk Students .............................................................................................................. 16

Discussion and Recommendations.............................................................................................................. 17

Strengths and Limitations of the Study ................................................................................................... 18

Recommendations ................................................................................................................................... 19

References ................................................................................................................................................... 20

Appendix A. Methodology and Results ..................................................................................................... 21

Descriptive Statistics and Attrition Analyses .......................................................................................... 21

Instructional Responsiveness Profiles ..................................................................................................... 22

Academic Skill Growth Models .............................................................................................................. 22

Academically At-Risk Students .............................................................................................................. 25

Appendix B. Methodology and Results for Multi-level Models ................................................................ 27

Data and Methodology ............................................................................................................................ 27

Results ..................................................................................................................................................... 29

Within-Program Change ..................................................................................................................... 29

Within Program Change - 3rd and 4th graders ................................................................................... 31

Change in State Assessments - 3rd and 4th Graders ........................................................................... 36

Moderation Analysis - 3rd and 4th Graders ........................................................................................ 40

Appendix C. Plan for Quasi-Experimental Test of SPS Summer Program Participation .......................... 46

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 4

Summary

This quality-outcomes study was designed to both (a) describe performance in Seattle Public

Schools (SPS) summer learning programs in ways that are useful to staff and (b) provide evaluative

evidence (i.e., validity) for an instructional model that includes challenging academic content and

responsive instructional practices.

Results from this study were mainly positive yet partially ambiguous. Summer program offerings

were well-attended and characterized by high-quality instructional practices, with a majority of students

increasing their literacy and math skills during the program. Findings about the association between

exposure to more responsiveness instruction (e.g., quality) and academic skill change were mixed.

Results include:

Positive academic skill change was found in the raw data, including for academically at-risk

students. Positive change on the academic performance measures used during the summer program was

found for 73% of students, and positive change on the academic achievement tests was found for 74% of

students from the 2015 to 2016 school year. Standardized effect sizes for the full sample ranged from

medium to large (i.e., dz = .56 - .95) across the seven academic skill measures.

Attendance was regular, and instructional responsiveness was consistently high. Summer

program attendance for 21 or more days (out of a total possible 27 days) was observed for 77% of

students. Analysis of instructional responsiveness using the Summer Learning PQA revealed three

profiles of instructional responsiveness at the point of service: high, medium, and low quality. However,

compared to other urban samples, the “low” SPS profile is not very low.

Students in SPS summer programs had similar rates of skill change across profiles of

instructional responsiveness in the most rigorous models for 3rd and 4th grade students (N = 535); that is,

there was insufficient evidence in support of the hypothesized pattern of differential skill change across

profiles of instructional quality. However, these results should be interpreted with caution due to the

absence of a true low-quality instructional practices subgroup in the sample. Less statistically rigorous

but more theoretically well-specified models for the entire K-4 sample (N = 1060) revealed a positive

association between instructional quality and academic skill change, despite the lack of a true low-quality

subgroup.

Analyses of academically at-risk students revealed similarly mixed results. In the more

statistically rigorous models with grades 3-4, students who entered SPS summer programs below

proficient on academic achievement tests for the prior school year (2015-16) showed similar rates of

academic skill change across profiles of instruction. In the theoretically well-specified models,

academically at-risk students showed greater changes in academic skills in summer programs with higher-

quality instructional practices.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 5

Introduction

Since 2013, a collaborative of funders, public and private summer learning service providers, and

several technical services organizations have partnered to improve the quality and effectiveness of

summer learning services in cities such as Denver, CO; Grand Rapids, MI; Oakland, CA; Seattle, WA;

and St. Paul, MN. The method of improving quality and outcomes was a continuous improvement

intervention for summer service providers’ organizations called the Summer Learning Program Quality

Intervention (SLPQI). In the SLPQI, summer learning organizations follow a cycle of planning,

assessment, and improvement, building the quality of summer instruction toward validated standards and

benchmarks over successive cycles.

The SLPQI was designed to generate cumulative at-scale

improvement in the quality of instructional practices and student academic outcomes in school district and

community-based summer learning programs.

Over five summer cycles, an iterative sequence of design and development studies were

conducted to evaluate (a) SLPQI implementation fidelity and feasibility; (b) adaptations to the design,

training, and technical assistance supporting implementation; and (c) the validity of the standard for high-

quality instructional practices used in the SLPQI (Smith et al., 2017; Smith et al., 2015;

http://cypq.org/SummerLearningPQI ). The standard for high-quality instructional practice in the SLPQI

is the Summer Learning Youth Program Quality Assessment (Summer Learning PQA). The SLPQI and

Summer Learning PQA were designed to focus the science and practice of continuous improvement on

qualities of the summer curriculum (i.e., responsive practices and challenging content) that build social,

emotional, and academic skills.

As part of this broader SLPQI design and development work, during the summers of 2015 and

2016, the Weikart Center, Seattle Public Schools, Schools Out Washington, and the Raikes Foundation

collaborated to conduct two studies focused on the following research questions:

 Are Seattle Public Schools (SPS) summer programs high quality and well attended?

 Does participation in a higher-quality summer program predict more academic skill gain

during both the summer program and the subsequent school year?

 Do students who enter programs with lower academic skills gain more academic skills in

higher-quality programs?

In addition to describing performance in the Seattle Public Schools summer programs, these

studies address the validity of the Summer Learning PQA as a standard for high-quality instruction and

The Summer Learning Program Quality Intervention (SLPQI) is a continuous improvement intervention for

summer learning programs that includes four core components: (a) standards and measures for quality of

management and instructional practices (i.e., the Summer Learning PQA), (b) training and technical assistance

supports, (c) performance data products, and (d) a continuous improvement cycle that fits the prior three elements to

local circumstances and resources.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 6

the core performance metric in the SLPQI. Association between the quality of instruction and academic

skill development is a critical aspect of validity for that standard and for both the SLPQI and the Summer

Learning PQA. Results for the 2015 summer cohort are reported in Smith et al. (2015), and the current

report presents results for the 2016 summer cohort.

Theory of Change

Within the 60 SPS summer programs studied in summer 2016, there were important similarities:

First, the program-offering design included a curriculum with challenging academic content where expert

staff led youth through intensive skill-building sequences over many hours of practice. Second, each

curriculum emphasized responsive instructional practices that were designed to build social and

emotional learning (SEL) skills; that is, to help youth be successful at their learning threshold.

Sometimes learning a new skill can be frustrating, boring, or anxiety-provoking; however, in SPS summer

programs, staff were trained to step in, provide reassurance, and model appropriate thinking and behavior

when learning challenges occurred (i.e., to engage in co-regulation

). Third, each program offering was

based on the theoretically-informed and evidence-based idea that the combination of challenging

academic content and responsive instruction can help students grow skills simultaneously in both the

targeted academic skill domains and the SEL skill domains that support academic skill learning (e.g.,

managing emotions, problem solving). Figure 1 illustrates this model of integrated skill learning:

growing mastery in academic skills and growing mastery in SEL skills necessary to learn in any content

area.

Figure 1. SPS Summer Program Theory of Change

The term co-regulation refers to adult behavior designed to help children and youth successfully self-regulate; for

example, to stay focused, keep moving, process emotion, and get the task at hand completed (Murray et al., 2015).

Youth with atypical patterns of development due to exposure to trauma or chronic stress may need higher levels of

co-regulation from adults. Co-regulation is what happens when staff uses responsive practices to keep the stress and

strain of a challenging project curriculum in the optimal range.

2. Responsive SEL

instruction

1. Challenging

academic curriculum

Teacher Practices

in Program

Student Skill Growth

in Program

3. Academic skill

growth

4. SEL skill growth

Student Skill Transfer

in New Setting

6. Application of SEL

skills in new learning

tasks

5. Application of Academic

skills in new learning

tasks

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 7

This study includes measures for items 2, 3, and 5 of Figure 1. Measures of SEL skills during the

program (e.g., emotion expression) or during the subsequent school year (e.g., suspensions) were not

included. We assume that item 1, challenging academic content, was available to students and constant

across all 60 SPS programs that implemented the same Math and Literacy content curricula.

Literacy activities for different grade levels were drawn from two online literacy curricula from

Houghton Mifflin Harcourt: iRead and System44. Students in kindergarten through second grade used

iRead44 and received direct instruction and practice opportunities aimed at improving their reading

fluency, sight word accuracy, and comprehension. Students in grades three and four used System44 and

received direct instruction and practice opportunities aimed at improving reading rate, expression,

accuracy in oral fluency, phrasing, decoding skills, comprehension, and independent reading. During the

five-week program, instruction was delivered in small group settings and through adaptive software

tailored to student skill levels. Both literacy curricula models included manualized training and coaching

for all staff, and fidelity checks were performed by site coordinators.

The math curriculum, Summer Staircase, was developed locally by SPS staff and a Seattle-based

math education consultant, Math for Love (mathforlove.com), which develops play-based math curricula.

The math curriculum, which was developed in 2013, was constructed for each grade band (i.e., K, 1-2,

and 3-4) and aligned to the Common Core standards for each grade level. Teacher training was offered

before Summer Staircase began, and ongoing support was provided throughout the program. This

curriculum, which has been used since 2013, provided opportunities for students to develop their

mathematical content knowledge and skills and their perseverance in mathematical practices. In order to

create a positive experience for students, games and manipulatives (e.g., pig in pairs and pattern blocks)

were a core feature of the curriculum. All in-program assessments were built into, and aligned with, the

respective curriculum.

Method

Participants

Summer program staff consisted of 40 individual teachers grouped into 60 teacher teams across

19 schools where summer programs were located. Summer program offerings included 29 Kindegarten-

2nd grade offerings and 31 3rd-4th grade offerings.

Students in the study included 39 (4%) students in kindergarten, 173 (15%) in 1st grade, 250

(23%) in 2nd grade, 334 (31%) in 3rd grade, 282 (26%) in 4th grade, and 15 (1%) in 5th grade.

The overall

The program targeted K-4, however 15 5th graders entered the program due to an oversight in enrollment.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 8

sample was 48% female, with 50% of participants identified as having limited English proficiency and

21% as having an individualized education plan. Participants were identified as Asian or Other Pacific

Islander (23%), Black or African American (31%), Hispanic or Latino (25%), White (9%), Two or More

Races (8%), American Indian (< 1%), or Native Hawaiian or Pacific Islander (< 1%).

Measures

The following three measure of instructional responsiveness, seven individual-level measures for

academic skill (i.e., five in-program academic performance measures and two state academic

achievement tests), and nine individual-level covariates (e.g., variables that may influence selection into

summer learning programs or rates of academic learning) were used in the study. Mean, standard

deviation, range, and sample size for all measures are listed in Appendix Table A-1.

Instructional Responsiveness. Instructional responsiveness was assessed using the Summer

Learning PQA - Form A, an observation-based measure designed to assess the quality of instructional

practices. Three domains from this measure were used in the study: Supportive Environment (e.g., a

structured environment facilitated through guidance and encouragement; 23 items), Interaction (e.g.,

opportunities for leadership and collaboration; 10 items), and Engagement (e.g., opportunities for

planning and reflection; 11 items). Trained raters produced complete ratings at two time points that were

averaged together to create the three scores (ranging from 1 to 5) for each program.

Sight Words. Sight Words scores refer to the number of target words correctly read by students,

using either the iRead or System44 curricula. Teachers conducted sight words assessment at both the

beginning (Time 1) and end (Time 2) of the program, and a score was recorded for the number of correct

words read from an established list.

Oral Fluency. Oral Fluency scores refer to the number of words per minute correctly identified

from a previously read passage, and it was assessed for 3rd and 4th grade students only. Different passages

were read for one minute, and the correct number of words per minute was recorded.

Math Scores. Math assessments (aka, Math Assessment) were constructed for three grade-level

groupings (i.e., K only, grades 1-2, and grades 3-4) and consisted of ten items aligned to the Common

Core standards for each grade level. The items were developed by the Summer Staircase math curriculum

developer (http://mathforlove.com/) and aligned to the Summer Staircase curriculum. Assessments were

collected during the first and last week of the summer session.

Math Content. Teachers completed an observational checklist for each of their students. The

checklist is a set of grade-level objectives, accompanied by a rating scale (1 = not ready to begin learning

this topic – needs more attention on prerequisites; 2 = ready to begin learning about topic; 3 = making

basic progress; 4 = strong knowledge, with some gaps; 5 = student shows mastery or excellent progress).

The ratings were submitted at the end of the first three weeks of the program and at the end of the sixth

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 9

week. An average score for Common Core Content (aka, Math Content) topics was calculated by

averaging across relevant objectives for each student.

Math Practices. Teachers completed an observational checklist for each of their students. The

checklist is a set of grade-level objectives, accompanied by a rating scale (1 = not ready to begin learning

this topic – needs more attention on prerequisites; 2 = ready to begin learning about topic; 3 = making

basic progress; 4 = strong knowledge, with some gaps; 5 = student shows mastery or excellent progress).

The ratings were submitted at the end of the first three weeks of the program and at the end of the sixth

week. An average score for Common Core Practices (aka, Math Practice) was calculated by averaging

across relevant objectives for each student.

State Math Achievement Test. The Smarter Balanced Math Test was completed in March - June

of 2017 by students in grades 3-4 (http://www.k12.wa.us/smarter/).

State Literacy Achievement Test. The Smarter Balanced English Language Arts (ELA) Test was

completed in March - June of 2017 by students in grades 3-4. (http://www.k12.wa.us/smarter/).

State Math Proficiency Level. Smarter Balanced Math Test scores were divided into four

proficiency levels (i.e., Limited Knowledge, Not Proficient, Proficient, and Advanced) by the Smarter

Balanced Assessment Consortium.

State Literacy Proficiency Level. Smarter Balanced ELA scores were divided into four

proficiency levels (i.e., Limited Knowledge, Not Proficient, Proficient, and Advanced) by the Smarter

Balanced Assessment Consortium.

Covariates. Additional variables were included in statistical models to adjust for certain types of

bias; in particular, biased selection of higher performing students into higher quality classrooms. These

additional variables included: Attendance (days attending summer program), Grade Level (K-4), Gender

(% female), Limited English (% yes), Individualized Educational Plan (IEP) (% yes), Race (% White,

Asian, Black, & Hispanic), 2015-2016 State Math Achievement, and 2015-16 State Literacy

Achievement.

Data Collection

Observational data collection was conducted by Schools Out Washington (SOWA) using data

collection and data management protocols approved by the Weikart Center and SPS. All raters had a

reliability endorsement for the Summer Learning PQA. Raters observed each program for one entire 8:30

a.m. to 12:30 p.m. session on each of two days, at least 1.5 weeks apart, and produced one complete

rating for each observation day. The first sets of observations were conducted between June 29th and July

21st, 2016, and the second set of observations were conducted between July 16th and July 27th, 2016. SPS

staff coordinated collection of student assessment and background information and supplied the Weikart

Center with a complete, de-identified data file for analysis.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 10

Analytic Approach

The research questions were addressed using a three-step analytic approach that combined

pattern-centered and linear models.

First, we used pattern-centered analyses (e.g., cluster analysis) to

differentiate among summer learning offerings by identifying subgroups of instructional responsiveness

profiles (i.e., quality). The three Summer Learning PQA scale scores were used as input variables to

identify instructional responsiveness profiles; that is, subgroups of teachers whose instructional practices

were similar. Additional detail is presented in Appendix A.

Next, we used linear models to estimate both academic skill change, and differences in academic

skill change, across high, medium, and low instructional responsiveness profile subgroups. In one set of

models, we sought to maximize the rigor of inference in relation to the threat of model misspecification

and type II error.

We used the entire sample (grades K-4) to fit analysis of covariance (i.e., ANCOVA)

models that compare rates of individual student growth (e.g., pre-to-post change) across the instructional

responsiveness profile subgroups. These models allowed us to test a set of theoretically-driven planned-

contrasts that improved both model specification and sample sizes within the subgroups being compared.

Technical presentation of methodology and detailed results are presented in Appendix A.

In a second set of multilevel models, we sought to maximize the rigor of inference in relation to

the threat of selection bias by using multilevel models and propensity matching based on a set of relevant

covariates (e.g., state achievement test scores for the prior year). However, inclusion of the covariates

limited the sample to grades 3 and 4, reducing the sample of settings to 31 and increasing potential for

type II error. Technical presentation of methodology and detailed results are presented in Appendix B,

authored by Albright (2017).

Finally, we used each of the modeling approaches to compare the rate of individual growth for

students who entered the program at lower levels of academic skill (i.e., academic risk) across the quality

subgroups. In addition to the three-step approach, we also conducted attrition analyses to see if students

missing at T2 were different from the rest of the sample on the T1 measures. We found no statistically

significant differences in 10 of the 15 tests conducted. See Appendix A for further discussion.

This “skill growth by levels of quality” design has been used with some frequency in early childhood evaluations

(e.g., Karoly, 2014; Thornburg, Mayfield, Hawks, & Fuger, 2009).

A type II error is to falsely conclude that the null hypothesis is true (e.g., to conclude that the effect does not exist),

which can occur when an analysis does not have enough statistical power to detect the effect.

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 11

Results

Attendance

The program was well attended, with most students (i.e., 77%) attending between 21 and 27 days

of programming. Figure 2 describes the attendance of students.

Figure 2. Attendance in Seattle Public Schools Summer Programs, 2016

Source: Teacher Report

Academic Skill Change

Students in the study were assessed on literacy and math skills at the start and end of the summer

program using the five academic performance measures. Using simple subtraction of raw pre-scores from

post-scores, a majority of students demonstrated positive skill growth during the summer program period

on all five academic performance measures. Similar percentages of students improved on academic

achievement tests. Specifically:

 81% of students showed improvement on Site Words, with a standardized pre-to-post effect size

of .59 during the summer session.

 76% of student showed improvement on Oral Fluency, with a standardized pre-to-post effect size

of .73 during the summer session.

 64% of students showed improvement on Math Assessment, with a standardized pre-to-post effect

size of .59 during the summer session.

Cohen’s dz can be used to understand the size of the effect from Time 1 to Time 2 using a quantitative estimate that

does not depend on sample size and that can be compared across variables or study samples. Cohen’s dz can be

described as the “the standardized mean difference effect size for within-subjects designs” (Lakens, 2013, p. 4). It

differs from the standard Cohen’s d in that it is based on the average difference score between Time 1 and Time 2

divided by the standard deviation of the difference values, whereas the standard Cohen’s d is the difference in group

means divided by the pooled standard deviations of the two groups. Cohen’s dz also corrects for the auto-correlation

between the Time 1 and Time 2 scores.

100

200

300

400

500

1-5 days attended 6-10 days

attended 11-15 days

attended 16-20 days

attended 21-25 days

attended 26-27 days

attended

Number of Students

Days Attended

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 12

 71% of students showed improvement on Math Content, with a standardized pre-to-post effect

size of .95 during the summer session.

 66% of students showed improvement on Math Practice, with a standardized pre-to-post effect

size of .95 during the summer session.

 76% of students improved on State Math Achievement Test from the 2015-16 school year to the

2016-17 school year, with a standardized pre-to-post effect size of .61 between annual

assessments.

 73% of students improved on State Literacy Achievement Test from the 2015-16 school year to

the 2016-17 school year, with a standardized pre-to-post effect size of .56 between annual

assessments.

Profiles of Instructional Responsiveness

Three profiles of instructional responsiveness were identified using data from the Summer

Learning PQA scores for Supportive Environment, Interaction, and Engagement. These three scores for

each of the 60 offerings were subjected to pattern-centered analysis to identify relatively-homogeneous

subgroups of programs characterized by similar profiles of instructional responsiveness (see Figure 3).

In general, scores for this sample of programs were all quite high. For reference, Figure 4

presents the “low” quality instructional profile from Figure 3 (n = 9) with the low profile from a different

two-city sample (n = 21). Although this is good news for SPS summer programs, it also suggests that our

ability to detect statistically reliable differences in rates of skill learning by profile may be constrained

due to the absence of low-quality offerings.

Figure 3. Instructional Responsiveness Profiles

Source: Summer Learning Program Quality Assessment, 2016 (N = 60 program offerings)

4.65 4.39 3.97

4.10 3.75

3.15

4.40

3.58 3.83

1.00

2.00

3.00

4.00

5.00

High (n=16) Medium (n=35) Low (n=9)

Mean

Cluster Profile

Support Interaction Engagement

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 13

Figure 4. “Low” Instructional Responsiveness Profiles for SPS and Other Cities’ Programs

Source: Summer Learning Program Quality Assessment, 2016 (N = 60 program offerings); Low

Comparison profile excerpted from Figure 4 in Smith et al., 2017.

Academic Skill Gains by Profiles of Instructional Responsiveness

This section presents gain scores on the five measures of academic performance and the two

measures of academic achievement for students exposed to one of the profiles of instructional

responsiveness. Figures 5-9 present simple gain scores (post-score minus pre-score) for academic

performance of students in the program offerings identified by each of the three profiles of instructional

responsiveness described Figure 3. Table 1 presents the percentage of students moving from below

proficient to proficient or higher on the two academic achievement tests.

Literacy Academic Performance Change. Figures 5 and 6 show the gain scores for Sight Words

and Oral Fluency assessments, each separated out by subgroups of students exposed to one of the three

profiles of instructional quality. In each case, the expected relation between instructional quality and

gains in academic performance is demonstrated: Gains are larger in the higher-quality offerings.

3.55

3.97

2.72

3.15

2.44

3.83

1.00

2.00

3.00

4.00

5.00

Low Comparison (n=21) Low Seattle Public School (n=9)

Mean

Cluster Profile

Support Interaction Engagement

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 14

Figure 5. Sight Words Gains by Profiles of Instructional Quality

Figure 6. Oral Fluency Gains by Profiles of Instructional Quality

Math Academic Performance Change. Figures 7, 8, and 9 show subgroups of students in each of

the three instructional quality profiles. In each case, the expected relationship is demonstrated: Gains are

larger in the higher-quality offerings.

31.33

22.03 21.55

1.00

6.00

11.00

16.00

21.00

26.00

31.00

Sight Words Gains

Gain Score

High Medium Low

15.31

19.63

7.91

1.00

6.00

11.00

16.00

21.00

26.00

31.00

Oral Fluency Gains

Gain Score

High Medium Low

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 15

Figure 7. Math Assessment Gain Scores by Profiles of Instructional Quality

Figure 8. Math Content Gain Scores by Profiles of Instructional Quality

Figure 9. Math Practice Gain Scores by Profiles of Instructional Quality

Academic Achievement Change. Table 1 shows the number of SPS summer program students

who moved from below proficient in the 2015 school year to proficient or better in the 2016 school year

for both math and literacy achievement. In this case, the greatest gains are demonstrated by students

exposed to the lowest-quality instructional profile.

1.47

1.20 1.11

0.00

0.50

1.00

1.50

2.00

Math Assessment Gains

Gain Score

High Medium Low

0.93

0.63

0.46

0.00

0.20

0.40

0.60

0.80

1.00

Math Content Gains

Gain Score

High Medium Low

0.81

0.50 0.39

0.00

0.20

0.40

0.60

0.80

1.00

Math Practice Gains

Gain Score

High Medium Low

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 16

Table 1. Percent of Students Changing from Non-Proficient to Proficient in Math and Literacy

Achievement by Profiles of Instructional Quality.

High

Medium

Low

Math Achievement

15%

Literacy Achievement

13%

15%

Source: Seattle Public Schools

Multivariate Models for Grades K-4 (N = 1,069). We tested a series of statistical models that

allowed us to specify when students exposed to the medium profile should be grouped with students in

the higher or lower profiles (see Appendix A). In these models, statistically significant and positive

relationships between exposure to the higher instructional responsiveness and academic skill gains were

present for all five academic performance measures. No statistically significant relationship was detected

for the two academic achievement measures.

Multivariate Models for Grades 3-4 (N = 600). We also tested multilevel statistical models

designed to compare gains in student academic performance and academic achievement in the high versus

low instructional quality profiles (see Appendix B). These models included two powerful covariates (i.e.,

the 2015-16 State Math Achievement Test and the 2015-16 State Literacy Achievement Test) in addition

to the other six covariates listed in Table A-1. These covariates were used to construct propensity weights

for matching students in the high and low instructional quality subgroups, thereby providing the most

rigorous test of the association between exposure to high instructional responsiveness and academic skill

gains. Overall, these models failed to detect any statistically significant relationship between exposure to

higher levels of instructional responsiveness and greater academic skill gains.

Academically At-Risk Students

Finally, for all seven academic skill measures, we tested models that included only academically

at-risk students, where academic risk was defined as receiving a below proficient score on the prior year’s

achievement test. The same pattern of mixed findings was demonstrated. In the more theoretically driven

models with grades K-4, academically at-risk students showed greater gains on most academic

performance measures where exposed to higher versus lower profiles of instructional responsiveness (see

Appendix A). In the more rigorous, propensity-weighted models for grades 3 and 4, no statistically

significant differences were detected; that is, more academically at-risk students performed similarly

where exposed to higher versus lower profiles of instructional responsiveness (see Appendix B).

Quality-Outcomes Study for Seattle Public Schools Summer Programs, 2016 Program Cycle 17

Discussion and Recommendations

This quality-outcomes study was designed to both (a) describe performance in Seattle Public

Schools summer learning programs in ways that would be useful to staff and (b) provide evaluative

evidence (i.e., validity) for an instructional model that includes challenging academic content and

responsive instructional practices. In the study, a summer academic curriculum including both

challenging content and responsive practices was implemented with elementary-age children. Three

setting measures of responsive staff instructional practice and seven mental measures for student

academic skill were collected at two time points: five academic performance measures administered near

the beginning and end of the summer program and two academic achievement tests taken in the prior

(2015-16) and subsequent (2016-17) school years. A series of statistical models were developed to

address research questions related to the association between responsive instructional practices and

development of academic skills.