Content uploaded by Jerlando F. L. Jackson
Author content
All content in this area was uploaded by Jerlando F. L. Jackson on Dec 06, 2017
Content may be subject to copyright.
External Evaluation Report
2
ACKNOWLEDGEMENTS
The School District of the City of Saginaw’s Centric Program, along with the authors would like
to acknowledge the in-kind contributions of Wisconsin’s Equity and Inclusion Laboratory (Wei
LAB) at the University of Wisconsin-Madison to the production of this report.
On behalf of the School District of the City of Saginaw, the authors of this report respectfully
acknowledge individuals who contributed information presented in this report: Terry Pruitt,
Director, SPSD Centric Grant; Edmund Boell, SPSD Centric Grant; Edwin, SPSD Centric Grant;
and Benjamin Toff, Editorial Associate, Wei LAB.
Recommended Citation: Jackson, J. F. L., Charleston, L. J., Kam, J., & Niu, Y. (2015).
School District of the City of Saginaw’s Centric Program: External Evaluation Report.
Saginaw, MI: School District of the City of Saginaw.
Cover and page layout and design by: Jerlando F. L. Jackson and LaVar J. Charleston
Photos: Bob Johnson, The Saginaw News
© School District of the City of Saginaw
3
SCHOOL DISTRICT OF THE CITY OF SAGINAW’S CENTRIC
PROGRAM: EXTERNAL EVALUATION REPORT
EXECUTIVE SUMMARY
The School District of the City of Saginaw’s Centric Program is a comprehensive
intervention designed to close the achievement gap for male students in middle school and high
school, especially for those who are ethnic minorities. An external evaluation team analyzed the
impact of the Centric program using data gathered from 5 affiliated institutions, including Arthur
Hill High School, Saginaw High School, Thompson Middle School, Saginaw Arts and Science
Academy and Ziwaukee K-8 School. This evaluation covers the following topics related to the
Centric program: (1) Financial Incentives; (2) Behavioral Health; (3) Winning Futures; (4)
Kaplan ACT Prep; (5) Parent Engagement Survey; (6) Parental Workshop; (7) Summary of Non-
Participant Data.
Participants
From 2014 to 2015, the Centric Program served a population of 193 student participants.
African Americans accounted for 74.61% of these students while Hispanic and White students
each accounted for 11.92% of the total. The number of participants in each of the program’s sub-
components varied, with the Kaplan ACT Prep and Behavioral Health Program serving the
largest number of students (n=183) and for (n=186) and the Winning Futures Program serving
the smallest (n1=55, n2=25). Interventions for parents generally included fewer participants. The
number of participants for the Parent Engagement survey for the first semester and second
semester were 39 and 83, respectively. Far fewer parents attended the parental workshop, with
total number ranging from 3 to 20. Not all affiliated institutions completed all components of the
Centric program. There were more participants from Arthur Hill High School, Saginaw High
School and Thompson Middle School than the other two institutions.
Key Findings
The evaluation produced few statistically significant results to prove the effectiveness
of the Centric program on students’ academic outcomes. These results may be due to
imbalanced/small sample sizes, insufficient implementation of the experimental design,
and measurement error.
4
Offering financial incentives to students who participated in the program and
demonstrated academic improvement might be an effective strategy for motivating
students. Financial incentives awarded to students were positively correlated with GPA,
and negatively correlated with rates of absence and missing assignment, controlling for
all other variables.
Almost all parents who attended parental workshops rated these workshops as highly
effective. More parents should be recruited and encouraged to participate in these
workshops, which help to facilitate better communication between parents, children,
and teachers in order to create better study environments for students.
Future Practices
Interventions or workshops should be developed to help students manage emotional
challenges. Behavioral Health evaluations revealed that students who reported fewer
experiences of irritability and anger tended to perform better on reading tests.
The Centric program should recognize the negative influence of students’ absences and
tardiness on academic achievement and devise ways to reduce such occurrences.
The recruitment process should be altered to encourage more students and parents to
attend the program.
To better demonstrate and assess the impact of the program, the program
implementation should closely follow the guidelines of experimental/quasi-
experimental design and employ appropriate measures.
5
TABLE OF CONTENTS
Introduction
6
Centric Student Database
17
Behavioral Health Program
21
Winning Futures Program
31
Kaplan ACT Prep Program
40
Parental Engagement Survey
46
Parental Workshops
55
Non-participants
68
Summary and Implications
70
References
73
6
SCHOOL DISTRICT OF THE CITY OF SAGINAW’S CENTRIC
PROGRAM: EXTERNAL EVALUATION REPORT
1. Introduction
Information contained in this final report pertains to the Centric grant initiative of the
School District of the City of Saginaw. For the 2014-15 school year, the school district was
awarded a $2.472 million Michigan Department of Education Section 22K (Centric) grant to
address ongoing academic performance issues experienced by male students in the district. The
aim of this report is to provide a formal evaluation of quantitative and qualitative data regarding
instructional and character development services and activities delivered as part of the Centric
program to address the persistent achievement gap for male students in the Saginaw Public
School District. Data analyzed in the compilation of this report were collected from a variety of
sources using survey instruments, standardized educational assessment tools, diagnostic
assessment tools, and other pertinent methods.
1.1. Overview of the School District of the City of Saginaw
The School District of the City of Saginaw is a K-12 urban school district located in the
middle of the state of Michigan approximately 100 miles north of Detroit along what is
commonly referred to as the I-75 corridor. Many of the urban communities along Interstate 75
including Detroit, Pontiac, Flint, and Saginaw are characterized as economically depressed
communities due to a declining manufacturing base, population loss, high unemployment and
unstable financial conditions. As identified in the district’s Section 22K grant application, 92%
of the district’s students qualify for free or reduced lunch. The Saginaw community has been
severely impacted over the past two decades by the closure of major manufacturing facilities
including four General Motors plants.
A more detailed demographic profile of Saginaw and the State of Michigan is presented
in Table 1.1 below.
7
Table 1.1. Saginaw, MI Community Demographic Profile
Saginaw
Michigan
Total Population (2013)
50,303
9,898,193
Population, % change (2010-2013)
-2.3%
0.1%
Males, %
48.1%
49.1%
Females, %
52.9%
50.9%
Persons under 18 years, % (2010)
28.4%
23.7%
Race
White, % (2010)
37.5%
76.6%
Black, % (2010)
46.1%
14.2%
Hispanic, % (2010)
14.3%
4.4%
Other, % (2010)
2.1%
4.8%
Economic & Education Variables
Total households (2013)
19,353
3,823,280
Homeownership Rate (2013)
60.8%
72.1%
Median household income (2013)
$27,701
$48,411
Individuals below poverty, % (2013)
37.4%
16.8%
High school graduates 25+ yrs., % (2013)
77.8%
88.9%
Bachelor’s degree & higher, % (2013)
11.5%
25.9%
Unemployment in the City of Saginaw has consistently lagged behind state levels with
rates typically double the unemployment rate for the State of Michigan. The June 2014 reported
rate for the City of Saginaw was 13.4% while the rate for the State of Michigan was 7.7%.
Furthermore, nearly 60% of city households were headed by females, and the poverty rate for
female-headed households with children under five years of age (2010 census data) is 62%.
Information provided by the 2013 Michigan Student Data System (MSDS) shows that 92% of
Saginaw Public School students K-12 are eligible for free and reduced lunch.
The economic conditions of Saginaw were significantly compounded by the 2007-2012
global financial crises, which created significant concerns for community residents. The decline
of manufacturing jobs resulted in considerable declines in property values in the city, which in
turn decreased the amount of money the school district and city government are able to collect
through property taxes to support services and programs.
In short, many of the important economic and social measures indicate that the Saginaw
community severely lags the rest of the state as well as the nation and these variables have
impacted the overall capacity and ability of students to succeed academically.
8
1.2. School Organization and Governance
The district’s legally designated name is “School District of the City of Saginaw.” In
accordance with the Michigan Constitution and state legislature, the district operates under the
general supervision and auspices of the state board of education and Michigan Department of
Education. The local district is “maintained, developed, and operated” by a locally elected board
of seven members who serve staggered terms. Local governance is guided by the state
constitution, laws, and administrative regulations. The district is a part of the Saginaw
Intermediate School District, and serves the cities of Saginaw, Buena Vista Township, and
Zilwaukee, the eastern part of Kochville Township.
The Buena Vista School District closed in 2013. The Saginaw School District acquired
the Buena Vista headquarters building, all five school buildings and the majority of the school
district territory. More than 100 former Buena Vista students began attending Saginaw schools.
For the 2014-15 school year, the Saginaw school district operated 10 elementary, one K-9, one
middle (6-8), and two high schools, as well as schools and centers for vocational, gifted and
talented, and alternative education. Figure 1.1 below depicts a five-year summary of the total
student population in the district. The declining K-12 student count is largely attributable to city
population loss, schools of choice enrollment, and competition from charter schools. The
district’s total student count for the 2010-11 school year was 8,357, falling to 6,880 for the 2014-
15 school year.
Figure 1.1. The School District of the City of Saginaw Enrollment (2010-2015)
Source: State of Michigan Department of Education.
9
The academic performance of students in the Saginaw School District has exhibited
mixed results over the past ten years (see Table 1.2 below). Thompson Middle School, Saginaw
High School, and Arthur Hill High School have consistently underperformed statewide
benchmarks in reading, math, science, social studies, ACT, and other state and national
assessments. These three schools were the primary focus of Centric services with limited
programming and participation from students from Zilwaukee School and Saginaw Arts &
Science Academy.
Table 1.2. Student Performance for the School District of the City of Saginaw (2013-2014)
Student Performance Outcomes
Attainment
Students proficient in reading at end of 3rd grade
48.3%
Student academic growth 3-8 grades
10.6%
Students Proficient in Math & Reading 3-8 grades
20.0%
Students Proficient on MME (in all subjects)
9.2%
District ACT composite score
16.8
The Michigan Department of Education annually publishes an analysis comparing the
racial and ethnic characteristics of the academic performance of students in the top 30% and
bottom 30% within their respective schools on state-administered standardized academic tests
(Table 1.3). These data delineate the disparities in academic performance of students targeted for
participation in the Centric program. African American and Hispanic students’ academic
performance significantly trailed their peers across subject areas.
Table 1.3. Race and Ethnic Differences by Subject Area (2013-2014)
School/Race
Reading
Mathematics
Science
Arthur Hill High School
Top 30 – Bottom 30
Top 30 – Bottom 30
Top 30 – Bottom 30
African American
26.2%
31.2%
23.8%
35.3%
24.2%
31.7%
Hispanic
28.2%
38.5%
38.5%
30.8%
28.2%
33.3%
White
52.2%
13.0%
<10%
47.8%
60.9%
17.4%
Saginaw High School
Top 30 – Bottom 30
Top 30 – Bottom 30
Top 30 – Bottom 30
African American
36.9%
30.1%
30.1%
20.2%
31.0%
29.2%
Hispanic
<10.0%
<10.0%
33.0%
33.0%
33.0%
33.0%
White
--
--
--
--
--
--
Thompson High School
Top 30 – Bottom 30
Top 30 – Bottom 30
Top 30 – Bottom 30
African American
26.2%
31.2%
23.8%
35.3%
24.2%
31.7%
Hispanic
28.2%
38.5%
38.5%
30.8%
28.2%
33.3%
White
52.2%
13.0%
<10%
47.8%
60.9%
17.4%
10
Further examination of gender differences in the top 30%-bottom 30% (not reported
above) reveal that female students consistently outperformed male students across all content
areas with the exception of math where the performance was at parity.
1. 3. Overview of Saginaw Public School Centric Program
The Saginaw Public School District Centric Program implementation incorporated the
principles of Universal Design for Learning (UDL) in support of the instructional model for
delivering the defined curriculum to district students. The National Center on UDL (n.d.),
describes the UDL approach as a set of principles for curriculum development and
implementation that gives all students equal opportunities to learn. It provides a blueprint for
creating instructional goals, methods, materials, and assessments that work for everyone—not a
single, one-size-fits-all solution but rather flexible approaches that can be customized and
adjusted to target interventions to address specific individual student needs. The district’s
approach to academic decision-making has been consistently focused on the implementation of
curriculum content, ongoing assessment, and analysis and tactical intervention. The relationship
between these concerns is depicted in Figure 1.2 below:
Figure 1.2. Saginaw Public School District Centric Program Implementation Principles
The UDL approach affords students multiple means of representation and ways of
receiving information and content. The district’s implementation of the Centric program was
intended to exemplify this instructional model. The Centric implementation design consisted of
several steps including defining the cohort (experimental group) and control populations,
11
developing quantitative and qualitative assessments tools, and implementation of instructional
interventions and activities aimed at achieving positive measurable impact in a short span of
time. Finally, the last step involved quantitative and qualitative data analysis evaluating progress
toward the outcome targets.
Overall, the intent was to provide a comprehensive array of services and activities aimed
at impacting as many of the district’s targeted subgroup (male students) as possible. The Centric
implementation team recognized that the scope of the effort needed to be narrowed in order to
demonstrate measurable and meaningful outcomes by the end of one school year. Thus, a cohort
(or experimental group) of male students was selected for extensive academic and character-
related services as a central focus of the grant during the fall of 2014 (See Table 1.4).
Table 1.4. Number of SPSD Male Students by Grades Level
Grade Level
Students (N)
Grade 6-7
506
Grade 8-9
538
Grade 10-12
518
Total Count
1,562
1.4. The Cohort Profile and Program Administration
Staffing the Centric program required five math and science learning coaches (1 each at
the District’s two high schools and one for the middle school and K-8 population). The
implementation design called for 40 male students per coach per academic day for in-depth
supplemental support from each of the coaches. Thus, each coach provided direct supplemental
support to 5-10 students per class period. In addition, the program also involved parent outreach
specialists, research data analysts, and adult mentors. The organizational chart is summarized in
Figure 1.3 below, along with total staffing levels.
12
Figure 1.3. Organizational Chart Depicting the Program Structure and Staffing
Students were recruited, screened, and selected on a competitive basis and by
recommendation from core teachers, building administrators, and counseling staff. The cohort
initially consisted of 200 students across the seven grade levels with 50% of the group consisting
of 8th and 9th grade male students. Additional screening parameters for the cohort included:
(a) Academic performance below grade-level on students’ 2013-2014 Michigan Educational
Assessment Program (MEAP) exams and other assessment data during the previous 12-
month period (at level 3 and or high 4)
(b) Overall GPA range between 1.7 and 3.0 (science & math grades of “C” or “D” during
the previous 12-month period
(c) Student attendance at a minimum of 90% during the previous 12-month period
(d) No more than 5 suspensions or discipline referrals (individual occurrences) within the
previous 12-month period
The cohort group received intensive supplemental academic support from the Centric
instructional coaches (certified/highly qualified teachers) during the school day. A second tier of
13
supplemental support was provided as part of an extended day program that offered targeted
instructional assistance directly aligned with five STEM-related career exploration/focus areas
that emphasized science, math, reading, and life skills. These extended day activities included
experience with instructors and facilitators trained in the various STEM-related emphasis areas.
The extended day programs offered to students in the cohort group are summarized below.
HIRE
PROGRAM
(Helping
Individuals
Reach
Excellence)
100
students
Students participated in a 16-week college and career readiness
program geared toward exposing students to STEM-based careers
while simultaneously ensuring that they have the basic academic
skills to pursue those careers. Students studied and mastered a
mathematically-based core craft construction curriculum created
by the National Center for Construction and Educational Research.
Students learned about careers in the construction industry,
interacted with high-level industry leaders, received life and
leadership training, developed improved written and oral
communication skills by completing weekly writing assignments,
participated in field trips, and had an opportunity to earn incentive
based stipends, up to $599.00. This program was available to
students at Saginaw High School and Arthur Hill High School
with 50 students from each school divided into two classes of 25
students at each of the high schools.
D.I.C.E.
(Police & Fire
Diversity
Initiative
Community
Explorers)
20
students
Students in grades 10-12 participated in a 16-week career
exploration program offering an introduction to careers in public
safety. The program exposed students to over 30 topic areas
relating to public safety including traffic control and radar
operations, CSI and photography, criminal investigation, as well as
incorporated leadership and life skills. The aim of the program was
to improve students’ science, math, written, and oral
communication skills while providing students with training and
field trip experiences directly related to the public safety arena.
Classroom to
Workplace
20
students
The district partnered with the Greater Michigan Construction
Academy & Delta College to provide instructional and training
opportunities for career growth in the construction industry
through a job skills and life skills curriculum that served as a direct
pipeline into the construction industry. The program/curriculum
was designed to put students on a pathway to formal certification
and apprenticeships that increase employability skills. The
curriculum focused on students’ weaknesses in math, reading,
communications, soft skills, and other issues that may represent
barriers to employment.
14
Skilled Trades,
Technology
and
Manufacturing
20
students
This program was centered around a partnership with Delta
College, Nexteer Automotive, and Dow Chemical Company to
provide an extensive 16-week program that exposed students to
careers available in the area of skilled trades. The course explored
education in the technical, trades, and manufacturing disciplines.
The course included hands-on projects in each of the trade
disciplines. Much of the instruction was conducted at the District’s
Career Complex and Delta College for added hands-on skilled
trade lab assignments.
SOS
(Saving Our
Students)
20
students
This program was modeled on the awarding-winning Winning
Futures program, which achieved a high level of success in
providing students with the resources to prepare for their futures.
The program was designed to address barriers that many at-risk
students encounter. The effort focused on personal development,
academic success, and professional development with goals
directed at increasing attendance, improving grades, credit
recovery, and a specific focus on reading, math, and science
improvement.
Chemical
Processing
20
students
SPSD partnered with Dow Chemical Company, Dow Corning
Corporation, and Delta College to provide instructional and
training opportunities for career growth in the chemical industry
through a job skills and life skills curriculum that serves as a direct
pipeline into careers in the chemical industry. The
program/curriculum attempted to put students on a pathway to
formal certification leading to employment in chemical processing.
The curriculum focused on addressing students’ weaknesses in
math, science, reading, communications, and other skills needed to
attain employment.
The final component of the Saginaw implementation model included several incentives
made available to students in the cohort who successfully completed major milestones
throughout the academic year and completed the entire program. In addition to the incentives
that the Centric grant provided to students, the district collaborated with several regional
community partners. Notable incentives provided to students included:
• iPads for students demonstrating outstanding academic performance gains during the
year. (Funded by Centric)
• Stipends of up to $599.00 per student. The district partnered with area foundations,
corporate sponsors, and others to support these financial incentives.
15
• A year-end trip to Washington, D.C. to visit the White House, King Monument, etc.
(Funded by Centric)
Extensive in-school and extended-day support was also made available to students in the
following areas:
• Instructional intervention on a daily “Push-In” and “Pull-Out” basis from Centric
Learning Coaches
• Centric Learning Coaches worked with the District’s Instructional Coaches to define
specific intervention tactics designed to achieve high impact in short time frames
• Regular one-on-one and group mentoring support
• Career assessment and exploration support
• Kaplan ACT preparation and test-taking support services
• College/University campus visits and field trips
The primary common assessment tool used to measure and monitor academic progress
for Centric students was PLATO Accucess. At the onset, this assessment helped to define a
baseline achievement gap for each student and served to guide tailored instructional intervention
strategies. Data collection using this assessment tool enabled the district to evaluate measurable
progress toward targeted outcomes on an ongoing basis throughout the year.
1.5. The Control Group
For comparison purposes, a control group was selected consisting of 150-200 male
students within the subgroup universe. These students were of a similar academic/behavioral
profile as students in the defined cohort. Control group students received core curriculum
instruction and participated in limited direct Centric programming during the year including:
• Instructional intervention on a “Push-In” basis from Centric Learning Coaches
• Group mentoring support (intermittently)
• Career Fairs/Exposure experiences (first-come basis only)
• Kaplan ACT preparation & test-taking support services (first-come basis only)
• Field trips (first-come basis only)
Other male students were able to participate in group-oriented Centric activities on a
limited first-come basis including career fairs, group mentoring sessions, etc. A summary of the
various program activities is depicted in Figure 1.4 below.
16
Figure 1.4. Summary of Centric Program Activities
Participation in the Michigan Department of Education Section 22K Centric grant
represented a unique opportunity for students, staff, and community stakeholders. It is our belief
that even though this initiative transpired over the course of just one year, it will have lasting
impact for years to come for many of the male students who participated. In addition, it
represents a foundation for future sustainability efforts to be undertaken by the school district
and the Saginaw community to address the ongoing achievement gap for male students of color.
The remainder of this report details the specific outcomes of the program activities and
assesses the overall accomplishments of the School District of the City of Saginaw Centric
Program.
Cohort
6-7 Grade
8-9 Grade
10-12 Grade
Control
6-7 Grade
8-9 Grade
10-12 Grade
Other
6-7 Grade
8-9 Grade
10-12 Grade
STEM Instruction
Evaluation
Mentoring
Career/College
Kaplan /ACT
Field/Campus Trips
Parental Support
Incentives
√ √ √ √ √ √
√ √
√ √ √ √ √ √ √ √
√ √ √ √ √ √ √ √
√ √
√
√
√ √ √ √
√
√ √ √
√
√
√
√
√
√
√
√
√
Centric Services
17
2. CENTRIC STUDENT DATABASE
The Centric Student Database is a comprehensive dataset, which includes student
academic performance, attendance, disciplinary actions, and the accumulated amount of financial
incentives earned for all participants in the program. All students were eligible the same total
amount of participation incentives but the final amounts received varied according to students’
attendance records. Data were collected in two stages: before and after the program intervention.
In the first part of this section, we summarize the descriptive statistics of the final samples of
participants in the Centric Program. In the second part, we estimate the relationship between
financial incentives and students’ performance, measured in a variety of ways including
ACCUCESS
1
mathematics and reading scores, Grade Point Average (GPA), rates of missing
assignments, rates of absence, tardiness, and number of disciplinary occurrences. We analyze
how effects differ according to the size of the financial incentives.
As shown in Table 2.1, the program served a total of 193 participants from five affiliated
institutions, including Arthur Hill High School (31.09%), Saginaw High School (27.98%),
Thompson Middle School (15.03%), Saginaw Arts and Science Academy (20.21%), and
Zilwaukee K-8 School (5.70%). African Americans/Blacks accounted for 74.61% of the students
while both Hispanics/Latinos and Caucasians/Whites accounted for 11.92%. Participants were
not equally distributed across grade levels, with very few participants in 11th (3.63%) grade.
Participants earned an average of $271.99 in incentives for participating in the components of the
program. Prior to the program intervention, students averaged 689.54 for mathematics and
787.10 for reading on the ACCUCESS test, and earned an average GPA of 2.26. Test scores and
GPAs declined in the second semester as reported in column (2) of Table 2.1. The average
number of absences, however, decreased 1.98 as a result of the program intervention (from 5.29
to 3.31), although the average amount of time students were tardy increased 2.14 (from 16.52 to
14.38 minutes). The number of disciplinary occurrences and missing assignments at the mean
level remained fairly even with participants receiving approximately 1.8 disciplinary occurrences
and missing around 43 assignments before and after the program intervention.
1
ACCUCESS, a diagnostic assessment, measures students’ grade level in mathematics, reading, and writing skills.
18
Table 2.1. Summary Statistics for Centric Program
Pre-survey
Post-survey
(1)
(2)
Race/ethnicity (%):
- Hispanic
11.92
11.92
- Asian
1.55
1.55
- Black
74.61
74.61
- White
11.92
11.92
Grade (%):
- 6th
15.54
15.54
- 7th
11.92
11.92
- 8th
18.13
18.13
- 9th
16.58
16.58
- 10th
13.47
13.47
- 11th
3.63
3.63
- 12th
Affiliated institution (%):
- Arthur Hill High School
31.09
31.09
- Saginaw High School
27.98
27.98
- Thompson Middle School
15.03
15.03
- Saginaw Arts & Science
Academy
20.21
20.21
- Zilwaukee K-8 school
5.70
5.70
Financial incentive:
- Money earned
271.99
271.99
Academic performance:
- ACCUESS math
689.54
683.88
- ACCUESS reading
787.10
711.18
- GPA
2.26
2.034
Attendance:
- Absences
5.29
3.31
- Amount of time tardy
14.38
16.52
Disciplinary Action:
- Missing assignments
42.51
43.28
- Discipline occurrence
1.84
1.70
Number of observations
193
193
19
Pooling observations across the sample from both pre- and post- datasets, we
systematically analyzed the relationship between financial incentives and key outcome variables
of academic performance, attendance, and the number of disciplinary actions as functions of
involvement in the program intervention. We estimated ordinary least squares (OLS) linear
regression models of the form below (1):
(1)
where is the key outcomes of a participant i; is an indicator variable for the time periods
(coded 0 for pre-intervention and 1 for post-intervention); is the logarithm of
accumulated financial incentives a participant i earned; is the interaction term
between the intervention indicator variable and the logarithm of accumulated financial
incentives; is measured in years;
is the square of the age variable to account for non-
linear effects of age, is the student’s grade-level, is a set of indicator variables for
each of the five educational institutions, and is a disturbance. The two parameters of interest
concern the interaction between financial incentives and the program intervention, or ()
and . The key outcomes modeled for each participant i were academic
performance (the logarithm of ACCUESS mathematics, the logarithm of ACCUESS
mathematics reading score, and GPA), attendance (the number of absences and the logarithm of
minutes tardy), and other behavioral issues (the logarithm of the number of missing assignments
and the number of disciplinary occurrences). All missing data was imputed by taking the average
values during estimation. Standard errors were clustered at the institutional level.
Table 2.2 summarizes estimates of coefficients and standard errors for the variables of
interest. Overall, holding all the other explanatory variables constant, financial incentives are
shown to have a significant positive relationship with student academic performance such as
ACCUCESS mathematics and reading scores as well as GPA. Incentives were also negatively
related to the number of absences, length of time students were tardy, number of missing
assignments, and disciplinary occurrences at the 1-10 percent significance levels. On the other
hand, the apparent findings of a negative relationship between the program interventions could
be due to factors that are not accounted for in the models, such as variations in difficulty or
20
grading between pre- and post-exams. In order to avoid misinterpreting coefficient estimates, we
focus on a parameter, . The negative effects of program intervention on student academic
performance, attendance, and disciplinary action are smaller among participants earning larger
amounts of financial incentives. For example, a $1 increase in financial incentive, the program
intervention is predicted to decrease GPA by 0.69 points (-0.694+0.454/100+0.087/100) and to
an increase in class absences of 1.26 (1.267-0.242/100-0.236/100). But a $100 increase in
financial incentives is predicted to only decrease GPA by 0.15 points (-0.694+0.454+0.087) and
an increase in absences of only 0.79(1.267-0.242-0.236). The findings suggest the financial
incentives can enhance the effectiveness of the program intervention.
Table 2.2. Effects of Financial Incentive on Academic Performance, Attendance, and
Disciplinary Action
Academic performance
Attendance
Disciplinary action
Math
Reading
GPA
Absence
Tardy
Assign.
missing
Discipline
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Race/ethnicity:
- Asian
0.173*
0.079
0.268
-0.656+
-0.460
-0.230
-0.351
(0.055)
(0.078)
(0.330)
(0.287)
(0.657)
(0.199)
(0.330)
- Black
-0.066
-0.028
-0.348
0.168
1.607
0.661*
0.928*
(0.064)
(0.064)
(0.201)
(0.167)
(0.898)
(0.159)
(0.212)
- White
0.038
0.058
-0.181
0.320
1.734
-0.312+
0.119
(0.068)
(0.087)
(0.211)
(0.303)
(0.910)
(0.122)
(0.290)
Post
-0.013
0.031
-0.694**
1.267*
-6.937
-0.018
0.790
(0.364)
(0.274)
(0.099)
(0.322)
(4.228)
(0.768)
(0.571)
Money earned
0.063+
0.073+
0.454**
-0.242**
-2.894+
-0.436*
-0.461
(0.025)
(0.030)
(0.083)
(0.046)
(1.090)
(0.134)
(0.508)
Post×Money earned
-0.002
-0.027
0.087*
-0.236*
0.911
0.015
-0.172
(0.060)
(0.045)
(0.029)
(0.067)
(0.778)
(0.149)
(0.126)
Number of obs.
386
386
386
386
386
386
386
R-squared
0.245
0.214
0.223
0.459
0.176
0.266
0.133
Notes: OLS model. Other controls include indicators for grade level and institution. The standard errors clustered at
the institutional level are reported in parentheses. The asterisks +, *, **, and *** indicate statistical significance at
the 10%, 5%, 1%, 0.1% levels, respectively.
21
3. BEHAVIORAL HEALTH PROGRAM
The Behavioral Health Program is an extended school effort that aims to identify and
improve students’ behavioral health. As part of this program, students participated in pre- and
post-intervention surveys that tracked participants’ responses on a set of psychological and
behavioral questions described below. These responses were merged with students’ demographic
and educational status (e.g., affiliated institution and grade-level) information from the Centric
Student Database reported in Section 2. Using this merged dataset, two main quantitative
analyses were conducted to estimate the effects of the Behavioral Health Program on the key
outcome variables such as student academic performance, attendance, and disciplinary action. In
the first part of this section, the way we generate variables of interest (i.e. behavioral health) for
reliable statistical analyses is reported. Next, we analyze the summary statistics of the merged
data. Lastly, the mixed effects of individual behavioral health and program intervention on
ACCUCESS mathematics and reading scores, GPA, missing assignments, absences, tardy
periods, and discipline occurrences are investigated.
In both pre- and post- surveys, program participants responded to several binary
questions (Yes/No) regarding their behavioral health status. Using the MAYSI2 scale
2
, we
categorized questions into seven groups: alcohol/drug use (AD), angry-irritable feelings (AI),
depressed-anxious feelings (DA), somatic complaints (SC), suicide ideation (SI), thought
disturbance (TD), and traumatic experiences (TE). In order to test the reliability of combining the
binary measures, Kuder-Richardson 20 coefficients were calculated. The Kuder-Richardson 20
coefficients for each MAYSI2 scale are reported in Table 3.1. Almost all MAYSI2 scales are
shown to have modest internal consistency at r = 0.61-0.74. In the analysis of this section, we
disregard information on traumatic experiences (TE) because its Kuder-Richardson 20
coefficient is 0.52, which is not enough large to confirm the item reliability (<0.6). Based on the
reliability results, the key endogenous variables of behavioral health are finally generated by
summing each of the dichotomous variables across the items that compose the MAYSI2 scale.
2
MAYSI-2 scale, developed by Thomas Grisso and Richard Barnum, measures behavioral, clinical and offense
characteristics of youths 12 to 17 years old.
22
Table 3.1. Behavioral Health Field Categories
MAYSI2 Scale
Descriptions
Questions on Scale
KR20*
Alcohol/Drug Use
- Frequent use of
alcohol/drugs
- Risk of substance
use disorders or
Psychological
reaction to lack of
access to
substances
10. Have you done anything you wish you hadn’t, when you were drunk or
high?
19. Have your parents or friends thought you drink too much?
23. Have you gotten in trouble when you’ve been high or have been
drinking?
24. If yes [to #23], has the trouble been fighting?
33. Have you used alcohol or drugs to help you feel better?
37. Have you been drunk or high at school?
40. Have you used alcohol and drugs at the same time?
45. Have you been so drunk or high that you couldn’t remember what
happened?
0.74
AngryIrritable
- Experiences
frustration, lasting
anger, moodiness
- Risk of angry
reaction, fighting,
aggressive
behavior
2. Have you lost your temper easily, or had a short fuse?
6. Have you been easily upset?
7. Have you thought a lot about getting back at someone you have bee angry
at?
8. Have you been really jumpy or hyper?
13. Have you had too many bad moods?
35. Have you felt angry a lot?
39. Have you gotten frustrated easily?
42. When you have been mad, have you stayed mad for a long time?
44. Have you hurt or broken something on purpose, just because you were
mad?
0.74
DepressedAnxious
- Experiences
depressed and
anxious feelings
- Risk of
impairments in
motivation, need
for treatment
3. Have nervous or worried feelings kept you from doing things you want to
do?
14. Have you had nightmares that are bad enough to make you afraid to go to
sleep?
17. Have you felt lonely too much of the time?
21. Has it seemed like some part of your body always hurts you?
34. Have you felt that you don’t have fun with your friends anymore?
35. Have you felt angry a lot?
41. Has it been hard for you to feel close to people outside your family?
0.74
23
MAYSI2 Scale
Descriptions
Questions on Scale
KR20*
47. Have you given up hope for your life?
51. Have you had a lot of bad thought or dreams about a bad or scary event
that happened to you?
Somatic Complaints
- Experiences bodily
discomforts
associated with
distress
- Risk of
psychological
distress not
otherwise evident
When you have felt nervous or anxious…
27. …have you felt shaky?
28. …has your heart beat very fast?
29. …have you felt short of breadth?
30. …have your hands felt clammy?
31. …has your stomach been upset?
43. Have you had bad headaches?
0.71
Suicide Ideation
- Thoughts and
intentions to harm
oneself
- Risk of suicide
attempts or
gestures
11. Have you wished you were dead?
16. Have you felt like life was not worth living?
18. Have you felt like hurting yourself?
22. Have you felt like killing yourself?
47. Have you given up hope for your life?
0.81
Thought
Disturbance
- (Boys Only)
Unusual beliefs
and perceptions
- Risk of thought
disorder
9. Have you seen things other people say are not really there?
20. Have you heard voices other people can’t hear?
25. Have other people been able to control your brain or your thoughts?
26. Have you had a bad feeling that things don’t seem real, like you’re in a
dream?
32. Have you been able to make other people to things just by thinking about
it?
0.61
Traumatic
Experiences
- Lifetime exposure
to traumatic events
(e.g., abuse, rape,
observed
violence).
- Questions refer
youth to ever in the
Girls
48. Have you EVER IN YOUR WHOLE LIFE had something very bad or
terrifying happen to you?
49. Have you ever been badly hurt, or been in danger of getting badly hurt or
killed?
50. Have you ever been raped, or been in danger of getting raped?
51. Have you had a lot of bad thoughts or dreams about a bad or scary event
that happened to you?
0.52
24
MAYSI2 Scale
Descriptions
Questions on Scale
KR20*
past not past few
months
- Risk of trauma-
related instability
in
emotion/perception
52. Have you ever seen someone severely injured or killed (in person – not in
movies or on TV)?
Boys
46. Have people talked about you when you’re not there?
48. Have you EVER IN YOUR WHOLE LIFE had something very bad or
terrifying happen to you?
49. Have you ever been badly hurt, or been in danger of getting badly hurt or
killed?
51. Have you had a lot of bad thoughts or dreams about a bad or scary event
that happened to you?
52. Have you ever seen someone severely injured or killed (in person – not in
movies or on TV)?
Notes: *Kuder-Richardson 20 coefficient. Based on the estimated Kuder-Richardson 20 coefficients, we generateed new variables by summing across groups
dichotomous variables.
Source: Adapted from Skowyra, K.R., & Cocozza, J.J. (n.d.) Mental health screening within juvenile justice: The next frontier. Appendix C: Texas MAYSI2
Protocol Reference Card, pp.2425. Delmar, NY: National Center for Mental Health and Juvenile Justice. Accessed from
http://www.ncmhjj.com/pdfs/MH_Screening.pdf.
25
Table 3.2 reports summary statistics for our key variables of interest regarding the
Behavioral Health Program. A total 97 students were included in the pre-intervention survey and
102 in the post-intervention survey. The pre-intervention sample is roughly split between
students at Arthur Hill High School (51.61%) and Saginaw High School (48.39%). The post-
intervention sample included slightly more students from Arthur Hill High School (54.08%) than
Saginaw High (45.92%). The sample is not equally distributed across grade-levels, with fewer
participants in 11th (less than 15% in both surveys) and 12th grades (approximately 5% in each
survey). African Americans/Blacks accounted for the vast majority of the program participants
(79.38%) while Hispanics/Latinos accounted for 7.22%and Caucasians/Whites accounted for
12.37%. After the program intervention, the average scores of MAYSI2 scales decreased by
6.25-79.69 percentage points whereas the average mathematics score on the ACCUCESS
assessment increased by 1.42 percentage points. These changes, however, cannot necessarily be
interpreted as direct effects of the program intervention because such changes could have been
caused by differing distributions of participants before and after the program intervention and/or
the heterogeneity of variance deviations between the first and second semester exams.
26
Table 3.2. Summary Statistics for the Behavioral Health Program
Pre-survey
Post-survey
(1)
(2)
Average age
15.26
15.26
Race/ethnicity (%):
- Hispanic
7.22
7.22
- Asian
1.03
1.03
- Black
79.38
79.38
- White
12.37
12.37
Grade (%):
- 8th
27.96
26.53
- 9th
25.81
28.57
- 10th
26.88
25.51
- 11th
13.98
14.29
- 12th
5.38
5.10
Affiliated institution (%):
- Arthur Hill high school
51.61
54.08
- Saginaw high school
48.39
45.92
Average behavioral health scores:
- Alcohol/drug use (AD)
0.52
0.12
- Angryirritable (AI)
3.57
1.30
- Depressedanxious (DA)
1.96
0.80
- Somatic complaints (SC)
2.57
1.18
- Suicide ideation (SI)
0.21
0.07
- Thought disturbance (TD)
0.64
0.13
- Traumatic experiences (TE)
0.96
0.90
Academic performance:
- ACCUESS math
656.88
666.22
- ACCUESS reading
767.16
725.60
- GPA
2.11
2.08
Attendance:
- Absences
6.45
2.77
- Amount of time tardy
14.95
16.95
Disciplinary Action:
- Missing assignments
47.36
50.54
- Discipline occurrence
1.52
1.54
Number of observations
97
102
27
Similar to Section 2, we estimated linear regression models in order to systematically
analyze the relationship between behavioral health variables (AD, AI, DA, SC, SI, TD, and TE)
and key outcome measures of academic performance, attendance, and disciplinary according to
the following OLS model (2).
(2)
where is the key outcomes of a participant i; is an indicator variable indicating two time
periods (coded 0 for pre-intervention and 1 for the post-intervention); represents students’
behavioral health scores on the six variables generated in the first part of this section (AD, AI,
DA, SC, SI, and TD); is the interaction term between the indicator for time period
and the behavioral health scores. All other variables are the same as in equation 1. Again, the
parameters of interest are the behavioral health scores () and the interaction () between these
scores and the indicator for the intervention period. The key outcomes modeled for each
participant i included academic performance (the logarithm of ACCUESS mathematics, the
logarithm of ACCUESS mathematics reading score, and GPA), attendance (the number of
absence and the logarithm of tardy periods), and other behavioral issues (the logarithm of the
number of missing assignments and the number of disciplinary occurrences). The six
components of were standardized to the same scale, with a mean of 0 and standard deviation
of 1. All missing data was imputed by taking the average values during the estimation; and the
standard errors are clustered at the institutional level.
Table 3.3 provides estimates of coefficients and their standard errors for the variables of
interest. Overall, for students with same level of behavioral health, the program intervention
indicator appears to be negatively related to academic performance, attendance, and disciplinary
action are observed. Behavioral health problems are also found to be negatively related to
student academic performance and attendance but positively related to rates of disciplinary
occurrences. But again, these negative coefficients cannot be directly interpreted because they
could be driven from variations in external factors such as different levels of difficulty between
exams pre- and post-intervention, as previously noted above.
28
In order to avoid misinterpreting coefficient estimates, we focus on the interaction
parameter . Those negative effects of program intervention on student academic performance,
attendance, and disciplinary action are larger among participants with heavier behavioral health
problems. For example, a one-unit increase in thought disturbance, the program intervention is
predicted to decrease GPA by 0.25 points (-0.072-0.370+0.190) and class absence by 3.35 (-
3.560-0.114+0.320) as reported in columns (3) and (4) of Table 3.3, respectively. But a one
hundred-unit increase in thought disturbance, the program intervention is predicted to decrease
GPA by 18.07 points (-0.072-0.370×100+0.190×100) and to increase class absence by 17.04 (-
3.560-0.114×100+0.320×100). The findings suggest the problems related to behavioral health
can reduce the effectiveness of the program intervention.
29
Table 3.3. Effects of Behavioral Health on Academic Performance, Attendance, and
Disciplinary Action
Academic performance
Attendance
Disciplinary action
Math
Reading
GPA
Absence
Tardy
Assign.
missing
Discipline
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Race/ethnicity:
- Asian
0.627+
0.468
2.331
-3.636
-2.945*
-1.601
-3.588
(0.098)
(0.090)
(0.415)
(2.873)
(0.117)
(0.951)
(2.556)
- Black
0.047
0.138
0.401
2.823
-0.056
-0.446
0.603
(0.113)
(0.053)
(0.601)
(2.165)
(0.311)
(0.157)
(0.720)
- White
-0.008
0.253
0.769
0.457
-1.510*
-0.846+
-1.169
(0.061)
(0.046)
(0.226)
(3.163)
(0.117)
(0.092)
(1.093)
Post
-0.006
-0.076
-0.072
-3.560
0.204
0.041
0.163
(0.097)
(0.080)
(0.101)
(0.739)
(0.402)
(0.167)
(0.110)
Behavioral health:
- AD
-0.007
0.028
-0.149
0.984
0.206
0.049
0.260
(0.016)
(0.059)
(0.092)
(1.035)
(0.070)
(0.032)
(0.527)
- AI
-0.013
-0.159*
-0.184
0.235
0.069
0.008
0.150
(0.016)
(0.009)
(0.102)
(0.405)
(0.031)
(0.065)
(0.027)
- DA
-0.039
0.116
0.198
-0.509
-0.218
-0.145
0.161
(0.009)
(0.104)
(0.219)
(0.391)
(0.364)
(0.192)
(0.736)
- SC
0.064
0.050
0.030
-1.394
-0.000
0.037
0.023
(0.042)
(0.036)
(0.079)
(1.495)
(0.070)
(0.048)
(0.040)
- SI
-0.025
-0.058
0.004
-0.519
0.116
0.097
0.073
(0.038)
(0.047)
(0.196)
(0.229)
(0.128)
(0.163)
(0.705)
- TD
-0.010
-0.045
-0.370+
-0.114
0.157
0.104
0.197
(0.032)
(0.047)
(0.058)
(0.301)
(0.127)
(0.097)
(0.100)
Interactions:
- Post×AD
0.051
0.013
0.220
-0.762
-0.221
-0.062
0.124
(0.031)
(0.067)
(0.067)
(1.440)
(0.119)
(0.068)
(0.220)
- Post×AI
-0.060
0.114
0.041
-0.016
0.094
-0.044
0.241
(0.056)
(0.066)
(0.030)
(0.215)
(0.083)
(0.046)
(0.283)
- Post×DA
0.040
-0.023
-0.341
0.242
0.172
0.305
-0.952
(0.027)
(0.089)
(0.380)
(0.538)
(0.363)
(0.243)
(0.808)
30
- Post×SC
-0.016
-0.080
-0.080+
1.526
0.194+
0.052+
0.505
(0.020)
(0.080)
(0.008)
(1.863)
(0.020)
(0.007)
(0.118)
- Post×SI
-0.033+
-0.025
-0.020
0.959
0.400+
-0.070
1.689
(0.003)
(0.020)
(0.007)
(1.636)
(0.038)
(0.210)
(0.892)
- Post×TD
-0.045
0.002
0.190+
0.320+
0.066
-0.196
0.396
(0.046)
(0.008)
(0.026)
(0.049)
(0.023)
(0.042)
(0.117)
No. of observations
186
186
186
186
186
186
186
R-squared
0.190
0.238
0.182
0.170
0.258
0.262
0.268
Notes: OLS model. Other controls included age, age-squared, and indicators for grade-level and institution. The
standard errors clustered at the institutional level are reported in parentheses. Behavioral health scores are
standardized to the same scale, with a mean of 0 and standard deviation of 1. The asterisks +, *, **, and *** indicate
statistical significance at the 10%, 5%, 1%, 0.1% levels, respectively.
31
4. WINNING FUTURES PROGRAM
Like the Behavioral Health Program described in the previous program, the Winning
Futures Program aims to improve students’ psychological and behavioral health through training
students, particularly 6th and 7th graders, to develop strategies to be more aware of and improve
upon their behaviors. As the same manner as above, we merged data from the Winning Futures
Program and the Centric Student Database reported in Section 2 in order to consider additional
student characteristics including demographics and educational status (e.g., affiliated institution
and grade). Using this merged dataset, two main quantitative analyses were conducted to
estimate the effects of the Winning Futures Program and learned strategies on our key outcome
variables—student academic performance, attendance, and disciplinary actions. The first part of
this section reports how we generated variables of interest (i.e., behavioral and health strategies)
for reliable statistical analyses. Second, we summarize the descriptive statistics of the merged
sample. Third, the effects of participation in the Winning Futures Program are assessed
conditional on students’ embrace of individual behavior and health strategies in relation to
ACCUCESS mathematics and reading scores, GPA, missing assignments, rates of absences,
tardiness, and disciplinary occurrences.
In both pre- and post-intervention surveys, program participants responded to four-point
Likert scale items (“strongly disagree”, “disagree”, “agree”, and “strongly agree”) on their own
behavioral and health strategies. In order to combine these responses in a statistically reliable
manner, Cronbach’s alphas for several categories were calculated. Based on these parameter
estimates, we categorized behavioral and health strategies into five domains which we call
integrity (r = 0.64), generosity (r = 0.74), recognition (r = 0.83), effort (r = 0.69), and self-esteem
(r = 0.73). New composite variables were generated by taking the average across the grouped
ordinal variables. The related questions and Cronbach’s alpha for each behavioral and health
strategy are reported in Table 4.1 below.
32
Table 4.1. Behavior and Health Strategy Categories
Scales
Questions on Scale
Cronbach’s
Alpha
Integrity
A1.
A2.
A3.
A4.
B5.
E1.
E2.
E3.
E6.
D2.
D6.
F4.
F5.
F6.
Lying to others makes me feel uncomfortable.
If I knew for sure I wouldn’t get caught, I would probably
steal something I really wanted. (reversed)
I follow through on promises that I make.
It is important to me that I earn the true of other people.
Being an ethical person is more important than being rich.
If I play a game, I usually follow the rules even if I could
get away with cheating.
It is not important to listen to authority figures like adults.
(reversed)
I am able to be patient and wait my turn for things.
I have control over my own future.
I am in control of my actions.
Other people decide what happens to me. (reversed)
It is hard to get ahead without breaking the law now and
then. (reversed)
Getting in trouble is ok as long as my friends think it’s
cool. (reversed)
It is ok to break the rules as long as you do not get caught.
(reversed)
0.64
Generosity
A5.
B1.
B2.
B3.
B4.
C1.
C2.
C3.
C4.
D8.
E4.
E5.
F1.
F3.
F3.
My friends and family can count on me to help them if
they need me.
I treat other people how I would like to be treated.
It is ok for people to be different than me.
I swear at people who make me angry. (reversed)
I am sensitive to people’s feelings, even if they are not my
friends.
I choose to listen when other people talk to me.
I try to cheer people up when things are bothering them.
I give people compliments, even if I do not know them.
I am able to forgive others when they do something wrong
to me.
I act responsibly, even when other people might ridicule
me.
I do things that are really not fair to people who I don’t
care about. (reversed)
I do not blame someone else if I am the one who did
something wrong.
I make sure that I think about how what I do may bother
someone else.
I would like to volunteer to help my community and the
environment.
0.74
33
Scales
Questions on Scale
Cronbach’s
Alpha
I respect people like parents, teachers, and other adults
who have authority.
Recognition
D1.
D4.
G1.
G2.
G3.
G4.
G5.
G6.
G7.
G8.
G9.
G10.
G11.
It is important to work hard to get what you want.
To make a good decision, it is important to think critically
before you do something.
It is important to study hard for good grades.
It is important to help out around the house.
It is important to graduate from high school.
It is important to go to college or learn a trade.
It is important to own my own home.
It is important to have a happy family life.
It is important to work hard to get ahead.
It is important to save money for the future.
It is important to stay in good health.
It is important to avoid using drugs.
It is important to avoid using alcohol.
0.83
Effort
D3.
D5.
H1.
H2.
H3.
H4.
H5.
I usually complete a task even if I get frustrated in the
process.
I can set aside my short-term goals in order to achieve my
long-term goals.
I do some homework at home almost every day.
When I am supposed to read something for school, I
usually do.
I have good study habits.
I use my study time well.
If an average student used my study habits, they would do
well in school.
0.69
Self-esteem
D7.
I1.
I2.
I3.
L1.
L2.
L3.
L4.
L5.
L6.
L7.
L8.
L9.
L10.
It does not matter how hard I try, I still will not be able to
succeed. (reversed)
I believe I will graduate from high school.
I believe I will go to college.
I believe I will be able to stay out of trouble with the law.
I feel that I am a person of worth, at least on equal par
with others.
I feel that I have a number of good qualities.
I tend to feel like a failure. (reversed)
I am able to do things as well as other people.
I do not have much to be proud of. (reversed)
I take a positive attitude towards myself.
On the whole, I am satisfied with myself.
I wish I could have more respect for myself. (reversed)
I feel useless at times. (reversed)
Sometimes I think I am no good at all. (reversed)
0.74
Notes: New variables were generated by taking the average of ordinal variables grouped based on Cronbach’s alpha.
34
Table 4.2 reports the descriptive statistics for our key variables of interest. A combined
total of 80 students participated as respondents in the survey (55 in the pre-intervention and 25 in
the post-intervention). All program participants were students at Thompson Middle School,
Saginaw Arts and Science Academy, and Zilwaukee K-8 School. Half of all students in the pre-
intervention student were at Thompson Middle School, compared to 31.25% and 18.75% for the
other schools, respectively. The post-intervention survey was slightly skewed toward Saginaw
Arts and Science Academy (48%) over Thompson Middle School (44%) with just 2 students
(8%) from Zilwaukee K-8 School.
The sample was also not evenly distributed across grade-levels, with more participants in
6th grade (62.50% in the pre-intervention survey and 60.00% in the post-intervention survey).
African Americans/Blacks accounted for the majority of the sample (approximately 72% in each
survey) while Hispanics/Latinos and Asians/Pacific Islanders each accounted for less than 10%
of each survey. Caucasians/Whites varied between 15.63% of the pre-intervention survey and
12.00% of the post-intervention survey).
Students generally scored higher on the behavioral and health strategy metrics in the
post-intervention surveys, with scores on integrity, generosity, recognition increasing 2.11%,
0.68%, and 0.27%, respectively. The average score for effort, however, decreased by 2.31%, and
the average score for self-esteem remained flat at 3.29 points. As in the prior sections, because
pre- and post-intervention groups do not contain evenly balanced samples, the changes on these
measures cannot be directly attributed to the program intervention itself.
35
Table 4.2. Summary Statistics for the Winning Futures Program
Pre-survey
Post-survey
(1)
(2)
Race/ethnicity (%):
- Hispanic
6.25
8.00
- Asian
6.25
8.00
- Black
71.88
72.00
- White
15.63
12.00
Grade (%):
- 6th
62.50
60.00
- 7th
37.50
36.00
- 8th
0.00
4.00
Affiliated institution (%):
- Thompson Middle School
50.00
44.00
- Saginaw Arts and Science Academy
31.25
48.00
- Zilwaukee K-8 School
18.75
8.00
Average behavioral and health strategy scores:
- Integrity
3.25
3.32
- Generosity
2.92
2.94
- Recognition
3.67
3.68
- Effort
3.10
3.03
- Self-esteem
3.29
3.29
Academic performance:
- ACCUESS math
627.96
642.74
- ACCUESS reading
689.80
619.57
- GPA
2.53
2.06
Attendance:
- Absences
3.61
3.84
- Amount of time tardy
14.31
14.84
Disciplinary Action:
- Missing assignments
36.45
28.48
- Discipline occurrence
1.97
1.44
Number of observations
55
25
In order to estimate the effects of the program intervention and behavioral and health
strategies on our key outcomes, we regressed academic performance, attendance, and
disciplinary action on the behavioral and health strategy scores (integrity, generosity,
36
recognition, effort and self-esteem) and an indicator for the post-treatment period using the
following OLS model (3).
(3)
where is the key outcomes of a participant i; is an indicator variable indicating the two
time periods (coded 0 for pre-intervention and 1 for post intervention); is comprised of the
five behavior and health strategies described above (integrity, generosity, recognition, effort, and
self-esteem); is the interaction term between the indicator for the intervention and
each of the behavioral and health strategy scores. All other variables remain the same as in
previous models. Parameters of interest are the behavioral and health strategy scores () and the
interaction () between it and the indicator for program participation. In order to ease
interpretation of the model output, behavioral and health strategy scores were standardized to the
same scale, with a mean of 0 and standard deviation of 1. The key outcomes modeled were the
same as prior models: academic performance (the logarithm of ACCUESS mathematics, the
logarithm of ACCUESS mathematics reading score, and GPA), attendance (the number of
absences and the logarithm of minutes of tardiness), and disciplinary actions (the logarithm of
the number of missing assignments and the number of disciplinary occurrences) of a participant
i. All missing data was imputed by taking the average values during the model estimation, and
standard errors were clustered at the institutional level.
Table 4.3 summarizes estimates of model coefficients and standard errors for the
variables of interest. Overall, the effects of most behavioral and health strategy scores were
statistically insignificant when controlling for all other variables. Only the effects of self-esteem
and effort were significant at the 5-10% level. For example, self-esteem was found to be
positively related to ACCUCESS reading scores but also the number of absences. Its negative
effect on the number of disciplinary occurrences was statically significant at the 5% level
whereas its negative relationship with absences was statically significant at the 10 percent level.
Overall, the coefficient associated with participation in the program was negative but in order to
avoid misinterpretation, we focus on the combined effect of participation and the interaction
parameter . The negative effects of program intervention on student academic performance,
37
attendance, and disciplinary action are larger among participants with higher behavior and health
strategies. For example, a one-unit increase in self-esteem, the program intervention is predicted
to decrease ACCUCESS reading score by 0.11% (-0.093+0.125-0.139) and class absence by
0.31 (-0.001-0.135-0.178). Surprisingly, the findings suggest the behavior and health strategies
can worsen the effectiveness of the program intervention; however, there were too few
observations to reliably assess this programs’ effectiveness.
38
Table 4.3. Effects of Behavior and Health Strategies on Academic Performance,
Attendance, and Disciplinary Action
Academic performance
Attendance
Disciplinary action
Math
Reading
GPA
Absence
Tardy
Assign.
missing
Discipline
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Race/ethnicity:
- Asian
0.308+
0.154
0.549
-0.668
-1.329
0.191
0.026
(0.076)
(0.142)
(0.206)
(0.260)
(0.536)
(0.733)
(0.417)
- Black
-0.031
0.057
-0.835*
0.745+
1.012
1.083
0.611+
(0.111)
(0.054)
(0.193)
(0.237)
(0.703)
(0.906)
(0.154)
- White
0.279
0.076
0.086
-0.246
-3.440**
-0.106
-0.708
(0.237)
(0.353)
(0.402)
(0.369)
(0.115)
(1.166)
(1.283)
Post
-0.018
-0.093
-0.322
-0.001
0.182
-0.133
-0.799
(0.039)
(0.054)
(0.196)
(0.122)
(0.753)
(0.106)
(0.588)
Behavior and health strategy:
- Integrity
-0.027
-0.030
-0.111
0.153
1.514
-0.041
-0.193
(0.027)
(0.086)
(0.154)
(0.162)
(0.899)
(0.086)
(0.422)
- Generosity
-0.019
0.099
-0.184
0.378
0.600
0.014
0.386
(0.048)
(0.100)
(0.438)
(0.309)
(0.849)
(0.213)
(0.262)
- Recognition
-0.015
-0.081
0.140
-0.177
-0.086
0.087
0.583
(0.043)
(0.060)
(0.112)
(0.136)
(0.736)
(0.057)
(0.387)
- Effort
0.046
0.044
0.044
-0.399+
-2.438
-0.004
-0.665
(0.070)
(0.056)
(0.242)
(0.095)
(1.150)
(0.166)
(0.343)
- Self-esteem
0.029
0.125*
0.088
0.135*
0.018
-0.071
-0.856*
(0.029)
(0.019)
(0.135)
(0.024)
(0.498)
(0.064)
(0.126)
Interactions:
- Post×Integrity
0.053
0.045
0.056
-0.232
-0.635
0.336
0.520
(0.073)
(0.169)
(0.126)
(0.176)
(1.011)
(0.139)
(0.422)
- Post×Generosity
0.074
-0.167
0.329
-0.467
-1.778
-0.212
-1.124
(0.048)
(0.105)
(0.463)
(0.309)
(1.299)
(0.365)
(0.454)
- Post×Recog.
-0.044
0.094
0.142
0.151
-1.839
-0.309
-0.607
(0.053)
(0.092)
(0.341)
(0.254)
(1.131)
(0.119)
(0.428)
- Post×Effort
-0.040
0.079
-0.092
0.142
2.909*
0.196***
0.855
(0.150)
(0.139)
(0.242)
(0.095)
(0.595)
(0.003)
(1.109)
- Post×Self-est.
0.086
-0.139
0.118
-0.178*
-0.405
-0.047
0.482+
(0.122)
(0.069)
(0.081)
(0.021)
(1.282)
(0.073)
(0.123)
No. of observations
57
57
57
57
57
57
57
R-squared
0.419
0.347
0.563
0.790
0.442
0.448
0.334
39
Notes: OLS model. Other controls include indicators for grade-level and institution. Standard errors clustered at the
institutional level are reported in parentheses. The scores of behavioral health strategies are standardized to the same
scale, with a mean of 0 and standard deviation of 1. The asterisks +, *, **, and *** indicate statistical significance at
the 10%, 5%, 1%, 0.1% levels, respectively.
40
5. KAPLAN ACT PREP
The Centric Program, Kaplan ACT Prep, focuses on improving students’ academic
performance on the ACT, the university entrance exam. As in previous sections, to analyze the
effectiveness of the Kaplan ACT Prep Program, we merged data associated with students’
performance in this program and characteristics derived from the Centric Student Database
reported in Section 2. Using this merged data set, two main quantitative analyses were conducted
to estimate the effects of program participation on student performance. In particular, we
examined the interaction of participation in the program with behavioral characteristics including
rates of missing assignments, absences, tardiness, and disciplinary occurrences. Student
academic performance was measured using ACT composite, English, mathematics, reading,
science, and essay scores as well as ACCUCESS mathematics and reading scores and GPA. We
begin by analyzing the summary statistics of the merged sample. After, we model the effects of
student attendance and disciplinary actions on academic performance interacted with indicators
for participation in the program intervention.
Table 5.1 summarizes the descriptive statistics for our key variables of interest. A total of
183 students in 6th-12th grades participated in the program. Most were from Arthur Hill High
School (32.24%) and Saginaw High School (35.52%), or Saginaw Arts and Science Academy
(20.77%), with the remainder split between Thompson Middle School (6.01%) and Zilwaukee
K-8 School (5.46%). The sample was not equally distributed across grade levels, with fewer
participants in 6th (10.93%) and 12th (3.83%) grades. African Americans/Blacks accounted for
the vast majority of the sample (71.60%) while Hispanics/Latinos accounted for 13.58%, and
Caucasians/Whites for 12.96%. Asians/Pacific Islanders accounted for a fraction of the total
(1.85%).
The Kaplan ACT Prep Program did not appear to have a positive impact on students’
academic performance overall. After the program intervention, the average scores on most ACT
subjects decreased slightly (a decline of 0.81% in the overall composite), although this difference
was not statistically significant. The average essay score on the ACT increased, however, by
3.5%. Similarly, ACCUESS mathematics and reading scores decreased 3.05% and 3.69%,
respectively, and GPAs also declined 2.26% on average. These changes, however, cannot be
directly interpreted as effects of the program intervention because such changes might be caused
by other factors including differences in the difficulty of first and second semester exams.
41
Table 5.1. Summary Statistics for Kaplan ACT Prep
Pre-survey
Post-survey
(1)
(2)
Race/ethnicity (%):
- Hispanic
13.58
13.58
- Asian
1.85
1.85
- Black
71.60
71.60
- White
12.96
12.96
Grade (%):
- 6th
10.93
10.93
- 7th
4.92
4.92
- 8th
24.59
24.59
- 9th
20.22
20.22
- 10th
17.49
17.49
- 11th
18.03
18.03
- 12th
3.83
3.83
Affiliated institution (%):
- Arthur Hill high school
32.24
32.24
- Saginaw high school
35.52
35.52
- Thompson middle school
6.01
6.01
- Saginaw arts and science
academy
20.77
20.77
- Zilwaukee K-8 school
5.46
5.46
Average ACT scores:
- Composite
13.50
13.39
- English
12.96
12.86
- Math
14.38
13.91
- Reading
12.26
12.13
- Science
14.48
13.92
- Essay
4.18
4.33
Academic performance:
- ACCUESS math
719.53
697.60
- ACCUESS reading
781.79
752.97
- GPA
2.21
2.16
Attendance:
- Attendance
7.05
7.05
- Absences
4.23
3.61
- Amount of time tardy
12.87
15.91
Disciplinary Action:
- Missing assignments
39.36
41.33
42
- Discipline occurrences
1.41
1.46
Number of observations
183
183
In order to systematically assess the effects of student absences and disciplinary action on
academic performance as they related to participation in the Kaplan ACT Prep Program, we
regressed ACCUESS and ACT scores on the number of assignments students were missing,
absences, tardiness, and disciplinary occurrences both before and after the program intervention
using the following OLS model (4).
(4)
where is the key outcomes for each participant i; is an indicator variable for the two time
periods (coded 0 for pre-intervention and 1 for post intervention); consists of the attendance
and tardiness variables; is the interaction term between the indicator for the
intervention and the attendance and tardiness variables; is the set of disciplinary action
variables; is the interaction term between the indicator for the intervention and the
disciplinary action variables. All other variables are the same as in previous models. The key
academic performance outcomes examined are the logarithm of Kaplan ACT composite,
English, mathematics, reading, science, and essay, the logarithm of ACCUESS mathematics and
reading scores, and GPA of a participant i. We take a natural logarithm of the tardy periods and
missing assignments variables. All missing data was imputed by the taking the average value of
variables during the estimation procedure. Standard errors were clustered at the institutional
level.
Estimates of model coefficients and standard errors are reported in Table 5.2 below.
Overall, absences, tardiness, missing assignments, and being subject to disciplinary actions were
shown to be negatively related to student academic performance on both the ACCUCESS and
ACT exams, as well as on students’ GPAs. The indicator for program intervention was generally
not significant across the models. However, to properly interpret these coefficient estimates, we
focus on parameters, and , or the attendance and tardiness variables and their interaction with
program participation. The negative effects of program intervention on student academic
performance are smaller among participants with fewer attendance issues. For example, a one-
43
unit increase in student absences, the program intervention is predicted to decrease ACT
composite scores by 0.07% (-0.074-0.006+0.007) and ACT science scores by 0.11% (-0.116-
0.004+0.007) but increase GPAs by 0.20 points (0.224-0.036+0.015), holding all other variables
constant. If we assume a ten-unit increase in absences, the program intervention is predicted to
decrease ACT composite scores by 0.06% (-0.074-0.006×10+0.007×10) and ACT science scores
by 0.09%(-0.116-0.004×10+0.007×10) but increase GPA by 0.01 points (0.224-
0.036×10+0.015×10). That is, the program intervention was more effective among students with
higher attendance rates.
On the other hand, the effects of disciplinary actions on program intervention effects vary
considerably, depending on the outcome measure. For instance, a one-unit increase in
disciplinary occurrences, the program intervention is predicted to decrease ACT composite score
by 0.08% (-0.074-0.012+0.011) and ACT science score by 0.11% (-0.116-0.012+0.020) but
increase GPA by 0.17 points (0.224-0.115+0.057), holding all other variables constant. A ten-
unit increase in disciplinary occurrences and is predicted to decrease ACT composite scores by
0.08% among program participants and ACT science scores by 0.04% (compared to -0.12% in
the pre-intervention group), and GPA by 0.36 points (compared to -1.11 for the pre-intervention
group). In other words, the negative effects of program intervention on ACT science score are
smaller among participants with more disciplinary occurrences but its effects on ACT composite
score or GPA are larger. The findings suggest that the effects of participating the Kaplan ACT
Prep Program can be moderated by participants’ attendance issues and behaviors resulting in
disciplinary actions.
44
Table 5.2. Attendance and Discipline Actions on Academic Performance
Kaplan ACT Score
ACCUCESS Score
Composite
English
Math
Reading
Science
Essay
Math
Reading
GPA
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
Race/ethnicity:
- Asian
-0.128
-0.136
0.078
-0.313
-0.101
-0.214
0.144
0.067
-0.154
(0.108)
(0.109)
(0.110)
(0.383)
(0.104)
(0.118)
(0.090)
(0.115)
(0.226)
- Black
0.049
0.047
0.095
-0.001
0.030
-0.029
-0.034
0.007
-0.093
(0.065)
(0.065)
(0.078)
(0.072)
(0.118)
(0.054)
(0.083)
(0.070)
(0.252)
- White
0.167
0.150
0.226
0.166
0.167
0.019
0.116
0.121
-0.208+
(0.118)
(0.097)
(0.131)
(0.170)
(0.142)
(0.086)
(0.089)
(0.125)
(0.081)
Post
-0.074
-0.058
-0.323
-0.187
-0.116
-0.070
-0.103
0.000
0.224
(0.196)
(0.233)
(0.260)
(0.176)
(0.160)
(0.126)
(0.069)
(0.051)
(0.184)
Attendance:
- Absence
-0.006**
-0.022
-0.003
-0.003
-0.004**
-0.005
-0.009
-0.001
-0.036**
(0.001)
(0.011)
(0.003)
(0.002)
(0.001)
(0.004)
(0.004)
(0.001)
(0.005)
- Tardy
0.000
-0.009
-0.020**
-0.004
0.003
-0.008
0.001
-0.025
-0.115
(0.025)
(0.008)
(0.003)
(0.047)
(0.031)
(0.065)
(0.018)
(0.017)
(0.069)
Disciplinary actions:
- Assign. missing
-0.001
0.016
-0.010
-0.004
0.007
-0.006
-0.008
0.018
-0.318+
(0.022)
(0.033)
(0.026)
(0.018)
(0.021)
(0.019)
(0.018)
(0.027)
(0.132)
- Discipline
-0.012
-0.022
-0.003
-0.020
-0.012
-0.027+
-0.018
-0.014*
-0.115**
(0.009)
(0.018)
(0.006)
(0.011)
(0.015)
(0.010)
(0.009)
(0.004)
(0.014)
Interactions:
- Post×Absence
0.007
0.026
-0.005
0.009*
0.007
0.006
0.011*
0.006*
0.015
(0.003)
(0.014)
(0.009)
(0.003)
(0.013)
(0.005)
(0.003)
(0.002)
(0.009)
- Post×Tardy
-0.017
-0.008
-0.027
-0.012
0.015
0.030
-0.019
-0.016
-0.078*
(0.036)
(0.021)
(0.056)
(0.061)
(0.077)
(0.067)
(0.021)
(0.012)
(0.026)
- Post×Assign. Missing
0.019
-0.010
0.102
0.045
-0.007
0.015
0.012
-0.011
-0.074
(0.051)
(0.064)
(0.098)
(0.044)
(0.066)
(0.027)
(0.011)
(0.013)
(0.061)
- Post×Discipline
0.011
0.019
0.006
0.013
0.020
0.013
0.012
0.008
0.057*
45
(0.013)
(0.020)
(0.009)
(0.013)
(0.021)
(0.010)
(0.014)
(0.006)
(0.017)
Number of observations
324
324
324
324
324
324
324
324
324
R-squared
0.133
0.175
0.101
0.073
0.068
0.128
0.147
0.127
0.455
Notes: OLS model. Other controls include indicators for grade level and institution. Standard errors clustered at institutional level are reported in parentheses.
The asterisks +, *, **, and *** indicate statistical significance at the 10%, 5%, 1%, 0.1% levels, respectively.
46
6. PARENTAL ENGAGEMENT SURVEY
The Parental Engagement Survey served to examine how well the Centric Program
promoted engagement in several key areas including parenting (PA), communicating (CO),
volunteering (VO), learning at home (LH), decision-making (DM), and collaborating with the
community (CC). As in previous sections, we merged data from the Parent Engagement Survey
and the Centric Student Database including information on student demographics and
educational status (e.g., affiliated institution and grade). Using this merged dataset, two main
quantitative analyses were conducted to estimate the relationship between parental evaluations of
the Centric Program (parental evaluation) on our key outcome variables—student academic
performance, attendance, and disciplinary actions.
3
In the first part of this section, we test the
reliability of items from the Parental Engagement Survey that comprise the six variables noted
above. Next, we report the summary statistics of the merged sample. Finally, the effects of
parental evaluations on ACCUCESS mathematics and reading scores, GPAs, number of missing
assignments, absences, tardiness, and disciplinary occurrences are investigated in relation to the
program intervention.
In both pre- and post-intervention surveys, parents of program participants responded to a
number of four-point Likert scale items (“strongly disagree”, “disagree”, “agree”, and “strongly
agree”) regarding how effectively the Centric Program promoted their engagement. In order to
create statistically reliable variables which represent parental evaluation, Cronbach’s alphas for
different categories were calculated. Based on Cronbach’s alphas, we categorized parental
evaluations into six domains: parenting (PA, r = 0.93), communicating (CO, r = 0.91),
volunteering (VO, r = 0.92), learning at home (LH, r = 0.96), decision making (DM, r = 0.95),
and collaborating with the community (CC, r = 0.90). Based on these parameter estimates, we
generated new composite variables averaging across the ordinal variables. The related questions
and Cronbach’s alpha for each of the parental evaluation category are reported in Table 6.1.
3
Note that parental engagement does not mean how much parents engage in the Centric Program.
47
Table 6.1. Parental Engagement Categories
Scales
Questions on Scale
Cronbach’s
Alpha
Parenting (PA)
Centric program and school district
1. Conducts workshops, provides information for parents on the goals
and objectives of the centric program.
2. Provides information to all centric families who want or need it, not
just to the few who can attend workshops or meetings at the school
buildings.
3. Produces information for families that is clear, usable, and linked to
children’s success in school.
4. Asks families for information about children’s goals, strengths, and
talents.
5. Provides families with age-appropriate information on developing
home conditions or environments that support learning.
6. Respects the different cultures represented in our student population.
7. Career Development
8. College admission
9. Skyward training
10. Do’s and don’ts of parent involvement
11. Assessments
12. Study Skills
13. Other Type I Parenting Activities
0.93
Communicating
(CO)
Centric program and school district
1. Centric program has clear two-way channels for communication
from home to Centric and Centric to home.
2. Centric program encourages families to participate in parent
conference, Centric activities throughout the year.
3. Centric staff sends home student material for parent review and
comments.
4. Asks families for information about children’s coals, strengths, and
talents.
5. Provides clear information about curriculum, state tests, school and
student results, and report cards.
6. Contacts families of students having academic or behavior problems.
7. Centric staff will use e-mail and the centric website to communicate
with parents, including information on upcoming events.
8. Produces a regular centric newsletter with up-to-date information
about the centric program, special events, organizations, meetings,
and parenting tips.
9. Other type II communicating activities
0.91
Volunteering
(VO)
Centric program and school district
1. Centric program conducts pre- and post- surveys to identify
interests, talents, and availability of parent volunteers to match their
skills and talents with centric and classroom needs.
2. Provides a parent or family room for volunteers and family members
to meet and work, and to access resources about parenting, tutoring,
and related topics.
3. Creates flexible volunteering opportunities and schedules, enabling
employed parents to participate.
4. Schedules special events at different times of the day and evening so
that all families can attend as audiences.
0.92
48
Scales
Questions on Scale
Cronbach’s
Alpha
5. Reduces barriers to parent participation by providing child care.
6. Recognizes volunteers for their time productively.
7. Encourages families and the community to be involved with the
centric program in various was (e.g. lead talks or activities, serve as
audiences, mentor the young men)
8. Other type III volunteering activities
Learning at
Home (LH)
Centric program and school district
1. Centric program provides information to families on how to monitor
and discuss school work at home.
2. Provides information to families on required skills in major subjects.
3. Provides specific information to parents on how to assist students
with skills that they need to improve.
4. Asks parents to focus on reading, listen to children read, or read
aloud with their child.
5. Assists families in helping students set academic goals and select
courses and programs.
6. Provides information and ideas for families to talk with students
about college, careers, postsecondary plans.
7. Other type IV Learning at home activities.
0.96
Decision
Making (DM)
Centric program and school district
1. Involves parents in organized ongoing and timely ways in planning
and improving the centric program.
2. Involves parents in reviewing the centric program goals and
objectives.
3. Recruits parent leaders for committees from all racial, ethnic,
socioeconomic and other groups in the centric program.
4. Develops formal social networks to link all families with their parent
representatives.
5. Includes students (with parents) in decision making.
6. Deals with conflict openly and respectfully
7. Guides parent representative to contact parents who are less
involved for their ideas.
8. Develops the Centric program of family and community
involvement with input from educators, parents, and others.
9. Other type V decision-making activities.
0.95
Collaborating
with the
Community
(CC)
Centric program and school district
1. Centric provides a resource directory for parents and students on
community agencies, services, and programs.
2. Involves families in locating and using community resources.
3. Works with local businesses, industries, libraries, parks, museums,
and other organizations on programs to enhance student skills and
learning.
4. Offers after school programs for students with support from
community businesses, agencies, and volunteers.
5. Other Type VI collaborating with the community activities.
0.90
Notes: New variables were generated by taking the average of ordinal variables grouped based on Cronbach’s alpha.
49
Table 6.2 describes the summary statistics for our key variables of interest. A total of 39
parents participated in the pre-intervention survey, and 83 participated in the post-intervention
survey. Participants were parents of students from 6th-12th grades at Arthur Hill High School
(53.57% in the pre-survey and 45.57% in the post-survey), Saginaw High School (35.71% in the
pre-survey and 26.28% in the post-survey), Thompson Middle School (3.57% in the pre-survey
and 7.59% in the post-survey), Saginaw Arts and Science Academy (7.14% in the pre-survey
and 12.66% in the post-survey), and Zilwaukee K-8 School (0.00% in the pre-survey and 7.59%
in the post-survey). The sample was not equally distributed with respect to students’ grade-
levels, with more participants parents of 6th and 7th graders in the post-intervention survey
(12.66% and 5.06%) compared to 0% and 3.57% in the pre-intervention survey. Parents of
African Americans/Blacks accounted for the majority of the entire sample (85.71% in the pre-
intervention survey and 73.42% in the post-survey) while Hispanics/Latinos and
Caucasians/Whites accounted for 7.14% each in the pre-survey and 12.66% and 11.39% in the
post-survey, respectively. There were no parents of Asian/Pacific Islander students in the pre-
intervention survey, and they accounted for just 2.53% of the post-intervention survey.
After the program intervention, the average scales of parental evaluation increased by
0.12 percentage points on parenting (PA), 0.13 percentage points on communicating (CO), 0.15
percentage points on volunteering (VO), 0.15 percentage points on learning at home (LH), 0.16
percentage points on decision making (DM), and 0.15 percentage points on collaborating with
the community (CC). These changes, however, cannot be interpreted as the effects of the
program intervention because such changes can be caused by the different distribution of
participants before and after the program intervention and/or the external factors such as school
environments and climates.
50
Table 6.2. Summary Statistics for Parental Engagement Survey
Pre-survey
Post-survey
(1)
(2)
Students’ Race/ethnicity (%):
- Hispanic
7.14
12.66
- Asian
0.00
2.53
- Black
85.71
73.42
- White
7.14
11.39
Students’ Grade Level (%):
- 6th
0.00
12.66
- 7th
3.57
5.06
- 8th
28.57
22.78
- 9th
28.57
25.32
- 10th
25.00
21.52
- 11th
10.71
11.39
- 12th
3.57
1.27
Students’ Affiliated institution (%):
- Arthur Hill High School
53.57
45.57
- Saginaw High School
35.71
26.58
- Thompson Middle School
3.57
7.59
- Saginaw Arts and Science Academy
7.14
12.66
- Zilwaukee K-8 School
0.00
7.59
Parental Engagement:
- PA
4.08
4.57
- CO
3.93
4.45
- VO
3.85
4.43
- LH
3.88
4.47
- DM
3.89
4.52
- CC
4.01
4.63
Students’ Academic performance:
- ACCUESS math
624.00
701.41
- ACCUESS reading
789.32
714.45
- GPA
2.18
2.29
Students’ Attendance:
- Absences
5.11
39.42
- Amount of time tardy
13.29
2.24
Students’ Disciplinary Action:
- Missing assignments
49.18
16.29
- Discipline occurrence
2.11
1.35
No. of observations
39
83
51
In order to estimate the effects of parental evaluations on our key outcomes before and
after the program intervention, we regressed academic performance, attendance and disciplinary
actions on parental evaluations of the Centric Program’s efforts to promote parental engagement
with respect to parenting (PA), communicating (CO), volunteering (VO), learning at home (LH),
decision making (DM), and collaborating with the community (CC) and an indicator for the post-
treatment period using the following OLS model (5).
(5)
where is the key outcomes of a participant I; is an indicator for the time periods (coded 0
for pre-intervention and 1 for post-intervention); consists of the scales of parental
evaluations of the Centric Program’s five major efforts on promoting parental engagement (PA,
CO, VO, LH, DM, and CC); is the interaction between the indicator for the
intervention and the parental evaluation items. All other variables are the same as in previous
models. The parameters of interest include the parameters for the set of parental engagement
scores and its interaction ( with the intervention indicator. In order to ease interpretation of
the results, parental evaluation scales were standardized with a mean of 0 and standard deviation
of 1. The key outcomes modeled for each participant i were academic performance (the
logarithm of ACCUESS mathematics, the logarithm of ACCUESS mathematics reading score,
and GPA), attendance (the number of absences and logarithm of amount of time students were
tardy), and disciplinary actions (the logarithm of the number of missing assignments and the
number of discipline occurrences). All missing data was imputed by taking the average during
the estimation process. Standard errors are clustered at the institutional level.
Table 6.3 provides estimates of coefficients and their standard errors for the variables of
interest. Overall, the effects of parental evaluations on the Centric Program’s efforts to promote
their engagement varied depending on the type of outcomes. For example, positive evaluations
with respect to decision-making (DM) were negatively associated with ACCUCESS
mathematics and reading scores, findings that were significant at the 5% to 10% significance
level. This same measure was positively related to the number of missing assignments and
students’ disciplinary occurrences at the one percent significance level. The program intervention
52
negatively affected ACCUCESS reading scores and tardiness at the 0.1% to 5% significance
levels but it might be driven from variations in external factors such as different levels of
difficulty between pre- and post-exams and/or changes in school climates as explained in the
second part of this section.
In order to avoid misinterpreting coefficient estimates, the intervention indicator must be
evaluated in conjunction with the interaction parameter τ. The negative effects of program
intervention on student academic performance, attendance, and disciplinary action are larger
among participants whose parents positively reflected the Centric Program’s efforts in promoting
parental engagement. For example, program intervention combined with a one-unit increase in
parental evaluations with respect to decision-making (DM) is predicted to decrease ACCUCESS
reading scores by 0.16% (compared to a decline of 0.15% for the pre-intervention group) and the
number of disciplinary occurrences by 0.88 (compared to an increase of 3.499 for the pre-
intervention group) but only slight increases in the number of missing assignments by 0.02
(compared to 0.732 for the pre-intervention group). These findings suggest that parents’ positive
reflections on the Centric Program’s efforts to promote parental engagement have little effect on
student’s academic performance but are associated with enhancements in student attendance and
disciplinary actions.
53
Table 6.3. Effects of Parental Engagement on Academic Performance, Attendance, and
Disciplinary Action
Academic performance
Attendance
Disciplinary action
Math
Reading
GPA
Absence
Tardy
Assign.
missing
Discipline
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Race/ethnicity:
- Asian
0.294*
0.015
1.237*
-2.314*
-0.588
-1.917***
-1.658
(0.096)
(0.057)
(0.334)
(0.760)
(0.387)
(0.212)
(0.859)
- Black
0.019
-0.002
0.150
0.416
-0.678
0.704*
0.510
(0.088)
(0.056)
(0.368)
(0.560)
(0.405)
(0.170)
(0.552)
- White
0.220
0.066
0.534
-0.298
-1.140
-0.616
-0.675+
(0.183)
(0.153)
(0.480)
(0.440)
(0.773)
(0.496)
(0.283)
Post
0.067
-0.146***
0.096
0.084
-3.135*
0.342
-0.692
(0.040)
(0.012)
(0.117)
(0.064)
(0.985)
(0.280)
(0.431)
Parental Engagement:
- PA
-0.068*
0.022
0.253
0.203
-2.603*
-0.627+
-2.262*
(0.017)
(0.136)
(0.521)
(0.457)
(0.701)
(0.253)
(0.688)
- CO
0.082*
0.007
-0.124
-0.590
1.408
0.243
0.681
(0.025)
(0.087)
(0.280)
(0.445)
(0.677)
(0.289)
(0.707)
- VO
-0.019
0.044
-0.109
0.419
1.937+
0.366
-0.942
(0.046)
(0.124)
(0.238)
(0.410)
(0.902)
(0.250)
(0.447)
- LH
0.080
0.075*
0.394
-0.290+
-0.459
-1.038**
-1.309
(0.040)
(0.026)
(0.403)
(0.123)
(0.849)
(0.196)
(1.332)
- DM
-0.184+
-0.145*
-0.486
-0.144
1.640
0.732**
3.499**
(0.073)
(0.044)
(0.985)
(0.313)
(1.102)
(0.123)
(0.424)
- CC
0.088+
-0.074+
-0.049
0.822+
-0.730
0.578*
-0.433
(0.032)
(0.030)
(0.737)
(0.328)
(0.892)
(0.187)
(0.616)
Interactions:
- Post×PA
0.119
0.028
-0.093
-0.037
2.021
0.455
1.661*
(0.089)
(0.048)
(0.336)
(0.347)
(1.448)
(0.363)
(0.553)
- Post×CO
-0.064
-0.067
0.166
0.294
-2.150+
0.178
-0.183
(0.057)
(0.148)
(0.489)
(0.406)
(0.898)
(0.190)
(0.743)
- Post×VO
0.091
0.051
-0.004
-0.139
-0.326
-0.815
1.270
(0.116)
(0.159)
(0.354)
(0.428)
(1.391)
(0.510)
(1.203)
- Post×LH
-0.191*
-0.214+
-0.354
0.007
-0.593
0.895*
1.474
(0.064)
(0.089)
(0.663)
(0.246)
(1.421)
(0.294)
(2.221)
- Post×DM
0.081
0.130
-0.035
0.425
-1.057
-0.247
-3.684**
(0.095)
(0.073)
(1.078)
(0.406)
(1.566)
(0.285)
(0.730)
54
- Post×CC
-0.000
0.115
0.308
-0.907+
0.718
-0.632
-0.228
(0.048)
(0.074)
(0.766)
(0.409)
(0.692)
(0.407)
(0.726)
Number of obs.
107
107
107
107
107
107
107
R-squared
0.298
0.246
0.186
0.348
0.294
0.292
0.186
Notes: OLS model. Other controls include indicators for grade level and institution. Standard errors clustered at
institutional level are reported in parentheses. The scores of parental engagement are standardized to the same scale,
with a mean of 0 and standard deviation of 1. The asterisks +, *, **, and *** indicate statistical significance at the
10%, 5%, 1%, 0.1% levels, respectively.
55
7. PARENT WORKSHOP DATA
The Centric Program offered a series of workshops for parents in order to enhance
parental engagement. The following sections assess the effectiveness of each section based on
the surveys collected from parent participants.
Overall, the number of parents who participated the parental workshop was limited with
more participants from Thompson Middle School and Arthur Hill High School than other
affiliated institutions. Parents who participated reported high effectiveness of the workshops on
average.
7.1 Qualitative Data and Analyses
All data from the qualitative component of the Centric program evaluation (open-ended
survey responses) were coded and analyzed using grounded theory (Glaser & Straus, 1967;
Lincoln & Guba, 1985; Mason, 1996; Scott, 1995). This approach involved analyzing data to
develop theoretical and thematic explanations in order to explain, compare, and trace the
development of the researched phenomena. The process involved four steps: (1) comparing data
applicable to each conceptual category, (2) integrating categories and their properties, (3)
delimiting the emergent theory, and (4) writing up the theory (Jorgensen, 1989, p. 113). This
procedure continued until saturation and redundancy occurred (Scott, 1995). Data analysis relied
heavily on the survey transcripts (Morgan, 1998).
The open-ended answers from participants were coded and analyzed using the grounded
theory approach. Members of the evaluation team initially coded the data independently.
Subsequently, the team met collectively to identify and discuss patterns in the data and emergent
themes. This method of analysis allowed these data to be consistently compared and organized
(Glaser & Strauss, 1967; Jorgensen, 1989). Once patterns and themes were identified, the
evaluation team used direct excerpts from the survey transcripts to present and illustrate the
themes and subthemes within the resulting report.
Five parent participants participated in the Communication workshop and completed the
survey summarized in Table 7.1. On average, participants expected improvement in their
effectiveness communicating with their children after attending the workshop and rated the
materials provided in meeting as highly effective. However, the core information on appropriate
of parent-child communication types seems not to have been clearly delivered; only one
participant selected “assertive” as the type of communication they would use in the future.
56
Table 7.1. Communication Effectiveness
Thompson
Middle
(1)
1.
Prior to this workshop, how effective do you feel you were at communicating with
your child
3.60
2.
How effective do you feel you will be at communicating with your child now?
4.60
3.
How effective do you feel this material will be for you and your student?
5.00
4.
Prior to this workshop, which communication type do you think you used?
1.75
5.
After this workshop, which communication type do you think you will use?
2.00
Number of Observations
5
Five participants submitted open-ended responses regarding the tools or techniques they
learned during the course of the workshop. The majority of participants said that the information
they received regarding ways to engage their children were significantly helpful. Among the
most salient attributes of the workshop, participants indicated that they were provided more
effective ways to not only address their children, but to also “be heard by their children.” More
clearly, “listening” was among one of the most useful tools parents took away from this activity.
Participants cited that the activity enabled them to, as one participant put it, “Handle my
boys on the level that they are.” Overwhelmingly, participants indicated that they felt “all” points
were useful within the workshop. Additionally, participants expressed a desire to have received
this information when their children were younger. As one participant expressed, “I wish this
program was available years ago! I called and asked for help for my child and didn’t get much.”
Ultimately, participants valued the “openness of the group” and “learning what ways we
communicate.” Participants indicated that they learned to be more assertive and that they would
try these techniques at home. Participants overwhelmingly expressed that they would
recommend this workshop to a friend or family, stating that “Our biggest hindrance is
communication. I believe it will help others.”
7.2. Conflict Resolution
As noted in Table 7.2, 16 parents participated in the Conflict Resolution Workshop and
completed the evaluation forms (11 from Thompson Middle School and 5 from Zilwaukee K-8
School). On average, compared to parents whose children went Thompson Middle School,
participants whose children went to Zilwaukee K-8 School reported fewer previous experiences
learning about conflict resolution, lower self-perceived effectiveness in handling conflicts with
57
their children, and perceived their children as less effective at handling the conflict resolution
materials. In general, all participants rated high on the effectiveness of the workshop and
expected improvement in their effectiveness in handling the conflict after the workshop.
Table 7.2. Conflict Resolution Effectiveness
Total
Thompson
Middle
Zilwaukee
K8
(1)
(2)
(3)
1.
How effective do you feel you handled this
topic prior to this event?
4.44
4.82
3.60
2.
How often have you used material like this
or this topic in the past?
3.71
4.11
3.00
3.
How effective do you feel you will be with
this topic or use it after this event?
4.56
4.82
4.00
4.
How easy do you feel this material will be
to use after this event?
4.81
4.82
4.80
5.
How effective was this event in making
you aware of consequences?
4.63
4.64
4.60
6.
How effective do you feel your child
currently is at handling this topic?
2.81
2.82
2.80
7.
How easy do you feel this material will be
for your child to use?
4.50
4.64
4.20
8.
How effective do you feel this material will
be at improving your child’s behavior?
4.56
4.82
4.00
9.
How effective do you feel this material will
be at improving your child’s
academics?
4.63
4.82
4.20
10.
How effective do you feel this workshop
would be for your family and friends?
4.75
4.82
4.60
11.
How effective was the speaker/leader for
this event?
4.88
5.00
4.60
12.
How effective were the handouts for this
event?
4.81
5.00
4.40
13.
How effective was the "ice breaker" or
warm-up session for this event?
4.88
5.00
4.60
14.
How effective were the breakouts or role
play sessions for this event?
4.88
5.00
4.60
15.
How effective was the room environment
for this event?
5.00
5.00
5.00
Number of observations
16
11
5
58
A total of 10 participants submitted qualitative responses to open-ended questions. These
data were analyzed and thematic representations were presented through data analyses.
Participants cited several areas of conflict management they felt familiar with prior to the Centric
Conflict Resolution workshop. The most salient of these were: a) stepping away from a conflict
(cool down period); and b) effective communication techniques and strategies. Overwhelmingly,
participants cited utilizing the “6 Step” and “Step Back” methods as key points of learning that
they would use at home or in other situations. The “step back” method was a key point to
recognizing the need to calm down among participants. Moreover, participants felt that this was
a “great learning tool overall.” As one participant reported, “I love these programs/workshops. It
really brought our family closer! I am forever grateful for that!” Overall, participants indicated
the conflict resolution workshop was extremely “helpful” and “effective,” and that all aspects of
the workshop was useful.
7.3. Parent Study Skills Effectiveness
Five parent participants attended the Parent Study Skills workshop and completed the
evaluation survey (Table 7.3). These participants on average rated the study environment they
provided to their child as unsatisfactory. On average, participants expected increases in their own
effectiveness at helping their children with study skills, tips and strategies after attending this
workshop and rated the material as highly effective.
Table 7.3. Parent Study Skills Effectiveness
Saginaw
High
(1)
1.
Prior to this workshop, please rate your ability to help your child with study skills,
tips, and strategies.
3.60
2.
How effective do you feel you will be at helping your child with study skills, tips,
and strategies now?
4.25
3.
How effective do you feel this material will be for you and your child?
4.75
4.
Prior to this workshop, did you prepare a study environment for your child?
2.80
Number of Observations
5
Four useable qualitative responses were extracted from open-ended question presented at
the conclusion of the Parent Study Skills workshop. These responses were analyzed for thematic
representations. While some participants indicated that they would utilize test taking and time
59
management strategies as a result of the information they received from this workshop, other
participants indicated they were more likely to utilize the book that was given to them. In fact,
participants indicated that the booklet was the most helpful tool provided at the workshop.
7.4. Parent-Teacher Conferencing
Six parent participants (3 from Arthur Hill High School, 3 from Thompson Middle
School) participated in Parent Teacher Conferencing workshop and completed the evaluation
form (Table 7.4). Compared to parents whose children went to Arthur Hill High School,
participants whose children went to Thompson Middle School on average reported fewer
experiences attending similar workshops, and perceived their children to be less effective at
handling this issue. They also thought the material would be more difficult for their children. In
general, all participants rated the workshop’s effectiveness highly. However, there was one
participant who reported that they expected to handle this topic less effectively after this event.
Table 7.4. Parent-Teacher Conferencing Effectiveness
Total
Arthur
Hill High
Thompson
Middle
(1)
(2)
(3)
1.
How effective do you feel you handled this
topic prior to this event?
5.00
5.00
5.00
2.
How often have you used material like this
or this topic in the past?
3.83
4.67
3.00
3.
How effective do you feel you will be with
this topic or use it after this event?
4.67
5.00
4.33
4.
How easy do you feel this material will be
to use after this event?
5.00
5.00
5.00
5.
How effective was this event in making
you aware of consequences?
5.00
5.00
5.00
6.
How effective do you feel your child
currently is at handling this topic?
4.17
4.67
3.67
7.
How easy do you feel this material will be
for your child to use?
4.17
5.00
3.33
8.
How effective do you feel this material will
be at improving your child's behavior?
4.67
5.00
4.33
9.
How effective do you feel this material will
be at improving your child's
academics?
4.67
5.00
4.33
10.
How effective do you feel this workshop
would be for your family and friends?
4.67
5.00
4.33
60
11.
How effective was the speaker / leader for
this event?
5.00
5.00
5.00
12.
How effective were the handouts for this
event?
5.00
5.00
5.00
13.
How effective was the "ice breaker" or
warm-up session for this event?
5.00
5.00
5.00
14.
How effective were the breakouts or role
play sessions for this event?
5.00
5.00
5.00
15.
How effective was the room environment
for this event?
5.00
5.00
5.00
Number of observations
6
3
3
Seven useable qualitative responses were extracted from surveys collected at the
conclusion of three separate Effective Parent Conferencing workshops. These an open-ended
responses were analyzed for thematic representations. Participants overwhelmingly indicated that
the Five Steps of Effective Conferencing was the most useful tool or strategy gained from
attending the workshop. They indicated that these tools enabled them to understand how to talk
with the teacher to make inquiries without being off-putting and that ultimately, they were very
effective. One participant indicated, “This is great insight for parents. We need this so much.”
Another participant said, “Awesome workshop. Very informative and connects to real life.”
Participants indicated that they enjoyed the role playing and that it helped them learn to
use a conversational approach and to seek collaboration from the teachers. Overall, participants
indicated that it was “very good information.”
7.5. Parental Involvement
Three parent participants from Arthur Hill High School participated in the Parental
Involvement Workshop and completed the evaluation form (Table 7.5). In general, all
participants rated all major aspects of the workshop highly.
61
Table 7.5. Parental Involvement Effectiveness
Arthur
Hill High
(1)
1.
How effective do you feel you handled this topic prior to this event?
4.67
2.
How often have you used material like this or this topic in the past?
4.00
3.
How effective do you feel you will be with this topic or use it after this event?
4.67
4.
How easy do you feel this material will be to use after this event?
4.67
5.
How effective was this event in making you aware of consequences?
4.33
6.
How effective do you feel your child currently is at handling this topic?
4.33
7.
How easy do you feel this material will be for your child to use?
4.67
8.
How effective do you feel this material will be at improving your child's behavior?
4.50
9.
How effective do you feel this material will be at improving your child's academics?
4.50
10.
How effective do you feel this workshop would be for your family and friends?
4.67
11.
How effective was the speaker / leader for this event?
4.67
12.
How effective were the handouts for this event?
4.67
13.
How effective was the "ice breaker" or warm-up session for this event?
4.50
14.
How effective were the breakouts or role play sessions for this event?
4.67
15.
How effective was the room environment for this event?
4.67
Number of Observations
3
Five useable qualitative responses were analyzed from surveys administered upon the
conclusion of the Parental Involvement workshop. These responses were assessed for thematic
representations. The majority of participants indicated that they felt they were effective with
regards to their involvement prior to the workshop. However, participants felt that the most
useful tool gained from participation in the workshop was how and why they should be even
more involved. They expressed that they ascertained more clearly how to work together as a
family as a result of their participation. Additional key points participants indicated they will use
at home was the importance of better communication. Overall, participants felt all of the
information provided was useful.
7.6. Skyward Training
Seven parents from Arthur Hill High School participated in the Skyward Training
Workshop and completed evaluation forms (Table 7.6). On average, participants reported low
usage of Skyward before the workshop and did not feel they were effective at using Skyward for
tracking grades, attendance, missing assignments, or for communicating with teachers. In
general, all participants expected to use of Skyward more after the workshop and rated the
material offered by the workshop as highly effective. All participants felt the workshop would be
helpful in improving their child’s grades and academic performance.
62
Table 7.6. Skyward Training Effectiveness
Arthur
Hill High
(1)
1.
Before this workshop how often did you utilize Skyward?
2.29
2.
How often do you feel you will use Skyward now?
4.71
3.
Before this training, how effective did you feel using Skyward for tracking
grades?
2.71
4.
Before this training, how effective did you feel using Skyward for tracking
attendance?
2.86
5.
Before this training, how effective did you feel using Skyward for tracking
missing assignments?
2.86
6.
Before this training, how effective did you feel using Skyward for
communicating with teachers?
3.14
7.
How effective do you feel using Skyward now, after the training, to track
grades?
4.67
8.
How effective do you feel using Skyward now, after the training, to track
attendance?
4.67
9.
How effective do you feel using Skyward now, after the training, to track
missing assignments?
4.67
10.
How effective do you feel using Skyward now, after the training, to
communicate with teachers?
4.50
11.
How effective do you feel this material will be for you and your student?
4.83
12.
Do you feel that because of this workshop you will be able to help your child
improve their grades and academic performance?
100%
Number of Observations
7
Qualitative data were extracted from three useable surveys related to the Centric Skyward
Training workshop. Data were analyzed from these qualitative results and several themes were
analyzed. Participants overwhelmingly said that the workshop provided their child with tools to
reach their desired GPA. They indicated that all of the information was useful and that “more
parents should attend.” Participants indicated that the tool was very valuable in enabling them to
track attendance and grades. One participant said, “Hey, incomplete assignment means no
computer. This is a weapon, a very good one, I like it.” Another participant said, “This training
was helpful, quick, and to the point.”
Participants expressed great satisfaction with the training and indicated that it is a
valuable and useful tool that will help them stay on top of their children’s school work.
63
7.7. Strengthening Families Series
Twenty parents (4 from Saginaw High School, 16 from Thompson Middle School)
participated in Parent Teacher Conferencing and completed the evaluation forms (Table 7.7). All
participants felt their children were not previously effective in handling this issue. Compared to
parents whose children went to Saginaw High School, participants whose children went to
Thompson Middle School on average reported fewer previous experiences attending similar
workshops and expressed less self-perceived efficacy in handling this topic. In general, all
participants rated the effectiveness of all major aspects of the workshop highly and expected to
increase their ability to handle this topic after attending this workshop.
64
Table 7.7. Strengthening Families Series Effectiveness
Total
Saginaw
High
Thompson
Middle
(1)
(2)
(3)
1.
How effective do you feel you handled this
topic prior to this event?
4.10
4.75
3.94
2.
How often have you used material like this
or this topic in the past?
3.88
4.00
3.86
3.
How effective do you feel you will be with
this topic or use it after this event?
4.63
4.25
4.73
4.
How easy do you feel this material will be
to use after this event?
4.53
4.75
4.47
5.
How effective was this event in making
you aware of consequences?
4.70
5.00
4.63
6.
How effective do you feel your child
currently is at handling this topic?
3.55
3.75
3.50
7.
How easy do you feel this material will be
for your child to use?
4.30
4.75
4.19
8.
How effective do you feel this material will
be at improving your child's behavior?
4.42
4.75
4.33
9.
How effective do you feel this material will
be at improving your child's
academics?
4.30
4.50
4.25
10.
How effective do you feel this workshop
would be for your family and friends?
4.55
4.50
4.56
11.
How effective was the speaker / leader for
this event?
4.80
4.50
4.88
12.
How effective were the handouts for this
event?
4.85
4.75
4.88
13.
How effective was the "ice breaker" or
warm-up session for this event?
4.78
4.75
4.79
14.
How effective were the breakouts or role
play sessions for this event?
4.79
4.75
4.80
15.
How effective was the room environment
for this event?
4.75
4.75
4.75
Number of observations
20
4
16
65
Thirty-nine useable surveys were collected from the Strengthening Families Series
Workshops from which qualitative data were extracted from open-ended questions and analyzed
thematically. The most salient aspects participants took away from the series included: listening
and understanding your children (good communication), establishing rules/limits; and
disciplining with love (consequences). One participant stated, “This is helpful for parents to help
them better understand how to deal properly with their family members.”
Additionally, participants really responded to the “point system” presented in the series
as information that will help them work with their students to develop the values they want to see
in their child. One parent said, “The point system will give more incentive to complete the task
without a bad attitude.” Another participant posited, “The point system will be effective for our
family… adding the ground rules to our family meetings.”
Participants indicated that among the things they learned, they would employ and utilize
compliments to provide positive encouragement for their children. One participant said,
“Valuable information which I believe will lead to more effective results including
communication,” as a response to recommending this workshop to others. Participants indicated
that the workshop provided “realistic guidance” that served to improve family structure with the
least amount of resistance. Ultimately, participants valued this workshop and thought that it
would be a good class for “all parents and families.”
7.8. Master Student Series Workshops
Ten parent participants from Arthur Hill High School participated in the Master Student
Series Workshop and completed the evaluation forms (Table 7.8). The first workshop in this
series provided information on how parents can better communicate their expectations to their
children. The second workshop focused on how to monitor students’ academic progress. On
average, participants reported that they could more effectively communicate their expectations
and monitor their children’s academic progress. All participants felt the workshop would be
helpful in improving students’ grades and academic performance.
66
Table 7.8. Master Student Series Workshops Effectiveness
Arthur
Hill High
(1)
Series 1
1.
Prior to this workshop, how effective do you feel you were at communicating
your expectations with your child?
3.30
2.
How effective do you feel you will be at communicating expectations for your
child now?
4.50
3.
How effective do you feel this material will be for you and your child?
4.50
4.
How many different ways did you learn to communicate expectations to your
child during this workshop?
3.56
Series 2
1.
Prior to this workshop, how effective did you feel you were at monitoring
your student’s academic progress?
3.60
2.
How effective do you feel you will be at monitoring your student's academic
progress now, after this workshop?
4.50
3.
Before this workshop, how effective do you feel that you were at establishing
clear rewards and consequences for your student's actions?
3.10
4.
How effective do you feel that you will be at establishing clear rewards and
consequences for your student's actions now after this workshop?
4.60
5.
How effective do you feel this material will be for you and your student?
5.00
Number of Observations
10
Master student surveys were distributed on February 5th and 27th. Of the qualitative
portions of the surveys submitted, 25 total useable surveys were collected. Qualitative data from
these surveys were collected and analyzed for consistent thematic representations. The data
represented addressed the value of the workshop, as well as useful tools and techniques that were
taken away as a result of participation therein. The most salient tools and techniques that
participants took away from the workshop were: a) the value of setting and verbalizing
expectations; b) setting mission and vision statements for the family; and c) positive
affirmations. Participants indicated that as a result of the workshop, they understood the need for
the parent’s and the child’s vision to be properly aligned. They also indicated that encouraging
student to teacher communication through the student/teacher commination form was also a
valuable tool that they would adopt. Participants cited that the workshop was extremely valuable
67
and that it served to have a positive influence on students. As one participant expressed, “This
program will help my child become successful.”
Participants felt the workshop was “encouraging” and “needed.” Suggestions posited by
the participant that might enhance the issues presented in this workshop were to include this
information at school orientations and at parent/teacher conferences. Participants also indicated
the need for “more workshops” like this one. Overall, participants concluded that this workshop
presented “very helpful information” and that they would recommend this workshop to other
families and friends because, as one participant posited, “It works! It’s needed!”
7.9. Parent Orientation
Forty-one useable surveys were collected that related to the Parent Orientation. These
open ended surveys sought to enable Centric to ascertain the best way to stay in contact with
parents, as well as useful information they should know about the parents. As a result of the
survey, the majority of parents indicated that the best time to hold parent trainings would be in
the evenings, after school, although a few parents preferred mornings. Additionally, the majority
of participants indicated their willingness to work with other parents or students in an effort to
share their expertise. Topics that parents wanted more information about included: computers,
college readiness and educational programs, improving organizational skills, and math and
writing skills. In other words, there were a spectrum of interest from parents. Ultimately, parents
had a great interest in everything that was being discussed by the presenters. One participant
shared:
This seems to really come at an excellent time. XXXX [child’s
name] really needs to improve grades and he would give him lots
of things to get him involved with activities.
While few parents had additional information to share, most were really interested in learning
more about the program and were willing to contribute when and if they were called upon to do
so.
68
8. NON-PARTICIPANTS
Information from non-participants was collected initially as control group to help
evaluate the effectiveness of intervention programs. However, in the currently available dataset,
there is no information on non-participants’ academic achievement, attendance or disciplinary
actions, which prevent comparisons across groups. We summarize what information is available
about non-participants, focusing on students’ perceptions of the importance of college education
and cognitive attribution for success.
This dataset contains 257 students from a total of 5 affiliated institutions, including
Arthur High School, Saginaw High School, Thompson Middle School, Saginaw Arts and
Science School and Zilwaukee K-8 School. The distribution of the students in each institution is
summarized in Table 8. On average, students perceived college education as very important.
Students generally had moderate to high confidence in their likelihood of obtaining a college
degree and achieving success based on their current academic performance. Regarding the
cognitive attribution for success, students in this sample placed the most emphasis on obtaining a
college degree and other skills, followed by “luck” and “social support.” Motivation was rated as
only moderately important to achieving success. On average, students rated their leadership
abilities as moderate to high.
69
Table 8. Non-Participants’ Cognitions
Total
Arthur
Hill High
Saginaw
High
Thompson
Middle
Saginaw
Arts & Sci
Zilwaukee
K8
(1)
(2)
(3)
(4)
(5)
(6)
To what extent do you feel…
- your current level of academic performance will allow you to actually
earn a college degree someday?
3.94
3.92
3.71
4.28
3.92
3.87
- you can achieve a high level of success based on your current
academic performance?
3.98
4.04
3.98
4.04
3.78
3.97
How important is…
- it to you to earn a college degree?
4.71
4.65
4.65
4.82
4.78
4.69
- a college degree in helping you achieve a high level of success in a
career or job you may want to pursue?
4.87
4.83
4.86
4.94
4.92
4.83
- special training or skills in helping you achieve a high level of
success in a career or job you may want to pursue?
4.73
4.59
4.63
4.96
4.84
4.69
- being lucky in helping you achieve a high level of success in a career
or job you may want to pursue?
4.33
4.44
4.31
4.22
4.16
4.47
- having desire in helping you achieve a high level of success in a
career or job you may want to pursue?
3.51
3.71
3.53
3.71
3.00
3.31
- having someone to help you get where you want to be, in helping you
achieve a high level of success in a career or job you may want to
pursue?
4.15
4.21
4.26
3.73
4.43
4.18
- talking to others in helping you achieve a high level of success in a
career or job you may want to pursue?
4.05
4.23
4.04
4.17
3.73
3.89
How would you rate…
- your current leadership abilities?
4.05
3.85
3.74
3.89
3.88
3.91
Number of observations
257
78
52
51
37
39
70
9. SUMMARY AND IMPLICATIONS FOR FUTURE PRACTICES
The Centric Program is a comprehensive initiative, which includes various components
that aim to close the achievement gap for male students in middle school and high school,
especially those who are ethnic minority. Incorporating the principles of Universal Design for
Learning (UDL), the Centric Program offered students and their parents multiple activities and
interventions to enhance students’ academic achievement. The program’s implementation design
featured a defined cohort (experimental group), control population, quantitative and qualitative
assessment tools, and sought to achieve a positive measurable impact in a compressed period of
time.
The external evaluation team examined the impact of the Centric program after an
academic year of program implementation using data gathered from 5 affiliated institutions,
including Arthur Hill High School, Saginaw High School, Thompson Middle School, Saginaw
Arts and Science Academy and Ziwaukee K-8 School. The evaluation mainly focused on the
following components of Centric program: (1) Financial Incentives; (2) the Behavioral Health
Program; (3) the Winning Futures Program; (4) the Kaplan ACT Prep Program; (5) Parental
Engagement Survey; (6) Parental Workshops; (7) A Summary of Non-participant Data.
Based on the evaluation results showed in the previous sections, several promising
practices are elucidated that can be implemented and enhanced at other institutions that are
seeking to narrow the achievement gap for male minority students. Firstly, offering financial
incentives for students who participated the program and gained academic improvement is an
effective strategy to motivate students. The evaluation of this program showed that students who
gained more financial reward throughout the program received higher GPAs, experienced fewer
absences, and were found to have fewer missing assignment than students in the pre-intervention
period. Second, intervention programs should include components that help students learn to
better manage their emotions. Evaluation of the Behavioral Health Program results revealed that
students who had little experience being irritable and angry tended to perform better on reading
tests. Interventions improving students’ emotional management skills could help students to
better manage stress, resolve challenges and perform better at school. Third, programs should
recognize the negative consequences of students’ absences and tardiness on academic
achievement and design remedies to reduce such occurrences. Analysis of the Kaplan Act Prep
71
Program show that students’ absences, tardiness, and rates of missing assignments were all
negatively associated with academic outcomes.
Apart from the promising practices that emerged from the evaluation, we also found it
difficult to conduct more systematic analysis based on the data collected. Several data collection
and research design improvements could be made in the future to allow for more comprehensive
analysis. These included:
Participant Recruitment
In the current dataset, there is a large imbalance between the number of participants who
took the pre-survey and the number of participants who took the post-survey. Therefore, we are
not able to match pre- and post-intervention test results up to generate convincing comparisons.
Also, for some interventions such as the Winning Futures Program and Parental Workshops,
sample sizes were too small to allow for systematic quantitative analysis. In the future, efforts
should be made to improve participant recruitment process.
Experimental Design
To evaluate the effectiveness of intervention program, the prescribed approach is to
implement an experimental or quasi-experimental design wherein which program participants’
outcomes can be compared with a group similar in every way except for their participation in the
intervention. Although the current design of Centric Program includes selection of an
experimental cohort and control group, the information collected from non-participants (control
group) is incompatible with that from experimental group. Identical survey instruments should
be used for each group in order to meet the basic requirements and assumptions of a true
experimental design.
Measurement Concerns
A variety of measures were utilized examining participants’ characteristics and for
evaluating the effectiveness of the interventions. However, there were several applications of
measures that presented barriers to effective inference. First, some measures were not always
appropriate to the task or rooted in established theoretical frameworks. For example, to measure
students’ behavioral and health strategies, a literature search should be conducted to locate the
most suitable measure that could serve this goal. Second, measures for evaluating the
effectiveness of the intervention should track changes in students’ progress throughout the
intervention. For example, in the survey evaluating the Behavioral Health Program intervention,
72
students were asked a set of Yes/No questions about their previous experiences. Such questions
are unlikely to capture changes in students in a short period of time. Finally, measures should be
developed in a way that fits the target population best. In the survey evaluating the parental
workshops, several open-ended questions were included to gather qualitative information from
parents. However, most parents were not able or willing to provide detailed, written responses.
Interviews or focus groups may be a better approach for gathering such information particularly
for the parental population.
To sum up, the implementation of the Centric programs have generated several promising
results that could help innovate in-school and out-of-school practices with males of color in K12
school settings. It also serves as baseline success for the Centric Program to narrow the
achievement gap in the school district and ultimately become a national model.
73
10. REFERENCES
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago, IL: Aldine.
Jorgensen, D. L. (1989). Participant observation: Methodology for human studies. Newbury,
CA: Sage.
Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. Newbury, CA: Sage.
Mason, J. (1996). Qualitative research. Newbury, CA: Sage. Miles, M. B., & Huberman, A. M. (1984).
Qualitative data analysis: A source book of new methods. Beverly Hills, CA: Sage.
Morgan, D. L. (1998). The focus group guidebook. Newbury, CA: Sage.
Scott, D. (1995). Conditions related to the academic performance of African American students
at Virginia Polytechnic Institute and State University. Unpublished doctoral dissertation,
Virginia Polytechnic Institute and State University.
