Content uploaded by Jeffrey F. Milem
Author content
All content in this area was uploaded by Jeffrey F. Milem on Aug 27, 2015
Content may be subject to copyright.
From the SelectedWorks of Nolan L. Cabrera
January 2014
Missing the (student achievement) forest for all the
(political) trees: Empiricism and the Mexican
American Studies controversy in Tucson
Contact
Author
Start Your Own
SelectedWorks
Notify Me
of New Work
Available at: hp://works.bepress.com/nolan_l_cabrera/24
http://aerj.aera.net
Journal
American Educational Research
http://aer.sagepub.com/content/51/6/1084
The online version of this article can be found at:
DOI: 10.3102/0002831214553705
2014 51: 1084 originally published online 15 October 2014Am Educ Res J
Nolan L. Cabrera, Jeffrey F. Milem, Ozan Jaquette and Ronald W. Marx
Empiricism and the Mexican American Studies Controversy in Tucson
Missing the (Student Achievement) Forest for All the (Political) Trees:
Published on behalf of
American Educational Research Association
and
http://www.sagepublications.com
can be found at:American Educational Research JournalAdditional services and information for
http://aerj.aera.net/alertsEmail Alerts:
http://aerj.aera.net/subscriptionsSubscriptions:
http://www.aera.net/reprintsReprints:
http://www.aera.net/permissionsPermissions:
What is This?
- Oct 15, 2014OnlineFirst Version of Record
- Nov 12, 2014Version of Record >>
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Missing the (Student Achievement)
Forest for All the (Political) Trees:
Empiricism and the Mexican American
Studies Controversy in Tucson
Nolan L. Cabrera
Jeffrey F. Milem
Ozan Jaquette
Ronald W. Marx
University of Arizona
The Arizona legislature passed HB 2281, which eliminated Tucson Unified
School District’s (TUSD’s) Mexican American Studies (MAS) program, argu-
ing the curriculum was too political. This program has been at the center of
contentious debates, but a central question has not been thoroughly exam-
ined: Do the classes raise student achievement? The current analyses use
administrative data from TUSD (2008–2011), running logistic regression
models to assess the relationship between taking MAS classes and passing
AIMS (Arizona state standardized tests) and high school graduation.
Results indicate that MAS participation was significantly related to an
NOLAN L. CABRERA is an assistant professor in the Department of Educational Policy
Studies & Practice at the University of Arizona, 1430 E. Second Street, PO Box
210069, Tucson, AZ 85705; e-mail: ncabrera@email.arizona.edu. His research
focuses on Whiteness formation and racial dynamics on college campuses.
JEFFREY F. MILEM is the Ernest W. McFarland Distinguished Professor of Leadership for
Educational Policy and Reform at the Department of Educational Policy Studies and
Practice at the University of Arizona. His current research focuses on the ways in
which colleges and universities can be organized to enhance equity, access, and suc-
cess for all students; the racial context within higher education; and the relationship
between how colleges and universities organize themselves and student outcomes
and faculty role performance.
OZAN JAQUETTE is an assistant professor in the Department of Educational Policy &
Practice at the University of Arizona. His research program analyzes how postsecond-
ary institutions change their behavior to generate enrollments from desired student
populations.
RONALD W. MARX is a professor of educational psychology and dean of education at
the University of Arizona. His research focuses on how classrooms can be sites for
learning that is highly motivating and cognitively engaging.
American Educational Research Journal
December 2014, Vol. 51, No. 6, pp. 1084–1118
DOI: 10.3102/0002831214553705
Ó2014 AERA. http://aerj.aera.net
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
increased likelihood of both outcomes occurring. The authors discuss these
results in terms of educational policy and critical pedagogy as well as the
role academics can play in policy formation.
KEYWORDS: Mexican American Studies, ethnic studies, program assessment,
HB 2281
On April 26, 2011, nine students in the Tucson Unified School District
(TUSD) took over the school board meeting to protest the potential dis-
mantling of the Mexican American Studies (MAS) program. The dramatic
images of students chaining themselves to school board members’ chairs
made headlines nationally and have been used by supporters and detractors
of the program alike. About 500 community members and over 100 police
officers attended the subsequent school board meeting (Cabrera, Meza,
Romero, & Rodriguez, 2013). The controversy surrounding the MAS program
broiled for months, and emotions were charged. Numerous threats followed,
including a Youtubeävideo claiming the way to deal with student protesters
was to ‘‘shoot them in the head’’ (Cabrera et al., 2013, p. 10). These incidents
occurred after the passage of Arizona’s HB 2281 (now A.R.S. § 15-112), which
allowed the state superintendent of public instruction to withhold 10% of
state funding if he found a district offered classes that
1) Advocate ethnic solidarity rather than treating pupils as individuals,
2) Promote resentment toward a race or class of people,
3) Are designed primarily for pupils of a particular ethnic group, or
4) Promote the overthrow of the U.S. government (Prohibited courses and clas-
ses, 2010, p. 1).
Although he never attended a MAS class or conducted an audit of these
courses, State Superintendent Tom Horne found TUSD out of compliance
with this statute. He asserted that modifications of the classes to align the
program with the law were impossible and, therefore, ‘‘the only way in
which compliance can be effective within the next 60 days is by elimination
of the Mexican American Studies program’’ (Horne, 2010, p. 2).1The contro-
versy received extensive coverage from the New York Times, Fox News, and
CNN, and it was even a segment on The Daily Show with Jon Stewart
(Cabrera et al., 2013). Within the education research community, this issue
was the subject of an invited presidential session at the 2012 American
Educational Research Association meeting, two resolutions by American
Educational Research Association (2012a, 2012b), and a literature review
regarding the value of ethnic studies commissioned by the National
Education Association (Sleeter, 2011).
Commentators across the country debated the broader questions sur-
rounding the MAS controversy, including, ‘‘To what extent can a
Missing the (Student Achievement) Forest
1085
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
non-Eurocentric curriculum and pedagogy be sanctioned as ‘legitimate edu-
cation’? Additionally, can critical approaches to oppression be part of public
secondary education?’’ (Cabrera et al., 2013, p. 20). While the program sup-
porters pointed to the increased educational achievement that stemmed
from MAS participation (Romero, 2008; Romero, Arce, & Cammarota, 2009;
Save Ethnic Studies, 2011), this issue was frequently lost in the media cover-
age, overshadowed by the politics (e.g., Huicochea, 2011b; Martinez, 2011).
Additionally, the relationship between the courses and student achievement
was irrelevant to those seeking to end the program, as they believed the MAS
program had no place in public education (Horne, 2010; Huppenthal, 2011).
Interestingly, all sides of this contentious issue agree that student
achievement should be a focal point of public education, although they
have put forth differing rationales. For example, the current state superinten-
dent claims that student achievement is important because Arizonans should
receive returns to their educational investments and having an educated
populace is key to the state’s economic prosperity. He further states, ‘‘I am
especially committed to driving our education policies and practices with
quality research that have demonstrated proven results’’ (Huppenthal,
n.d.). Supporters of the program also argue that student achievement is
important, but they frame the issue from a critical pedagogy approach
(Freire, 2000). Within this paradigm, changing a student’s relationship to
school through Critically Compassionate Intellectualism (CCI) should lead
to increased student achievement (Cammarota & Romero, 2014). The ulti-
mate goal is developing students as educated, critically engaged citizens
who are committed to transforming oppression within their communities
(see Conceptual Underpinnings of MAS section for an elaborated
discussion).
While both sides of the issue profess that student achievement should be
of the upmost importance, the racial politics of the controversy have pushed
students’ academic success to the background. Thus, a fundamental question
about MAS in TUSD was lost within this highly contentious political debate:
What effect, if any, did these classes have on student academic achievement?
The bulk of prior research on this subject was descriptive and therefore not
able to determine impact (e.g., Cabrera, 2012; Cappellucci et al., 2011;
Department of Accountability and Research, 2011a, 2011b; Romero, 2008).
The current research addresses this limitation by assessing the relationship
between MAS course participation and student achievement via multivariate
analyses using TUSD’s student-level, administrative data.
Relevant Literature and Theory
The TUSD MAS program follows in the intellectual and activist lineage of
the ethnic studies programs created in the 1960s (Cammarota & Romero,
2014). Within this framework, the more Latina/o students see themselves
Cabrera et al.
1086
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
and their experiences reflected in the curriculum, the more likely they are to
be engaged in school, leading to greater educational success. Additionally,
the MAS program was developed from a critical theory paradigm (e.g.,
Freire, 2000). This perspective means the MAS version of ethnic studies
was more than celebrating racial/ethnic difference or positive identity devel-
opment but also examining, critiquing, and fighting systemic oppression
(Cammarota & Romero, 2014). The high school diploma becomes important
in this context because it helps stem the school-to-prison pipeline, provides
students economic opportunities, paves pathways to higher education, and
can improve the material conditions of the community (Cammarota &
Romero, 2014; Duncan-Andrade & Morrell, 2008).
The Development of Ethnic Studies
The MAS program began with a critique of traditional forms of curricula
where minority experiences and voices were noticeably absent. Ronald
Takaki (1993) highlighted this issue in A Different Mirror when he described
his choice for the book’s title.
While the study of the past can provide collective self-knowledge, it
often reflects the scholar’s particular perspective or view of the world.
What happens when historians leave out many of America’s peoples?
What happens, to borrow the words of Adrienne Rich, ‘‘when some-
one with the authority of a teacher’’ describes our society, and ‘‘you
are not in it’’? Such an experience can be disorienting—‘‘a moment of
psychic disequilibrium, as if you looked into a mirror and saw noth-
ing.’’ .America does not belong to one race or one group, the peo-
ple in this study remind us, and Americans have been consistently
redefining their national identity from the moment of first contact
on the Virginia shore. By sharing their stories, they invite us to see
ourselves in a different mirror. (pp. 16–17)
These ‘‘different mirrors’’ created by Takaki (1993) and other scholars,
including the MAS creators, were facilitated by the growth of ethnic studies
programs initiated in the late 1960s in colleges and universities in the United
States. Student activists were inspired by the civil rights and antiwar move-
ments to seek transformational changes in higher education in the United
States, including increased access for racial minorities, the hiring of more fac-
ulty of color, and the creation of ethnic studies programs (Hu-DeHart, 1993;
Rojas, 2006). Hu-DeHart (1993) argued that this movement to create ethnic
studies programs signified ‘‘the beginning of multicultural curriculum reform
in higher education’’ (p. 51).
Historically, ethnic studies scholars believed that a key purpose of the
field was to challenge the dominant discourse and paradigms of traditional
academic disciplines through interdisciplinary scholarship (Delgado &
Stefancic, 2001). Frequently rooted in critical approaches to education
(Freire, 2000), ethnic studies scholars realized the importance of perspective
Missing the (Student Achievement) Forest
1087
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
in shaping understanding of the world, that these perspectives are always
limited, and that they are positioned in relationship to power (Hu-DeHart,
1993). Thus, this historical/academic lineage contextualized the MAS pro-
gram’s focus on issues of power and oppression, especially racial
(Cammarota & Romero, 2014).
Ethnic studies is more than just a critical examination of power and per-
spective. Christine Sleeter (2011) offered an assessment of the character of
ethnic studies by suggesting five consistent themes of the field:
1) explicit identification of the point of view from which knowledge emanates,
and the relationship between social location and perspective;
2) examination of U.S. colonialism historically, as well as how relations of colo-
nialism continue to play out;
3) examination of the historical construction of race and institutional racism, how
people navigate racism, and struggles for liberation;
4) probing meanings of collective or communal identities that people hold; and
5) studying one’s community’s creative and intellectual products, both historic
and contemporary. (p. 3)
Each of these components was integrated in the overall structure of the MAS
program (Cammarota & Romero, 2014). There have been several studies
regarding the impacts ethnic studies courses have on student development,
but there are stark differences between the way higher education and K-12
scholars approach this issue.
The Impact of Ethnic Studies Courses
Scholarship in higher education using survey-based research has docu-
mented the positive impact that increased exposure to diverse information
and ideas has on a range of important social, cognitive, and democratic out-
comes for college students of all racial/ethnic backgrounds (e.g., antonio,
2001; Astin, 1993; Bowman, 2010a, 2010b; Chang, 2002; Denson & Chang,
2009; Engberg, 2004; Gurin, Dey, Hurtado, & Gurin, 2002; Hurtado, Dey,
Gurin, & Gurin, 2003; Johnson & Lollar, 2002; Laird, 2005; Laird, Engberg,
& Hurtado, 2005; Milem, 1994; Milem & Umbach, 2003; Milem, Umbach, &
Liang, 2004; Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1996; Tsui,
1999). These studies are limited because they do not differentiate between
students taking traditional ethnic studies courses (e.g., examinations of rac-
ism) and those taking ones that include a diversity component (e.g., readings
by scholars of color). Thus, it is unclear to what extent a critical perspective
that centers power relations is related to student development. Moreover,
none of this research addressed whether taking ethnic studies courses
improved students’ academic performance.
Sleeter’s (2011) analysis of the literature regarding ethnic studies courses
in K-12 settings revealed that TUSD was the only school district in the United
Cabrera et al.
1088
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
States to have ‘‘a full-fledged ethnic studies program’’ (p. 7), and this pro-
gram was soon eliminated. Hence, the body of evidence documenting the
relationship between these courses in K-12 schools and student outcomes
is much more limited. Some of the research Sleeter (2011) cited argued
that ethnic studies curricula foster a positive relationship between racial/
ethnic identity and academic achievement among students of color (e.g.,
Carter, 2008; O’Connor, 1997). Additionally, Sleeter cited studies demonstrat-
ing that engagement increased when students read literature written by
authors of the students’ racial/ethnic background (e.g., Bean, Valerio,
Senior, & White, 1999; Brozo & Valerio, 1996; Copenhaver, 2001).
Moreover, Sleeter found that ethnic studies curricula led to enhanced literacy
skills (e.g., Krater & Zeni, 1995; Krater, Zeni, & Cason, 1994; C. D. Lee, 1995,
2001, 2006, 2007; Lomawaima & McCarty, 2006; McCarty, 1993; Rickford,
2001) and higher achievement and more positive attitudes toward learning
in math and science among Native American students (Lipka, 1991; Lipka,
Hogan, et al., 2005; Lipka, Sharp, Brenner, Yanez, & Sharp, 2005;
Matthews & Smith, 1994). Finally, Sleeter found that ethnic studies in social
studies frequently led to enhanced academic achievement and sense of
agency among students (Cammarota, 2007; Cammarota & Romero, 2009;
Lewis, Sullivan, & Bybee, 2006; Romero et al., 2009; Tyson, 2002).
The research Sleeter (2011) reviewed was informative but also limited in
scale (i.e., single site), had no comparison group (i.e., students who did not
take ethnic studies courses), had small sample sizes, and tended to be qual-
itative. The K-12 research Sleeter reviewed did show some relationships
between taking ethnic studies courses and increased academic achievement.
The bulk of the research Sleeter reviewed was published after the MAS pro-
gram was created; thus, the MAS design was drawn primarily from a theoret-
ical argument. There is now some justification in the empirical literature to
support a positive relationship between ethnic studies participation and stu-
dent achievement, but to date there has not been an assessment of these
courses like the one conducted in this study. Simply put, quantitative anal-
yses of large-scale data that explore the impact of ethnic studies on academic
outcomes do not exist.
Context of the Study
Development of the MAS Program
In 2002, TUSD deputy superintendent Dr. Becky Montan
˜o appointed
Augustine Romero as head of the district’s Hispanic Studies Department
and assigned him the responsibility of addressing the White/Latina/o
achievement gap as mandated through No Child Left Behind (Cammarota
& Romero, 2014). This department was soon changed to the Mexican
American/Raza Studies Department, and Romero, with his colleague
Missing the (Student Achievement) Forest
1089
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Dr. Julio Cammarota, developed the Social Justice Education Project in one
TUSD classroom. This program identified the lowest performing students in
the school and engaged them in participatory action research on the belief
that it would develop in them a sense of empowerment by encouraging
them to be social change agents (Cammarota & Romero, 2006a, 2006b).
These classes were designed to be 1 year in length (two consecutive semes-
ters) and counted as core social studies requirements. The first cohort had
only 17 students, of whom 16 graduated from high school (Romero,
2008). Over time, the Social Justice Education Project was scaled up to
include more classes and schools (Cammarota & Romero, 2014). By the
2005–2006 academic year, under the umbrella of ‘‘Mexican American
Studies,’’ the program expanded to four schools and offered language arts
classes, which also counted as core class requirements. Participation in the
MAS courses was voluntary, but for the cohorts included in our analyses
(2008–2011), approximately one-fifth of all students who attended schools
that offered MAS courses took at least one class.
Conceptual Underpinnings of MAS
MAS courses were intended to depart from the way most classes in
TUSD were taught. The approach was rooted in the work of Paulo Freire
(2000, 2008), especially the development of conscientizacxa˜o: the combina-
tion of critical consciousness, self-reflection, and engaging in anti-
oppressive, collective action. Students learn to read the word and the world
(Freire & Macedo, 1987) by situating themselves as historical subjects (Freire,
2008), seeing themselves as potential agents of social change, and develop-
ing praxis (Freire, 2000) while being critically self-reflective (Freire &
Macedo, 1987).
To adapt this approach to the experiences of low-income Latinas/os in
Tucson, the developers used the concept of authentic caring (Valenzuela,
1999). Authentic caring ‘‘means that the material, physical, psychological,
and spiritual needs of youth will guide the education process’’
(Valenzuela, 1999, p. 110). This version of caring requires educators to
move beyond developing students’ cognitive abilities, although this is
important. Rather, and specific to marginalized youth, teachers engage
with the structures of oppression that inform the experiences of their stu-
dents (Romero, 2008). As Valenzuela (1999) argues, it is a pedagogy that
‘‘deliberately [brings] issues of race, difference, and power into central focus’’
(p. 109).
To enact a curriculum derived from Freire and a position of authentic
caring, the program developers asked MAS educators not to see students
as blank slates but as capable people who were cocreators of knowledge
(Freire, 2000). This required educators to recognize students’ funds of
knowledge (Gonza
´lez, Moll, & Amanti, 2005), or the knowledge students
Cabrera et al.
1090
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
bring into the classroom gained through their families and cultural practices.
Identifying funds of knowledge in this context requires educators to blur the
lines between the school and community, developing relationships with stu-
dents’ families and making efforts to bring the community into the school
(Cammarota & Romero, 2014).
Finally, the MAS curriculum incorporated components of Critical Race
Theory (CRT; Delgado & Stefancic, 2001), which places interrogations of
race and racism at the center of the curriculum, as the developers intended
to racismize the classroom (Romero et al., 2009). However, the process of
racismization was not meant to ignore or even downplay other forms of
oppression (e.g., sexism or homophobia). Rather, this entailed a pedagogy
that centered racism in classroom discussions while concurrently examining
other forms of oppression in students’ lived experiences (Cammarota &
Romero, 2014).
Cammarota and Romero (2014) labeled the pedagogy Critically
Compassionate Intellectualism (see Figure 1). This approach required teach-
ers to develop the critical consciousness of the students, make meaningful
connections with students and their families, push students to see them-
selves as intellectuals, and help students become agents of change.
This shift in a student’s relationship to school was intended to increase
academic engagement, which in turn was expected to increase academic
Figure 1. Visual model of Critically Compassionate Intellectualism.
Note. Reproduced by permission of A. F. Romero (2013, April 5).
Missing the (Student Achievement) Forest
1091
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
performance on traditional metrics (e.g., standardized tests and graduation).
This is an interesting departure from traditional Freirian pedagogy and CRT
where standardized test scores are either irrelevant or oppressive. By con-
trast the CCI is a critical approach to education where the development of
basic academic competencies is integrally related to the process of social
transformation (Delpit, 2012).
MAS and Student Achievement
Prior research on the MAS program on student outcomes has been
inconclusive. Some studies indicated that MAS was effective in raising stu-
dent academic achievement, but these were either descriptive or qualitative
(e.g., Cabrera, 2012; Department of Accountability and Research, 2011a,
2011b; Romero, 2008; Romero et al., 2009; Save Ethnic Studies, 2011).
Scott (2011, as cited in Huicochea, 2011a) used descriptive analyses of
TUSD student-level data and found MAS to be effective in raising academic
achievement, but he argued these increases were small, similar to the effect
of participating in extracurricular activities. One prior study that utilized
a form of multivariate analysis to assess the impact of MAS on Arizona’s
Instrument to Measure Standards (AIMS) test scores found no significant
impact when comparing MAS students to all other high school students in
the state (Franciosi, 2009). However, students outside of TUSD are not an
appropriate comparison group.
TUSD’s Desegregation Plan
An initial version of this study was conducted at the request of Dr. Willis
Hawley, the special master appointed by U.S. District Judge David Bury in
the long-standing but unresolved TUSD desegregation case. Although the
desegregation case was a different legal and policy issue than HB 2281, it
touched on issues that were closely associated with MAS. Judge Bury
directed Dr. Hawley to develop a unitary status plan (USP) for TUSD that
would chart the path for getting out from under the 40-year-old desegrega-
tion order. The final USP entailed both integrating the schools and address-
ing persistent achievement gaps in TUSD between White and Latina/o stu-
dents. Black students were also part of this desegregation case, but African
American Student Services did not offer classes equivalent to MAS. As part
of his deliberations regarding the USP, Dr. Hawley wanted to know what
relationship, if any, enrollment in MAS courses had with subsequent student
achievement. He requested that a research team from the University of
Arizona, in collaboration with officials from TUSD, conduct the analyses.
The research team had 6 weeks to create an analytical strategy, gain
institutional review board approval, secure the data from TUSD, prepare
them for analyses, conduct the analyses, prepare the report, and submit it
to the special master by June 20, 2012 (Cabrera, Milem, & Marx, 2012).
Cabrera et al.
1092
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
When the entire set of recommendations from the special master was sub-
mitted to the court, it included the research team’s report, which was
released to the public. As is often the case in such matters, the report was
subjected to comment and critique by others, including a member of the
school board (Stegeman, 2013) and journalists (Acun
˜a, 2012; Herreras,
2012; MacEachern, 2012, 2013; Maxwell, 2012). With these critiques in
mind, we have continued to refine and develop our analytical approach.
Methods
Empirical Strategy
From a program evaluation perspective, analyzing the effect of MAS par-
ticipation on educational achievement is analogous to evaluating the effect
of class size on educational achievement (Angrist & Lavy, 1999; Hoxby,
2000) or student aid on college attendance and completion (DesJardins,
2005). Each of these examples estimates the causal effect of a policy inter-
vention on a dependent variable of interest. The concept of a counterfactual
is critical for program evaluations of causal effects. For each subject who par-
ticipates in a program, the counterfactual is what the outcome would have
been if the subject had not participated in the program (Khandker,
Koolwal, & Samad, 2010). Using a counterfactual approach, the average
causal effect of MAS participation is estimated by calculating the difference
between the actual outcome for students who participated in MAS and
what the outcome would have been had they not participated in MAS.
Unfortunately, counterfactuals do not exist in reality.
In cross-sectional analyses such as this one, students who do not partic-
ipate in the program are used as counterfactuals for students who do
(Khander et al., 2010). Random assignment is the gold standard for creating
comparison groups in cross-sectional analyses, but random assignment was
not possible for the MAS program, as participation was voluntary. When
individuals self-select into the program, rather than being randomly
assigned, estimates of the causal effect may be biased due to the omission
of relevant covariates from the model. Omitted variable bias occurs when
two conditions are met: (1) the omitted variable affects the outcome, and
(2) the omitted variable is correlated with the independent variable of inter-
est. When omitted variable bias occurs, the estimated causal effect is partially
picking up the effect of the omitted variable on the outcome.
In the absence of random assignment, empirical program evaluations
often attempt to overcome omitted variable bias by including in the model
all covariates that satisfy the conditions of omitted variable bias (Stock &
Watson, 2011). Our regression models attempted to include as many covari-
ates as possible that potentially affected the outcome of interest and were
correlated with MAS participation, acknowledging the limitations of using
Missing the (Student Achievement) Forest
1093
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
administrative data. Alternative approaches, such as instrumental variables
or regression discontinuity, that attempt to isolate exogenous variation in
program participation were not feasible for the present study. Given that
we attempted to eliminate omitted variable bias using covariates, interpret-
ing regression coefficients as causal effects rests on the assumption that there
are no omitted variables that affect the outcome and have a systematic rela-
tionship with programmatic participation. It is usually unrealistic for pro-
gram evaluation studies to satisfy this assumption using control variables
(Angrist & Pischke, 2009; Hoxby, 2000), and in particular this study omits
several potentially important control variables because the only available
data were administrative. Therefore, our results may suffer from omitted var-
iable bias and should not be considered true causal effects. However, it is
important to note that applying matching methods such as propensity score
matching to our analysis would suffer from the same limitation.2
We used logistic regression to analyze the binary dependent variables of
whether the student graduated from high school and passed AIMS tests after
initial failure (an explanation of AIMS is in a subsequent section). Equation
(1) presents the logistic regression model, where Yiis the dependent vari-
able, Xirepresents participation in MAS, and Wirepresents the matrix of
controls (demographic characteristics, prior academic achievement, and
school-level context).
PY
ijXi;W
i
ðÞ5eðbXi1W0
id1uiÞ
11eðbXi1W0
id1uiÞð1Þ
Therefore, the expression PY
ijXi;Wi
ðÞrepresents the probability of having
a certain outcome (e.g., graduation), given MAS participation and control
variables, and brepresents the relationship between MAS participation on
the probability of having a certain outcome. Models used robust standard
errors to relax the assumption of homoscedasticity (Stock & Watson,
2011). Finally, we modeled the probability of our outcomes of interest given
differing levels of programmatic participation (e.g., one class, two classes,
three classes, etc.).
Data and Sample
Analyses were based on student-level administrative data in TUSD.
TUSD administrators provided access to de-identified student data on
courses taken, state standardized test scores, school services utilized (e.g.,
Gifted and Talented Education [GATE] programs), and basic demographic
data (e.g., race, median income of the census block in which students
resided). Using these data sources, we developed a student-level dataset
for analysis. The sample consisted of TUSD student cohorts who would
have graduated in the 2008, 2009, 2010, and 2011 cohorts (N= 26,022).
Cabrera et al.
1094
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
For example, for the 2008 cohort we have data beginning in the 2004–2005
academic year, when these students were freshmen. We ran analyses for all
students (2008–2011) and also separately by cohort. These four cohorts were
chosen because they provided the most complete student data as well as
representing times when MAS participation peaked.
As MAS was offered to juniors and seniors, students who did not attend
TUSD as juniors and seniors were excluded from analyses. Including stu-
dents who did not reach junior or senior status in the analysis sample would
lead to upward bias in the coefficients on MAS participation because those
who reach junior and senior status generally have higher levels of educa-
tional achievement than students who dropped out as sophomores or fresh-
men. Additionally, we included only students who attended schools that
offered MAS because students outside of these schools had no opportunity
to participate in the program (n= 8,382). This approach has been critiqued
as being unnecessarily restrictive (Stegeman, 2013), so we also conducted
analyses that included participants at all TUSD schools (n= 16,917). The
results presented in this study include both but are primarily focused on
the analysis of MAS schools only.
Variables
Independent Variable of Interest
The independent variable of interest was a binary measure of student
participation in the MAS program (see Table 1). A small number of freshmen
and sophomores enrolled in MAS courses (n= 84, 1.0% of the sample), and
these students were dropped from analyses. For the purposes of this study,
we defined nonparticipants as all students attending schools offering MAS
classes but who never took one of the courses. Students who enrolled in
one or more MAS courses were defined as participants. MAS participants
enrolled in anywhere from one to nine MAS courses, with students taking
an average of three courses. There was considerable diversity in terms of
the number of MAS classes students completed. Therefore, we ran two
sets of analyses: one that sought to determine whether taking any MAS clas-
ses affected student achievement and a second that sought to determine
whether the likelihood of student success changed as participation
increased.
Dependent Variables
The conceptual underpinnings of the program presented previously
argue that participation in MAS should positively affect academic achieve-
ment by changing student academic orientation, leading to greater engage-
ment, and subsequently increasing performance on standard measures of
academic success (see Figure 1). Within this context, we evaluated the
Missing the (Student Achievement) Forest
1095
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
efficacy of MAS participation against two outcomes valued by Arizona poli-
cymakers: (1) a binary variable indicating whether a student passed the AIMS
tests after initial failure and (2) a binary variable indicating whether a student
Table 1
Description and Measures for Variables Used in Regression Analyses
Dependent variables
Graduate Student graduated from high school at any point (1 =
Yes; 0 = No)
Arizona’s Instrument to Measure
Standards (AIMS), writing
Student passed the high school AIMS writing test after
initially failing (1 = Yes; 0 = No)
AIMS, reading Student passed the high school AIMS reading test after
initially failing (1 = Yes; 0 = No)
AIMS, math Student passed the high school AIMS math test after
initially failing (1 = Yes; 0 = No)
AIMS, all subjects Student passed all the high school AIMS writing tests
after initially failing at least one (1 = Yes; 0 = No)
Independent variables
Gender 1 = Female; 0 = Male
African American 1 = Yes; 0 = No
Latina/o 1 = Yes; 0 = No
White 1 = Yes; 0 = No
Native American 1 = Yes; 0 = No
Asian American 1 = Yes; 0 = No
Free/reduced-price lunch Participated in the Federal Meals program while
enrolled in Tucson Unified School District (1 = Yes; 0 =
No)
Census block median income Median income of the census block in which the student
resides (in thousands)
English language learner (ELL) Student was at some point classified as ELL in high
school (1 = Yes; 0 = No)
Gifted and Talented Education
(GATE)
Student was at some point classified as GATE in high
school (1 = Yes; 0 = No)
Special education (Special Ed.) Student was at some point classified as Special Ed. in
high school (1 = Yes; 0 = No)
Transfer Number of times transferred school 9th and 10th grade
School attended: A 1 = Yes; 0 = No
School attended: B 1 = Yes; 0 = No
School attended: C 1 = Yes; 0 = No
School attended: D 1 = Yes; 0 = No
School attended: E 1 = Yes; 0 = No
School attended: F 1 = Yes; 0 = No
9th-grade GPA (weighted) Range: 0.0–5.0
10th-grade GPA (weighted) Range: 0.0–5.0
10th-grade AIMS: mathematics Scaled score
10th-grade AIMS: reading Scaled score
10th-grade AIMS: writing Scaled score
Mexican American Studies (MAS) Student completed at least one semester credit of MAS
(1 = Yes; 0 = No)
MAS credits Number of MAS semesters a student completed (0 = 0;
1 = 1; 2 = 2; 3 = 3 or 4; 4 = 5 or more)
Note. GPA = grade point average.
Cabrera et al.
1096
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
graduated from high school. All Arizona high school students are required to
take the AIMS test in 10th grade as a graduation requirement. The test con-
sists of subject components in math, reading, and writing. There is a science
test as well, but it was not implemented until 2008 (Arizona Department of
Education, 2011). As the majority of the students in the sample never took
the test, we did not include it in the analysis. Students who fail any subject
of the test must retake it in subsequent years. Because MAS enrollment was
restricted to juniors and seniors, it was possible to analyze the relationship
between MAS participation and AIMS performance after initial failure on
the AIMS test; however, it was not possible to evaluate the relationship
between MAS participation and AIMS performance for students who passed
the AIMS in 10th grade because 10th graders were not yet eligible for MAS
participation. Students who passed a subject test in 10th grade and later
retook the test (e.g., to qualify for a Regents Scholarship) were excluded
from analyses.
We created four dichotomous dependent variables for AIMS test passage
defined as whether a student passed the AIMS test(s) s/he initially failed by
senior year. This included the three subject tests as well as a variable for stu-
dents who failed any of the AIMS tests initially and eventually passed all of
them. We used a dichotomous measure of passing AIMS rather than a contin-
uous measure of AIMS score because passing these tests is a prerequisite to
graduation. The high school graduation dependent variable was a dichoto-
mous measure of whether the student graduated from high school, including
those who graduated with a high school diploma in 5 years (see Table 1 for
measures and Table 2 for descriptive statistics). We hoped to also analyze the
relationship between MAS participation and postsecondary enrollment;
however, the data from the TUSD senior (self-report) survey had high levels
of missing data, and the accuracy was questionable. Additionally, Pima
Community College, the primary postsecondary destination for TUSD stu-
dents, did not subscribe to the National Student Clearinghouse until 2012.
Therefore, we could not conduct this analysis.
Control Variables
We attempted to include all covariates that affected the outcome variable
of interest (i.e., AIMS passing or high school graduation) and were correlated
with MAS participation. Measures of prior academic achievement satisfied
both conditions of omitted variable bias because prior academic achieve-
ment affects future educational outcomes and prior academic achievement
may be correlated with MAS participation. Academic achievement covariates
were measured prior to junior year; otherwise variation in these measures
could be caused by MAS participation. We therefore created measures of
grade point average (GPA) in 9th grade and GPA in 10th grade using
a ‘‘weighted’’ version of GPA that assigned higher grade values to advanced
Missing the (Student Achievement) Forest
1097
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Table 2
Descriptive Statistics by Cohort Group
All Cohorts (n= 8,382) 2008 (n= 1,792) 2009 (n= 2,130) 2010 (n= 2,106) 2011 (n= 2,354)
MAS Non-MAS MAS Non-MAS MAS Non-MAS MAS Non-MAS MAS Non-MAS
n= 1,707 n= 6,675 n= 448 n= 1,344 n= 408 n= 1,722 n= 403 n= 1,703 n= 448 n= 1,906
Dependent variables
Graduate 82.2%*** 73.8%*** 88.4%*** 72.5%*** 82.4%*** 72.9%*** 82.4%*77.2%*75.9% 72.6%
AIMS writing (passed) 91.0%*** 85.5%*** 93.1%*** 84.0%*** 89.2%** 84.3%** 92.0%** 86.7%** 89.5% 86.8%
AIMS reading (passed) 90.5%*** 85.2%*** 93.0%*** 82.6%*** 88.0%*84.2%*89.3% 86.7% 91.3%** 86.7%**
AIMS math (passed) 83.3%*** 78.0%*** 89.5%*** 78.2%*** 84.8%** 78.2%** 81.3% 79.3% 77.5% 76.6%
AIMS passed all 78.7%*** 74.0%*** 84.3%*** 73.2%*** 77.6% 73.4% 78.1% 75.7% 74.3% 73.7%
Independent variables
Gender (female) 54.5%*** 46.8%*** 55.8%** 44.7%** 51.5% 45.6% 58.1%** 48.0%** 52.9% 48.2%
African American 3.2% 8.7% 3.1% 7.9% 3.2% 8.2% 4.5% 7.7% 2.2% 10.7%
Latina/o 84.8%*** 55.7%*** 86.2%*** 57.4%*** 84.1%*** 55.1%*** 83.4%*** 56.5%*** 85.5%*** 54.5%***
White 7.2%*** 28.5%*** 7.1%** 28.3%** 6.1%** 28.7%** 7.2%** 29.8%** 8.3%** 27.4%**
Native American 4.1% 3.9% 3.3% 4.2% 5.6% 4.4% 5.0% 3.2% 2.7% 3.8%
Asian American 0.6% 3.1% 0.2% 2.2% 1.0% 3.5% 0.0% 2.9% 1.3% 3.5%
Free/reduced-price lunch 76.6%*** 65.5%*** 69.0%*** 57.8%*** 76.2%*** 62.4%*** 75.7%*** 65.1%*** 85.5%*** 74.1%***
Census block median
income (mean)
$33,831*** $35,793*** $35,153 $36,054 $32,326*** $35,165*** $33,079** $35,501** $34,556*$36,435*
ELL 14.9%*** 7.8%*** 20.8%** 9.7%** 13.2% 7.7% 11.7% 6.3% 13.4% 8.0%
GATE 20.9% 22.9% 21.4% 21.9% 19.4% 21.9% 17.4% 21.6% 24.8% 25.7%
Special education 10.1%** 19.0%** 9.8% 18.4% 10.0% 19.6% 10.9% 19.4% 9.6% 18.6%
Transfer (mean number of times) 0.09 0.09 0.08 0.08 0.07 0.08 0.08 0.11 0.13 0.10
School: A 8.4%*15.2%*8.7%*19.3%*10.8% 15.4% 8.9% 13.1% 5.6% 14.0%
School: B 24.5%*** 16.8%*** 13.4%*22.9%*35.3%*** 14.3%*** 27.3%** 16.9%** 23.2%*14.7%*
School: C 1.1% 4.8% — — — — — — 3.1% 15.7%
School: D 17.9% 18.4% 38.6%*** 15.4%*** 10.5% 19.6% 8.7%*20.1%*12.3% 18.0%
(continued)
1098
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Table 2 (continued)
All Cohorts (n= 8,382) 2008 (n= 1,792) 2009 (n= 2,130) 2010 (n= 2,106) 2011 (n= 2,354)
MAS Non-MAS MAS Non-MAS MAS Non-MAS MAS Non-MAS MAS Non-MAS
n= 1,707 n= 6,675 n= 448 n= 1,344 n= 408 n= 1,722 n= 403 n= 1,703 n= 448 n= 1,906
School: E 10.8% 11.5% — — 10.0% 16.8% 14.9% 15.7% 17.6%*10.3%*
School: F 37.3%*33.3%*37.7% 40.8% 33.3% 33.4% 39.7% 33.9% 38.2%** 27.3%**
9th-grade GPA 2.28*** 2.44*** 2.37 2.46 2.18** 2.36** 2.29** 2.45** 2.25*** 2.48***
10th-grade GPA 2.14*** 2.35*** 2.20*** 2.39*** 2.05*** 2.24*** 2.15*** 2.36*** 2.15*** 2.39***
10th-grade AIMS: math 689** 692** 690 690 686*693*688*693*692 694
10th-grade AIMS: reading 686*** 694*** 686*691*685** 693** 685** 694** 690*695*
10th-grade AIMS: writing 684 685 678*674*687 689 686 687 686 688
Note. MAS = Mexican American Studies; AIMS = Arizona’s Instrument to Measure Standards; ELL = English language learner; GATE = Gifted and
Talented Education; GPA = grade point average. Dashes indicate data were unavailable for these schools during the specific year because MAS
was not offered. Significance values reported for continuous variables utilized two-tailed ttests between MAS and non-MAS participants, while
dichotomous variables were calculated using a two-sample test of proportions.
For all results, ***p,.001. **p,.01. *p,.05.
1099
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
placement courses (e.g., A = 5, B = 4, etc.; see Table 1). Second, we created
covariates of 10th grade AIMS test scores in math, reading, and writing.
Third, we created measures of whether the student utilized GATE services
in high school, was ever classified as a special education student, and was
ever classified as an English language learner.
Measures of socioeconomic characteristics also satisfy the criteria for
omitted variable bias. Prior research shows these characteristics have an
effect on academic achievement (Aikens & Barbarin, 2008; Coley, 2002;
Orr, 2003; Palardy, 2008), and it is reasonable to suggest that they may be
correlated with MAS participation. For example, the MAS program partici-
pants tended to be from lower SES backgrounds than their peers (see
Table 2). Our models include participation in the federal free/reduced-price
lunch program and median household income in the census block where
the student resides as proxies for socioeconomic status.
Our models also include demographic covariates, specifically race and
gender. Race/ethnicity was correlated with MAS participation and may
have an effect on educational achievement. Latina/o students, as expected,
were more likely than any other race/ethnicity to enroll in MAS courses
(see Table 2), but Latina/o students have lower academic achievement, on
average, than White students (Ga
´ndara & Contreras, 2009). The opposite
trend exists for gender. Females were more likely than males to participate
in MAS (see Table 2), and prior research finds that young women have
higher academic achievement, on average, than young men (Ga
´ndara &
Contreras, 2009; Sa
´enz & Ponjuan, 2009). Finally, our models used indicator
variables to identify which school the student attended. Prior research shows
that school characteristics affect academic achievement (V. E. Lee & Bryk,
1989; Teddlie & Stringfield, 1993). Additionally, there was diversity in the
size and scope of MAS programs within individual schools. Therefore, we
controlled for school-level factors by including binary indicators for each
school in TUSD while also using a measure of the number of times a student
transferred. We used pseudonyms (Schools A–F) instead of school names.
One limitation is that our study did not include measures of parents and
peers, both of which can affect educational achievement (Rumberger,
2011). Unfortunately, administrative data do not include these measures.
Missing Data
A small number of students had missing data for 9th and 10th grade GPA
as well as 10th grade AIMS tests. Results from a two-tailed ttest showed that
students with missing GPA data had significantly lower average AIMS test
scores than students without missing data. Similarly, students with missing
AIMS test scores had significantly lower GPAs than students without missing
scores. Without imputing missing values for the GPA and 10th grade AIMS
Cabrera et al.
1100
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
test covariates, students with lower academic achievement would be dispro-
portionately excluded.
We used different imputation strategies for GPA and 10th-grade AIMS
scores. For GPA, we found that 9th-grade and 10th-grade GPAs were signif-
icantly correlated (r= .766), as were 10th- and 11th-grade GPAs (r= .770).
Therefore, we began our imputation strategy by replacing missing GPA
data with the subsequent year’s data. For 9th-grade GPA, there was 14.7%
missing data (n= 1,229), and 10th-grade GPA was missing 10.8% (n=
907). Using this method yielded a substantial decrease in the missing data
(9th-grade GPA, n= 685, 8.2%; 10th-grade GPA, n= 28, 0.3%). For the
remaining missing data we used the expectation-maximization algorithm
within SPSS 21.0ä. The expectation-maximization algorithm represents
a general method for obtaining maximum likelihood estimates (Dempster,
Laird, & Rubin, 1977, cited in Allison, 2002; McLachlan & Krishnan, 1997).
While a multiple imputation method is generally preferred, single imputation
is acceptable when the proportion of missing data is less than 10% (Schafer,
1999; Scheffer, 2002). This approach was the only one we used for missing
AIMS data, and each of the AIMS tests met this criterion (math, n= 626, 7.5%;
reading, n= 619, 7.4%; writing, n= 639, 7.6%). For the other variables, miss-
ing data were either not an issue or represented less than 0.3% of the obser-
vations for an individual variable.
Results
Characteristics of MAS and Non-MAS Students
To examine how the MAS students compared to non-MAS students, we
cross-tabulated and conducted mean or two-sample proportion comparisons
of all of the independent and dependent variables in the models (see Table
2). This revealed some important trends. First, and not surprisingly, Latinas/
os were significantly more likely than non-Latinas/os to enroll in MAS
courses, but they also represented over 55% of the non-MAS students—high-
lighting that schools offering these classes tended to have strong majority-
Latina/o student bodies. In addition, MAS students tended to be from lower
income backgrounds than their non-MAS peers. Across all four cohorts, MAS
students lived in census blocks where the median income was significantly
lower than that of their non-MAS peers. In addition, MAS students were less
likely to be designated as special education students but more likely to be
English language learners. Additionally, MAS students had significantly
lower 9th- and 10th-grade GPAs than their non-MAS peers. The only excep-
tion to this trend was the 2008 cohort 9th-grade GPA where there was a dif-
ference, but it was not statistically significant. A similar trend existed for 10th-
grade AIMS scaled scores. Across all tests and cohorts, non-MAS students
tended to score higher, and the majority of these differences were statistically
Missing the (Student Achievement) Forest
1101
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
significant. The exception was the 2008 cohort where the MAS students’
math scores were equal to the non-MAS students’ and the MAS students
scored significantly higher on the writing test.
The measure of academic performance at the end of high school (AIMS
passing and graduation) showed a different trend in which MAS students
generally outperformed their non-MAS peers. In all five measures, MAS stu-
dents had significantly higher rates of test passing and graduation. This is
counterintuitive because significantly lower GPAs and standardized test
scores early in a high school career generally correspond to lower levels
of high school completion. At the very least, there is no reason to expect
the achievement trends to reverse.
Logistic Regression Results
To more closely examine the role that MAS participation played in these
outcomes, we computed a series of logistic regression analyses for all stu-
dents and ran separate regression analyses disaggregated by cohort. The
logistic regression approach allowed us to isolate the relationship between
MAS participation and an outcome of interest (e.g., graduation) when con-
trolling for the previously described covariates. The resulting average mar-
ginal effects (herein marginal effects) are easier to interpret than odds ratios
and log-odds coefficients, as they documented the changes in likelihood that
a specific outcome would occur as a result of MAS participation when con-
trolling for the other covariates in the model.3
Table 3 shows logistic regression model results of the effect of participat-
ing in at least one MAS class on the probability of success in passing AIMS
tests after initial failure and graduating. There are two different samples in
Table 3: one that only includes schools offering MAS and the other including
all TUSD schools (full regression models are available upon request). Table 3
shows that, for all cohorts combined, the marginal effect of participating in
MAS on graduating was .095 (p,.001), meaning that participating in MAS
increased the probability of graduation by 9.5% for the sample of all cohorts
combined. For the 2008 cohort, participating in MAS increased the probabil-
ity of graduation by 16.1% (p,.001). The marginal effects were smaller but
still positive and significant for subsequent cohorts: 10.6% for the 2009
cohort (p,.001), for the 2010 cohort 6.6% (p,.001), and 7.0% for the
2011 cohort (p,.001).
Sample sizes were smaller for the AIMS test outcome variables because
only students who initially failed that particular AIMS test were included in
the analysis sample. For students who initially failed the AIMS reading test,
participating in MAS increased the probability of subsequently passing the
AIMS reading test by 9.3% for all cohorts (p,.001), by 17.9% for the
2008 cohort (p,.001), and by 6.2% for the 2010 cohort (p,.01) but did
not significantly affect the probability of success for the 2009 and 2011
Cabrera et al.
1102
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
cohorts. For students who initially failed the AIMS math test, participating in
MAS positively affected the probability of subsequently passing the AIMS
math test by 8.7% for all cohorts (p,.001), 15.3% for the 2008 cohort (p
,.001), by 8.6% for the 2009 cohort (p,.05), and by 7.0% for the 2010
cohort (p,.001) but did not significantly affect the probability of success
Table 3
Marginal Effect of Participating in Mexican American Studies (MAS)
All Cohorts 2008–2009 2009–2010 2010–2011 2011–2012
Schools offering MAS classes
Graduation 0.095*** 0.161*** 0.106*** 0.066*** 0.070***
(0.008) (0.026) (0.016) (0.018) (0.020)
n8,342 1,779 2,123 2,096 2,344
AIMS: reading 0.093*** 0.179*** 0.053 0.062*** 0.095
(0.017) (0.023) (0.037) (0.016) (0.052)
n2,801 647 733 684 737
AIMS: writing 0.086*** 0.141*0.080** 0.092*** 0.033
(0.030) (0.058) (0.029) (0.018) (0.042)
n2,950 743 691 782 734
AIMS: math 0.087*** 0.153*** 0.086*0.070*** 0.008
(0.010) (0.021) (0.035) (0.015) (0.031)
n3,263 737 895 792 839
AIMS: all tests 0.068*** 0.104*** 0.043*0.071*** 0.024
(0.011) (0.012) (0.021) (0.011) (0.022)
n4,384 1,031 1,101 1,115 1,137
All Tucson Unified School District schools
Graduation 0.090*** 0.138*** 0.099*** 0.067*** 0.073***
(0.008) (0.017) (0.015) (0.017) (0.015)
n16,867 4,282 4,289 4,238 4,057
AIMS: reading 0.085*** 0.151*** 0.060 0.048** 0.084
(0.018) (0.025) (0.034) (0.017) (0.044)
n4,670 1,261 1,160 1,133 1,112
AIMS: writing 0.067*0.111*0.075** 0.065*** 0.035
(0.029) (0.045) (0.026) (0.015) (0.040)
n5,015 1,513 1,092 1,316 1,067
AIMS: math 0.092*** 0.146*** 0.096** 0.068*** 0.024
(0.012) (0.013) (0.034) (0.015) (0.027)
n5,644 1,510 1,489 1,362 1,268
AIMS: all tests 0.066*** 0.092*** 0.042*0.065*** 0.034
(0.013) (0.009) (0.021) (0.009) (0.024)
n7,569 2,123 1,837 1,915 1,679
Note. Cluster, robust standard errors are in parentheses.
***p,.001. **p,.01. *p,.05.
Missing the (Student Achievement) Forest
1103
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
for the 2011 cohort. Model results for passing the AIMS writing test and pass-
ing all AIMS tests were similar to model results for passing the AIMS math
test. Additionally, the sample that included all TUSD schools yielded very
similar results to the models that included only students attending schools
offering MAS.
As a supplemental analysis, we examined whether the effect of partici-
pating in MAS differed by the number of MAS courses taken. Table 4 tabu-
lates the number of MAS courses taken by graduating cohort, for students
attending schools where MAS was offered to their graduating cohort. For
the 2008 cohort, 43% (188 out of 436) of students who participated in
MAS took only one course, but in the 2009, 2010, and 2011 cohorts, most stu-
dents took two or more. MAS courses were designed as two-semester
sequences (Cammarota & Romero, 2014). Therefore, MAS participants
were more likely to take two, four, six, or eight MAS courses, rather than
one, three, five, or seven. We created aggregate categories for the indepen-
dent variable, specifically, ‘‘three or four MAS classes’’ and ‘‘five or more MAS
classes.’’ We would have preferred to keep the groupings in two-course
increments, but this option was not viable due to the small sample sizes
for students who took more than six MAS courses. Table 5 shows marginal
effects regarding the relationship between taking a specific number of MAS
courses and the five outcomes of interest, relative to the reference group of
students taking zero MAS courses. Consistent with our primary focus, these
analyses were conducted only on students attending schools that offered
MAS.
Table 4
Number of Mexican American Studies (MAS) Courses Taken
in Schools Offering MAS, by Cohort
# MAS Courses All Cohorts 2008–2009 2009–2010 2010–2011 2011–2012
0 6,676 1,343 1,722 1,703 1,908
1 371 188 68 49 66
2 601 108 171 146 176
3 97 7 40 27 23
4 313 50 66 106 91
5 73 32151115
6 103 11 20 32 40
7 2810396
8 81 30181320
9 2 0002
Total 8,345 1,779 2,123 2,096 2,347
Cabrera et al.
1104
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Table 5
Average Marginal Effect of Taking ‘‘n’’ Mexican American Studies (MAS) Classes, Relative to Taking Zero MAS Classes
Graduation All Cohorts (n= 8,342) 2008–2009 (n= 1,779) 2009–2010 (n= 2,123) 2010–2011 (n= 2,096) 2011–2012 (n= 2,344)
1 (0 = ref) 0.051 (0.036) 0.171*** (0.046) –0.000 (0.037) –0.035 (0.029) –0.013 (0.061)
2 (0 = ref) 0.088*** (0.020) 0.139*** (0.020) 0.115*** (0.035) 0.045 (0.047) 0.070** (0.023)
3–4 (0 = ref) 0.115*** (0.015) 0.098*** (0.005) 0.165*** (0.031) 0.119*** (0.017) 0.078** (0.025)
5 or more (0 = ref) 0.159*** (0.016) 0.218*** (0.028) 0.135*** (0.008) 0.132** (0.051) 0.146*** (0.029)
AIMS: Reading All Cohorts (n= 2,801) 2008–2009 (n= 647) 2009–2010 (n= 733) 2010–2011 (n= 684) 2011–2012 (n= 737)
1 (0 = ref) 0.064*(0.029) 0.238*** (0.043) 0.026 (0.100) –0.026 (0.041) –0.116 (0.146)
2 (0 = ref) 0.073*(0.032) 0.146*** (0.043) 0.029 (0.032) 0.050 (0.029) 0.119 (0.073)
3–4 (0 = ref) 0.115*** (0.028) 0.126 (0.066) 0.078 (0.100) 0.115*** (0.029) 0.110*** (0.027)
5 or more (0 = ref) 0.176*** (0.043) 0.173*(0.078) 0.208** (0.076) 0.124*(0.057) 0.216*** (0.057)
AIMS: Writing All Cohorts (n= 2,950) 2008–2009 (n= 743) 2009–2010 (n= 691) 2010–2011 (n= 782) 2011–2012 (n= 734)
1 (0 = ref) 0.069 (0.045) 0.227*** (0.036) 0.007 (0.072) –0.133** (0.048) –0.039 (0.141)
2 (0 = ref) 0.073*(0.036) 0.046 (0.084) 0.076 (0.061) 0.126** (0.045) 0.055 (0.050)
3–4 (0 = ref) 0.085*(0.036) 0.043 (0.186) 0.160*** (0.036) 0.122** (0.039) 0.025 (0.082)
5 or more (0 = ref) 0.170*** (0.015) 0.270*** (0.020) 0.045 (0.115) 0.264 (0.162) 0.076 (0.053)
AIMS: Math All Cohorts (n= 3,263) 2008–2009 (n=737) 2009–2010 (n= 895) 2010–2011 (n= 792) 2011–2012 (n= 839)
1 (0 = ref) 0.060 (0.047) 0.156** (0.058) –0.028 (0.058) –0.046 (0.049) –0.082 (0.105)
2 (0 = ref) 0.086*** (0.013) 0.133*** (0.023) 0.079 (0.069) 0.053 (0.042) 0.044 (0.083)
3–4 (0 = ref) 0.092*** (0.010) 0.150*** (0.031) 0.175*** (0.032) 0.139*** (0.028) –0.066*** (0.020)
5 or more (0 = ref) 0.124*** (0.014) 0.193*** (0.053) 0.116** (0.041) 0.060 (0.038) 0.114** (0.035)
AIMS: All Tests All Cohorts (n= 4,384) 2008–2009 (n= 1,031) 2009–2010 (n= 1,101) 2010–2011 (n= 1,115) 2011–2012 (n= 1,137)
1 (0 = ref) 0.060 (0.037) 0.150** (0.051) –0.078 (0.042) –0.037 (0.084) –0.018 (0.117)
2 (0 = ref) 0.051** (0.017) 0.073 (0.039) 0.055 (0.049) 0.067*(0.027) 0.013 (0.050)
3–4 (0 = ref) 0.073*** (0.006) 0.034 (0.049) 0.090** (0.032) 0.120*** (0.014) 0.004 (0.018)
5 or more (0 = ref) 0.112*** (0.015) 0.124*(0.052) 0.095*(0.039) 0.070*(0.033) 0.105** (0.038)
Note. ref = reference. Cluster, robust standard errors are in parentheses.
***p,.001. **p,.01. *p,.05.
1105
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
For the 2008 cohort, taking one MAS course (compared to the reference
group of zero) increased the probability of graduation by 17.1% (p,.001;
see Table 5). The marginal effects for taking two MAS courses (13.9%, p,
.001) and ‘‘three or four’’ MAS courses (9.8%, p,.001) were smaller in mag-
nitude than the marginal effects of taking one MAS course, while the mar-
ginal effect of taking five or more MAS courses (21.8%, p,.001) was larger
than the marginal effect of taking one MAS course. For subsequent cohorts
the marginal effect of taking one MAS course was insignificant, in contrast to
2008 cohorts, and the marginal effects generally increased in magnitude as
the number of MAS courses taken increased. A similar trend existed for
AIMS passing after initial failure. The 2008 cohort tended to have the largest
marginal effects stemming from taking one MAS course while the marginal
effects on the other three cohorts tended to increase as the number of
MAS courses increased. Finally, Appendix A shows results from models
where one MAS course was the reference group. These results showed
that taking more than one MAS course significantly increased the probability
of graduation for the 2009, 2010, and 2011 cohorts. The results were not as
pronounced for AIMS passing, but the general trend held.
Discussion
The estimated relationship between MAS participation and student educa-
tional attainment was surprisingly strong. Analyses from our initial report had
not included covariates for prior academic achievement (Cabrera et al., 2012).
We expected the estimated relationship to decline once these covariates were
included, but it did not. Instead, the robustness of the coefficient on MAS par-
ticipation was consistent with the descriptive statistics in Table 2. The MAS stu-
dents had significantly lower 9th- and 10th-grade GPAs as well as 10th-grade
AIMS scores than their non-MAS peers. However, they had significantly higher
AIMS passing and graduation rates than their non-MAS peers, which seems
counterintuitive. Decades of findings from education research would lead us
to expect higher 9th- and 10th-grade GPAs and higher 10th-grade standardized
test scores to be positively correlated with higher graduation rates (Alexander,
Entwisle, & Horsey, 1997; Rumberger, 2011). Instead, we found the MAS stu-
dents outperformed their non-MAS peers in terms of AIMS passing and gradu-
ation despite having 9th- and 10th-grade academic performances that were sig-
nificantly lower (see Table 2). These results corroborate findings that ethnic
studies can lead to increased student development (see, e.g., Astin, 1993;
Bowman, 2009, 2010a, 2010b; Sleeter 2011). Furthermore, descriptive statistics
in Table 2 do not support the claims of critics that MAS administrators selected
the most academically capable students and that this accounted for increased
AIMS passing and graduation rates (Stegeman, 2013). Instead, lower performing
students took the MAS courses. Future studies should examine more closely
whether this pedagogical approach effectively supports higher performing
Cabrera et al.
1106
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
students using traditional metrics. In the end, there was a selection bias in the
sampling procedures, but it was in favor of lower performing students.
We utilized multiple modeling and sampling strategies throughout this pro-
grammatic assessment (previous analyses and current study included), and all
of the results point to one conclusion: Taking MAS classes is consistently, signif-
icantly, and positively related to increased student academic achievement, and
this relationship grows stronger the more classes students take. It did not matter
how we operationalized academic success (i.e., graduation or one of three
AIMS tests); if we ran models using all students in TUSD (see Table 3), only
those in schools offering MAS (see Table 3), or a matched sample of students
(Cabrera et al., 2013); or if the models utilized clustered, robust standard errors
(current study), or omitted this component (Cabrera et al., 2013). The results
remained the same, and the only significant shift occurred when we included
controls for pre-MAS academic performance (current study), and as previously
discussed, the relationships actually became stronger.
There was one cohort—2011—in which the association between MAS
participation with outcomes was not as pronounced, and this was consistent
with the previous analyses (Cabrera et al., 2012). These results seem confusing
when analyzing the program in isolation. With a similar number of students
served, and the same pedagogical approach employed, why would one
cohort show a decline in the impact of MAS? Prior research finds that effect
sizes decline when additional sites adopt a program because fidelity of imple-
mentation is lower in new locales (Glennan, Bodilly, Galegher, & Kerr, 2004).
There is some evidence to support this claim as there were two more schools
offering MAS courses in the 2011 cohort than in the 2008 one; however, the
number of students served was the same in both cohorts, 446.
Another explanation involves the increasing political scrutiny of the MAS
program. The Arizona legislature passed HB 2281 in May of 2010, and in
December of that year, Superintendent Tom Horne (2010) found TUSD
out of compliance with the statute. Early in 2011 several community protests
occurred within TUSD over the potential elimination of MAS, including the
dramatic student takeover of the school board on April 26 (Cabrera et al.,
2013). The intense political turmoil of the time may have contributed to
the decline in the impact of MAS on academic achievement because the tur-
moil that students, families, and the school district experienced likely dis-
tracted from the day-to-day rhythm of classroom life.
Despite the weaker results from the 2011 cohort, the overall analyses
strongly support that participation in MAS was positively related to increased
academic achievement, and this generally increased the more classes stu-
dents completed. This result is critically important because TUSD is under
a federal desegregation order, in part because of the educational disparities
between Latina/o and White students. These analyses show that compared
to non-MAS students, MAS participants were more likely to be low-income
Latinas/os with low levels of academic performance prior to taking MAS
Missing the (Student Achievement) Forest
1107
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
courses. Future research needs to assess whether the elimination of the pro-
gram exacerbates existing inequalities. Additionally, these analyses pose
larger questions: To what degree can the MAS program be scaled up
throughout the district? Can it be adapted to different locales that are facing
similar achievement gaps? If so, will local districts be open to incorporating
critical theorists such as Freire into their curricula?
The results also provide some guidance for those adopting a MAS
approach to education. First, while one course was important, the most demon-
strable impacts stemmed from taking multiple courses. Therefore, those creat-
ing a program need to be intentional, coordinating efforts to offer more than
simply one ‘‘diversity’’ course. This requires a large-scale, organized, collective
effort. Some might argue that this is untenable, but here, a historical perspective
is warranted. The MAS program started very small and gradually expanded with
the expansion of both student interest and the availability of teachers trained in
CCI. The current analyses were conducted at the peak of student program par-
ticipation. Thus, the successes documented stem from sustained effort over the
course of a decade (Cammarota & Romero, 2014). Like all K-12 improvement
efforts, programs such as MAS need to be designed, implemented, and brought
to scale with care and diligence.
Additionally, our findings raise questions regarding which elements of
the MAS program enhance student achievement. Is it the process of consci-
entizacxa˜o (Freire, 2000, 2008)? Is it authentic caring (Valenzuela, 1999) and
valuing funds of knowledge (Gonza
´lez et al., 2005)? Or are the creators of
the program correct that the individual components of the program cannot
be separated and must function holistically to be maximize their effective-
ness (Cammarota & Romero, 2014)? We do not have the necessary variables
in the data provided by TUSD to explore these questions, and because the
classes were eliminated on January 10, 2012, we cannot collect the data nec-
essary to answer them.
This research has several implications for education research and policy.
The proportion of Latinas/os in Arizona’s public education system is rising
dramatically (Milem, Bryan, Sesate, & Montan
˜o, 2013), as is true throughout
the country (Ga
´ndara & Contreras, 2009). There are, additionally, persistent
gaps in educational achievement between Latina/o and White middle-class
students (Ga
´ndara & Contreras, 2009; Milem et al., 2013). Current
approaches to educating Latinas/os have not ameliorated these gaps, and
therefore new approaches to education are required to address this persis-
tent issue (Ga
´ndara & Contreras, 2009). MAS represents one option that
meets the state superintendent’s requirement for investing in educational
innovation with an empirically supported record of success (Huppenthal,
n.d.). Currently, there are few approaches to educating Latina/o students
that hold as much promise, but racial politics continue to overshadow
a needed focus on student achievement. The opponents of the program
have yet to offer an alternative to MAS that is as thoroughly assessed and
Cabrera et al.
1108
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
proven. This is a critically important point because removing this program
could potentially harm the most academically vulnerable students (i.e., those
students whom the MAS program tended to serve). Given the results of this
study and the absence of a viable replacement, the needed discussion is
how to replicate and expand the successes of this program beyond Tucson.
Nationwide, there are constant discussions of educational inequality and the
need to turn around ‘‘low-performing schools’’ while ‘‘rewarding excellence’’
(U.S. Department of Education, 2010). However, critical ethnic studies is not
included in the proposed solutions. If results matter, then ethnic studies needs
to be considered part of ‘‘real education’’ and reform on a national level.
Additionally, the results of this research suggest that taking MAS classes
fits within the program’s larger professed goals of engaging in liberatory
education (Cammarota & Romero, 2014). There appears to be a tension
given the current analysis’s focus on passing standardized tests and graduat-
ing high school, as these tend to represent a repressive paradigm that blames
the oppressed for their marginalized social position (Duncan-Andrade &
Morrell, 2008). However, the founders, supporters, and maintainers of the
program intentionally included traditional measures of academic achieve-
ment as part of their liberatory paradigm. While critical educators frequently
deride the oppressive nature and overuse of standardized tests (Cammarota
& Romero, 2014; Delpit, 2012), these tests are still pragmatic realities in the
lives of students. As Duncan-Andrade and Morrell (2008) argue, ‘‘The stand-
ards are the gatekeeper that stands between [students] and their futures’’
(p. 160). This fits within Duncan-Andrade and Morrell’s (2008) reframing
of critical pedagogy whereby teachers frequently believe they must ‘‘choose
between academically rigorous teaching and teaching for social justice. This
is a false binary’’ (p. 180). In their understanding, academic rigor and social
critique are mutually complementary goals. This allows students to continu-
ally hone their critical perspectives as they gain academic tools that aid in
their social critique (Cammarota & Romero, 2014).
Duncan-Andrade and Morrell (2008) further argue that the educational
system is created to maintain racial and economic inequality. Therefore,
‘‘raising individual academic performance among students attending urban
schools is itself a revolutionary act’’ (Duncan-Andrade & Morrell, 2008,
p. 189). Thus, MAS as a tool for improved student achievement, especially
among low-income Latinas/os, serves as a counterhegemonic means of dis-
rupting systemic inequality (Cammarota & Romero, 2014). Within this con-
text, Duncan-Andrade and Morrell argue that standardized tests are indefen-
sible and harmful, yet they still assert the following:
Critical pedagogy is the best approach to test preparation in that the
students are developing the important skills that will allow them to
perform on tests as they also develop the language to critique the
structure and nature of the tests that they must take if they are to
make it successfully through the K-12 system. (p. 157)
Missing the (Student Achievement) Forest
1109
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
This pushes against the foundations of another critical education perspec-
tive: CRT. A central component of CRT is a rejection of objective truth and
meritocracy, as they are viewed as camouflaging oppressive social practices
(Delgado & Stefancic, 2001). In contrast, the current analysis demonstrates
that students taking MAS courses outperform their peers who are not in the
program; thus, these students would be labeled meritorious. Currently, CRT
does not have a method for analyzing marginalized student success; instead,
the focus tends to be on how these students are systematically disadvantaged.
This suggests that CRT in education needs to develop an additional vein of
research examining practices that effectively disrupt systemic oppression.
Conclusion
In conducting this study we sought to heed the call of leaders in our
field who have encouraged educational scholars to conduct empirical stud-
ies that are policy relevant—to do work ‘‘that really matters.’’ In so doing, we
have provided empirical evidence regarding the positive impact of MAS
courses on students’ academic achievement. Our hope is that that policy
makers will use these findings as a source of information in constructing
wise, informed, and actionable policy. There are, of course, many other
inputs to policy besides research. Berkman and Plutzer (2005) include three
sources of input to school finance policies: political institutions, citizen pref-
erence, and organized interests. Research findings are most appropriately
included in the category of organized interests, and a key challenge arises
in doing policy-relevant scholarship. The work may not ‘‘matter’’ to policy
actors who are influenced by ideological commitments, political agendas,
or special interest group membership. Decisions might not appear to be
driven by a desire to provide opportunities that enhance student learning,
yet researchers should not be surprised that the rough and tumble of the pol-
icy process does not elevate their findings to the top tier of influence.
Instead, the challenge becomes trying to push policy actors to care about
empirical analysis or work with ones for whom it does matter.
Thus, these findings may play an important role in improving student
learning and development within TUSD as Special Master Hawley prioritized
empirical analyses over ideological commitments. Our findings establish that
taking MAS courses corresponded to a significant, increased likelihood that
students would pass the AIMS tests and graduate from high school. Based on
the findings of our earlier study, the approved USP included the following
provision: ‘‘By the beginning of the 2013–2014 school year, the District shall
develop and implement culturally relevant courses of instruction designed to
reflect the history, experiences, and culture of African American and
Mexican American communities.’’ In so doing, an agenda for the next round
of publicly engaged scholarship has been set.
Cabrera et al.
1110
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Appendix A
Average Marginal Effect of Taking ‘‘n’’ Mexican American Studies (MAS) Classes, Relative to Taking One MAS Class
Graduation All Cohorts (n= 8,342) 2008 (n= 1,779) 2009 (n= 2,123) 2010 (n= 2,096) 2011 (n= 2,344)
0 (1 = ref) –0.051 (0.036) –0.171*** (0.046) 0.000 (0.037) 0.035 (0.029) 0.013 (0.061)
2 (1 = ref) 0.037 (0.040) –0.032 (0.053) 0.115*(0.045) 0.079*(0.038) 0.084 (0.043)
3–4 (1 = ref) 0.064 (0.043) –0.073 (0.045) 0.165*(0.067) 0.154*** (0.018) 0.091 (0.081)
5 or more (1 = ref) 0.108*(0.045) 0.048 (0.040) 0.135** (0.044) 0.166** (0.057) 0.159** (0.051)
AIMS: Reading All Cohorts (n= 2,801) 2008 (n= 647) 2009 (n= 733) 2010 (n= 684) 2011 (n= 737)
0 (1 = ref) –0.064*(0.029) –0.238*** (0.043) –0.026 (0.100) 0.026 (0.041) 0.116 (0.146)
2 (1 = ref) 0.009 (0.020) –0.092 (0.079) 0.004 (0.103) 0.077 (0.064) 0.235 (0.122)
3–4 (1 = ref) 0.051 (0.043) –0.112 (0.081) 0.052 (0.138) 0.141** (0.049) 0.225 (0.147)
5 or more (1 = ref) 0.113 (0.062) –0.065 (0.070) 0.183 (0.127) 0.150*(0.075) 0.332*(0.157)
AIMS: Writing All cohorts (n= 2,950) 2008 (n= 743) 2009 (n= 691) 2010 (n= 782) 2011 (n= 734)
0 (1 = ref) –0.069 (0.045) –0.227*** (0.036) –0.007 (0.072) 0.133** (0.048) 0.039 (0.141)
2 (1 = ref) 0.004 (0.034) –0.182 (0.101) 0.069 (0.071) 0.259*** (0.052) 0.093 (0.130)
3–4 (1 = ref) 0.016 (0.061) –0.184 (0.182) 0.153 (0.105) 0.255*** (0.028) 0.064 (0.193)
5 or more (1 = ref) 0.101 (0.053) 0.043 (0.046) 0.038 (0.179) 0.397** (0.148) 0.115 (0.144)
AIMS: Math All Cohorts (n= 3,263) 2008 (n= 737) 2009 (n= 895) 2010 (n= 792) 2011 (n= 839)
0 (1 = ref) –0.060 (0.047) –0.156** (0.058) 0.028 (0.058) 0.046 (0.049) 0.082 (0.105)
2 (1 = ref) 0.026 (0.059) –0.023 (0.055) 0.108 (0.102) 0.100 (0.065) 0.126 (0.140)
3–4 (1 = ref) 0.032 (0.048) –0.006 (0.080) 0.204*** (0.055) 0.185** (0.068) 0.016 (0.122)
5 or more (1 = ref) 0.064 (0.054) 0.037 (0.082) 0.144 (0.076) 0.107 (0.076) 0.196 (0.113)
AIMS: All Tests All Cohorts (n= 4,384) 2008 (n= 1,031) 2009 (n= 1,101) 2010 (n= 1,115) 2011 (n= 1,137)
0 (1 = ref) –0.060 (0.037) –0.150** (0.051) 0.078 (0.042) 0.037 (0.084) 0.018 (0.117)
2 (1 = ref) –0.009 (0.045) –0.077 (0.077) 0.133 (0.070) 0.104 (0.093) 0.031 (0.140)
3–4 (1 = ref) 0.013 (0.036) –0.116 (0.072) 0.168** (0.064) 0.157 (0.088) 0.022 (0.119)
5 or more (1 = ref) 0.052 (0.036) –0.025 (0.072) 0.173** (0.065) 0.107 (0.066) 0.122 (0.116)
Note. ref = reference. Cluster, robust standard errors are in parentheses.
***p,.001. **p,.01. *p,.05.
1111
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Notes
A previous iteration of this paper was presented at an invited Division G, Vice
Presidential Session of the annual meeting of the American Educational Research
Association, 2013, San Francisco, California. We would like to thank Dr. Patricia
Ga
´ndara and Andrew R. Blatter for their comments on earlier version of the manuscript.
1By the time Mexican American Studies (MAS) was officially discontinued, former
state superintendent Tom Horne had been elected Arizona attorney general, and a new
superintendent, John Huppenthal, had been elected.
2Propensity score matching (PSM) creates comparison groups that attempt to repli-
cate the counterfactual for each treated observation. Each treated observation is matched
to a nontreated observation that has identical values for all other variables that affect the
outcome and have a systematic relationship with programmatic participation. However,
PSM is sensitive to the assumption of selection on observables (Guo & Fraser, 2010),
assuming that there are no omitted variables that affect the outcome and have a systematic
relationship with programmatic participation. Analyses by Smith and Todd (2005) showed
that matching estimators perform poorly when researchers lack appropriate control vari-
ables. PSM is appropriate when researchers have rich survey data that measure factors
conceivably relevant to program participation but is less appropriate for the present study,
which is based on administrative data. Therefore, we ran logistic regressions models and
acknowledged the potential presence of omitted variable bias as a limitation rather than
run PSM models.
3Average marginal effects (AMEs) are calculated as follows: For each observation, cal-
culate the predicted probability of success when X = 1; calculate the predicted probability
of success when X = 0; calculate the difference between these two probabilities, which is
the marginal effect of MAS for each observation; and calculate the mean value of these
individual marginal effects, which is the AME. AMEs are more desirable than ‘‘discrete
change’’ marginal effects because AMEs are calculated using the actual covariate values
of each observation rather than assigning arbitrary covariate values to each observation
(Mitchell, 2012).
References
Acun
˜a, R. (2012, November 15). The vindication of Mexican American Studies.
Counterpunch. Retrieved from http://www.counterpunch.org/2012/11/15/the-
vindication-of-mexican-american-studies/
Aikens, N. L., & Barbarin, O. (2008). Socioeconomic differences in reading trajecto-
ries: The contribution of family, neighborhood, and school contexts. Journal
of Educational Psychology,100, 235–251.
Alexander, K. L., Entwisle, D. R., & Horsey, C. S. (1997). From first grade forward:
Early foundations of high school dropout. Sociology of Education,70(2), 87–107.
Allison, P. D. (2002). Missing data. Thousand Oaks, CA: Sage.
American Educational Research Association. (2012a, February 5). AERA resolution
regarding the repeal of Arizona HB2281. Washington, DC: Author.
American Educational Research Association. (2012b, February 5). AERA resolution
regarding the teaching of Mexican American Studies. Washington, DC: Author.
Angrist, J. D., & Lavy, V. (1999). Using Maimonides’ rule to estimate the effect of class
size on scholastic achievement. Quarterly Journal of Economics,114(2), 533–
575.
Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist’s
companion. Princeton, NJ: Princeton University Press.
antonio, a. l. (2001). Diversity and the influence of friendship groups in college.
Review of Higher Education,25(1), 63–89.
Cabrera et al.
1112
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Arizona Department of Education. (2011). Guide to test interpretation: AIMS and
Stanford 10. Phoenix, AZ: Author.
Astin, A. W. (1993). Diversity and multiculturalism on the campus: How are students
affected? Change: The Magazine of Higher Learning,25(2), 44–49.
Bean, T. W., Valerio, P. C., Senior, H. M., & White, F. (1999). Engagement in reading
and writing about a multicultural novel. Journal of Educational Research,93(1),
32–37.
Berkman, M. B., & Plutzer, E. (2005). Ten thousand democracies: Politics and public
opinion in America’s school districts. Washington, DC: Georgetown University
Press.
Bowman, N. A. (2009). College diversity courses and cognitive development among
students from privileged and marginalized groups. Journal of Diversity in Higher
Education,2(3), 182.
Bowman, N. A. (2010a). College diversity experiences and cognitive development: A
meta-analysis. Review of Educational Research,80(1), 4–33.
Bowman, N. A. (2010b). Disequilibrium and resolution: The nonlinear effects of
diversity courses on well-being and orientations toward diversity. Review of
Higher Education,33(4), 543–568.
Brozo, W. G., & Valerio, P. C. (1996). A walk through Gracie’s garden: Literacy and
cultural explorations in a Mexican American junior high school. Journal of
Adolescent and Adult Literacy, 40(3), 164–170.
Cabrera, N. L. (2012). Empirical analysis of Tucson Unified School District’s Mexican
American Studies Department, 2010 graduating cohort. Tucson, AZ: Report to
Special Master Dr. Willis D. Hawley on the Tucson Unified School District
Desegregation Case.
Cabrera, N. L., Meza, E. L., Romero, A. J., & Rodriguez, R. (2013). ‘‘If there is no strug-
gle, there is no progress’’: Transformative youth resistance and the school of eth-
nic studies. Urban Review,45(1), 7–22.
Cabrera, N. L., Milem, J. F., & Marx, R. W. (2012). An empirical analysis of the effects
of Mexican American Studies participation on student achievement within
Tucson Unified School District. Tucson, AZ: Report to Special Master Dr. Willis
D. Hawley on the Tucson Unified School District Desegregation Case.
Cammarota, J. (2007). A social justice approach to achievement: Guiding Latina/o stu-
dents toward educational attainment with a challenging, socially relevant curric-
ulum. Equity and Excellence in Education,40(1), 87–96.
Cammarota, J., & Romero, A. (2006a). A Critically Compassionate Intellectualism for
Latina/o students: Raising voices above the silencing in our schools.
Multicultural Education,14(2), 16–23.
Cammarota, J., & Romero, A. (2006b). A critically compassionate pedagogy for Latino
youth. Latino Studies,4(3), 305–312.
Cammarota, J., & Romero, A. (2009). The Social Justice Education Project: A Critically
Compassionate Intellectualism for Chicana/o students. In W. Ayers, T. Quinn, &
D. Stovall (Eds.), Handbook for social justice education (pp. 465-476). New
York, NY: Routledge.
Cammarota, J., & Romero, A. (2014). Raza studies: The public option for educational
revolution. Tucson: University of Arizona Press.
Cappellucci, D. F., Williams, C., Hernandez, J. J., Nelson, L. P., Casteel, T., Gilzean, G.,
& Gershon, F. (2011). Curriculum audit of the Mexican American Studies
Department, Tucson Unified School District. Miami Lakes, FL: Cambium
Learning.
Missing the (Student Achievement) Forest
1113
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Carter, D. (2008). Achievement as resistance: Development of a critical race achieve-
ment ideology among Black achievers. Harvard Educational Review,78(3),
466–497.
Chang, M. J. (2002). The impact of an undergraduate diversity course requirement on
students’ racial views and attitudes. Journal of General Education,51(1), 21–42.
Coley, R. J. (2002). An uneven start: Indicators of inequality in school readiness.
Princeton, NJ: Educational Testing Service.
Copenhaver, J. (2001). Listening to their voices connect literary and cultural under-
standings: Responses to small group read-alouds of Malcolm X: A Fire. New
Advocate,14(4), 343–359.
Delgado, R., & Stefancic, J. (2001). Critical race theory: An introduction. New York,
NY: NYU Press.
Delpit, L. (2012). ‘‘Multiplication is for white people’’: Raising expectations for other
people’s children. New York, NY: New Press.
Denson, N., & Chang, M. J. (2009). Racial diversity matters: The impact of diversity-
related student engagement and institutional context. American Educational
Research Journal,46(2), 322–353.
Department of Accountability and Research. (2011a, January 6). AIMS achievement
comparison for students taking one or more ethnic studies classes: Initial passing
rate versus cumulative passing rate by AIMS subject and cohort year. Tucson,
AZ: Tucson Unified School District.
Department of Accountability and Research. (2011b, January 6). Selected statis-
tics—2010 (four year) graduation cohort. Tucson, AZ: Tucson Unified School
District.
DesJardins, S. L. (2005). Investigating the efficacy of using selection modeling in
research of the Gates Millennium Scholars program. Washington, DC: Gates
Millennium Scholars Research Advisory Council.
Duncan-Andrade, J. M. R., & Morrell, E. (2008). The art of critical pedagogy:
Possibilities from moving theory to practice in urban schools. New York, NY:
Peter Lang.
Engberg, M. E. (2004). Improving intergroup relations in higher education: A critical
examination of the influence of educational interventions on racial bias. Review
of Educational Research,74(4), 473–524.
Franciosi, R. (2009). The effect of Tucson Unified ethnic (‘‘Raza’’) studies on student
achievement. Phoenix: Arizona Department of Education.
Freire, P. (2000). Pedagogy of the oppressed (30th anniversary ed.). New York, NY:
Herder and Herder.
Freire, P. (2008). Education for a critical consciousness. New York, NY: Continuum.
Freire, P., & Macedo, D. (1987). Literacy: Reading the word and the world. Westport,
CT: Bergin & Garvey.
Ga
´ndara, P., & Contreras, F. (2009). The Latino education crisis: The consequences of
failed social policies. Cambridge, MA: Harvard University Press.
Glennan, T. K., Jr., Bodilly, S. J., Galegher, J. R., & Kerr, K. A. (2004). Expanding the
research of educational reforms: Perspectives from leaders in the scale-up of edu-
cational interventions. Santa Monica, CA: RAND.
Gonza
´lez, N., Moll, L., & Amanti, C. (2005). Funds of knowledge: Theorizing practices
in households, communities, and classrooms. Hillsdale, NJ: Lawrence Erlbaum.
Guo, S., & Fraser, M. W. (2010). Propensity score analysis: Statistical methods and
applications. Thousand Oaks, CA: Sage.
Gurin, P., Dey, E. L., Hurtado, S., & Gurin, G. (2002). Diversity and higher education:
Theory and impact on educational outcomes. Harvard Educational Review,
72(3), 330–367.
Cabrera et al.
1114
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Herreras, M. (2012, November 13). Want more evidence that Mexican-American stud-
ies works? Tucson Weekly. Retrieved from http://www.tucsonweekly.com/
TheRange/archives/2012/11/12/want-more-evidence-that-mexican-american-
studies-works
Horne, T. (2010). Findings by the state superintendent of public instruction of viola-
tion by Tucson Unified School District pursuant to A.R.S.§15-112(B). Phoenix:
State of Arizona Department of Education.
Hoxby, C. M. (2000). The effects of class size on student achievement: New evidence
from population variation. Quarterly Journal of Economics,115(4), 1239–1285.
Hu-DeHart, E. (1993). The history, development, and future of ethnic studies. Phi
Delta Kappan,75(1), 50–54.
Huicochea, A. (2011a, March 15). Ethnic studies claims in question. Arizona Daily
Star. Retrieved from http://azstarnet.com/news/local/education/ethnic-studies-
claim-in-question/article_c316a3d0-7ff5-5f87-874e-6b7cf3b515f3.html
Huicochea, A. (2011b, June 23). TUSD to appeal Huppenthal’s decision on ethnic
studies. Arizona Daily Star. Retrieved from http://azstarnet.com/article_
ebe8a50c-9de0-11e0-8423-001cc4c002e0.html
Huppenthal, J. (n.d.). A message from the superintendent. Retrieved from http://
www.azed.gov/superintendent/
Huppenthal, J. (2011). Superintendent of Public Instruction John Huppenthal state-
ment of finding regarding Tucson Unified School District’s violations of
A.R.S.§15-112. Phoenix: State of Arizona Department of Education.
Hurtado, S., Dey, E. L., Gurin, P. Y., & Gurin, G. (2003). College environments, diver-
sity, and student learning. In J. C. Smart (Ed.), Higher education: Handbook of
theory and research (pp. 145–189). Dordrecht, the Netherlands: Springer.
Johnson, S. M., & Lollar, X. L. (2002). Diversity policy in higher education: The impact
of college students’ exposure to diversity on cultural awareness and political par-
ticipation. Journal of Education Policy,17(3), 305–320.
Khandker, S. R., Koolwal, G. B., & Samad, H. A. (2010). Handbook on impact eval-
uation: Quantitative methods and practices. Washington, DC: World Bank.
Krater, J., & Zeni, J. (1995). Seeing students, seeing culture, seeing ourselves. Voices
From the Middle,3(3), 32–38.
Krater, J., Zeni, J., & Cason, N. D. (1994). Mirror images: Teaching writing in black
and white. Portsmouth, NH: Heinemann.
Laird, T. F. N. (2005). College students’ experiences with diversity and their effects on
academic self-confidence, social agency, and disposition toward critical think-
ing. Research in Higher Education,46(4), 365–387.
Laird, T. F. N., Engberg, M. E., & Hurtado, S. (2005). Modeling accentuation effects:
Enrolling in a diversity course and the importance of social action engagement.
Journal of Higher Education,76(4), 448–476.
Lee, C. D. (1995). A culturally based cognitive apprenticeship: Teaching African
American high school students skills in literary interpretation. Reading
Research Quarterly,30(4), 608–630.
Lee, C. D. (2001). Is October Brown Chinese? A cultural modeling activity system for
underachieving students. American Educational Research Journal,38(1), 97–
142.
Lee, C. D. (2006). ‘‘Every good-bye ain’t gone’’: Analyzing the cultural underpinnings
of classroom talk. International Journal of Qualitative Studies in Education,
19(3), 305–327.
Lee, C. D. (2007). Culture, literacy and learning. New York, NY: Teachers College
Press.
Missing the (Student Achievement) Forest
1115
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Lee, V. E., & Bryk, A. S. (1989). A multilevel model of the social distribution of high
school achievement. Sociology of Education,62(3), 172–192.
Lewis, K. M., Sullivan, C. M., & Bybee, D. (2006). An experimental evaluation of
a school-based emancipatory intervention to promote African American well-
being and youth leadership. Journal of Black Psychology,32(1), 3–28.
Lipka, J. (1991). Toward a culturally based pedagogy: A case study of one Yup’ik
Eskimo teacher. Anthropology and Education Quarterly,22(3), 203–223.
Lipka, J., Hogan, M. P., Webster, J. P., Yanez, E., Adams, B., Clark, S., & Lacy, D.
(2005). Math in a cultural context: Two case studies of a successful culturally-
based math project. Anthropology and Education Quarterly,36(4), 367–385.
Lipka, J., Sharp, N., Brenner, B., Yanez, E., & Sharp, F. (2005). The relevance of cul-
turally based curriculum and instruction: The case of Nancy Sharp. Journal of
American Indian Education,44(3), 31–54.
Lomawaima, K. T., & McCarty, T. L. (2006). ‘‘To remain an Indian’’: Lessons in
democracy from a century of Native American education. New York, NY:
Teachers College Press.
MacEachern, D. (2012, December 15). Insidious Latino studies being revived at
Tucson schools. Arizona Republic. Retrieved from http://www.azcentral.com/
arizonarepublic/viewpoints/articles/20121215insidious-latino-studies-being-
revived-tucson-schools.html
MacEachern, D. (2013, February 9). Bit of leftist engineering in Tucson expensive.
Arizona Republic. Retrieved from http://www.azcentral.com/opinions/articles/
20130209bit-of-leftist-engineering-in-tucson-expensive.html
Martinez, M. (2011, January 5). Arizona education chief moves to ban ethnic studies
in Tucson schools. CNN. Retrieved from http://edition.cnn.com/2011/US/01/04/
arizona.ethnic.studies.ban/
Matthews, C. E., & Smith, W. S. (1994). Native American related materials in elemen-
tary science instruction. Journal of Research in Science Teaching,31(4), 363–
380.
Maxwell, L. A. (2012, November 12). Ethnic studies could return to Tucson in deseg-
regation plan. Education Week. Retrieved from http://blogs.edweek.org/
edweek/learning-the-language/2012/11/ethnic_studies_could_return_to.html
McCarty, T. L. (1993). Language, literacy, and the image of the child in American
Indian classrooms. Language Arts,70(3), 182–192.
McLachlan, G. J., & Krishnan, T. (1997). The EM algorithm and extensions. New York,
NY: Wiley.
Milem, J. F. (1994). College, students, and racial understanding. Thought and Action,
9(2), 51–92.
Milem, J. F., Bryan, W. P, Sesate, D. B., & Montan
˜o, S. (2013). Arizona minority stu-
dent progress report, 2013. Phoenix: Arizona Minority Education Policy Analysis
Center.
Milem, J. F., & Umbach, P. D. (2003). The influence of precollege factors on students’
predispositions regarding diversity activities in college. Journal of College
Student Development,44(5), 611–624.
Milem, J. F., Umbach, P. D., & Liang, C. T. (2004). Exploring the perpetuation hypoth-
esis: The role of colleges and universities in desegregating society. Journal of
College Student Development,45(6), 688–700.
Mitchell, M. N. (2012). Interpreting and visualizing regression models using Stata.
College Station, TX: Stata Press.
O’Connor, C. (1997). Dispositions toward (collective) struggle and educational resil-
ience in the inner city: A case analysis of six African American high school stu-
dents. American Educational Research Journal,34(4), 593–629.
Cabrera et al.
1116
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
Orr, A. J. (2003). Black-White differences in achievement: The importance of wealth.
Sociology of Education,76, 281–304.
Palardy, G. J. (2008). Differential school effects among low, middle, and high social
class composition schools: A multiple group, multilevel latent growth curve anal-
ysis. School Effectiveness and School Improvement,19, 21–49.
Pascarella, E. T., Edison, M., Nora, A., Hagedorn, L. S., & Terenzini, P. T. (1996).
Influences on students’ openness to diversity and challenge in the first year of
college. Journal of Higher Education,67, 174–195.
Prohibited courses and classes, enforcement, Arizona Revised Statute § 15-112.
(2010). Retrieved from http://www.azleg.gov/FormatDocument.asp?inDoc=/
ars/15/00112.htm&Title=15&DocType=ARS
Rickford, A. (2001). The effect of cultural congruence and higher order questioning
on the reading enjoyment and comprehension of ethnic minority students.
Journal of Education for Students Placed at Risk,6(4), 357–387.
Rojas, F. (2006). Social movement tactics, organizational change and the spread of
African-American studies. Social Forces,84(4), 2147–2166.
Romero, A., Arce, S., & Cammarota, J. (2009). A barrio pedagogy: Identity, intellectu-
alism, activism, and academic achievement through the evolution of critically
compassionate intellectualism. Race Ethnicity and Education,12(2), 217–233.
Romero, A. F. (2008). Towards a critically compassionate intellectualism model of
transformative education: Love, hope, identity, and organic intellectualism
through the convergence of critical race theory, critical pedagogy, and authentic
caring. Unpublished doctoral dissertation, University of Arizona.
Rumberger, R. W. (2011). Dropping out: Why students drop out of high school and
what can be done about it. Cambridge, MA: Harvard University Press.
Sa
´enz, V. B., & Ponjuan, L. (2009). The vanishing Latino male in higher education.
Journal of Hispanic Higher Education,8(1), 54–89.
Save Ethnic Studies. (2011). Analysis and evaluation of Mexican American studies.
Tucson, AZ: Author. Retrieved from http://saveethnicstudies.org/
Schafer, J. L. (1999). Multiple imputations: A primer. Statistical Methods in Medical
Research,8, 3–15.
Scheffer, J. (2002). Dealing with missing data. Research Letters in the Information and
Mathematical Sciences,3, 153–160.
Sleeter, C. E. (2011). The academic and social value of ethnic studies: A research
review. Washington, DC: National Education Association.
Smith, J. A., & Todd, P. E. (2005). Does matching overcome LaLonde’s critique of non-
experimental estimators? Journal of Econometrics,125(1/2), 305–353.
Stegeman, M. (2013, January). January constituent letter. Retrieved from
stegeman4tusd.com
Stock, J. H., & Watson, M. W. (2011). Introduction to econometrics (3rd ed.). Boston,
MA: Addison-Wesley.
Takaki, R. (1993). A different mirror: A history of multicultural America. Boston, MA:
Back Bay Books.
Teddlie, C., & Stringfield, S. (1993). Schools make a difference: Lessons learned from
a 10-year study of school effects. New York, NY: Teachers College Press.
Tsui, L. (1999). Courses and instruction affecting critical thinking. Research in Higher
Education,40(2), 185–200.
Tyson, C. A. (2002). ‘‘Get up off that thing’’: African American middle school students
respond to literature to develop a framework for understanding social action.
Theory and Research in Social Education,30(1), 42–65.
Missing the (Student Achievement) Forest
1117
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
U.S. Department of Education. (2010). ESEA reauthorization: A blueprint for reform.
Washington, DC: Author.
Valenzuela, A. (1999). Subtractive schooling: U.S.-Mexican youth and the politics of
caring. New York: State University of New York Press.
Manuscript received October 28, 2013
Final revision received September 3, 2014
Accepted September 8, 2014
Cabrera et al.
1118
at UNIV ARIZONA LIBRARY on December 2, 2014http://aerj.aera.netDownloaded from
