Content uploaded by Jeffery H. Marshall
Author content
All content in this area was uploaded by Jeffery H. Marshall on Feb 22, 2018
Content may be subject to copyright.
This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
Author's personal copy
Economics of Education Review 28 (2009) 207–216
Contents lists available at ScienceDirect
Economics of Education Review
j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / e c o n e d u r e v
School quality and learning gains in rural Guatemala
Jeffery H. Marshall ∗
Sapere Development Solutions, Avenida Sete de Setembro, no. 2774, Salvador da Bahia, Bahia, Brazil
a r t i c l e i n f o
Article history:
Received 5 June 2007
Accepted 14 October 2007
JEL classification:
20
I28
J15
Keywords:
Human capital
Economic development
Efficiency
Resource allocation
a b s t r a c t
I use unusually detailed data on schools, teachers and classroomsto explain student achieve-
ment growth in rural Guatemala. Several variables that have received little attention in
previous studies – including the number of school days, teacher content knowledge and
pedagogical methods – are robust predictors of achievement. A series of decompositions by
student ethnicity and type of school shed some additional light on important questions in
the Guatemalan context, and beyond. The large indigenous test score gap is not explained
by differences in an extensive list of observable features of schools. The large effect for com-
munity characteristics suggests peer group effects or more general institutional differences
related to services or labor markets. PRONADE community schools are associated with mod-
erate gains vis-à-vis public schools in areas related to utilization of capacity, such as days
worked. But these gains are largely offset by lower teacher capacity, which highlights the
challenge of improving school quality in poor, rural areas.
© 2008 Elsevier Ltd. All rights reserved.
1. Introduction
Standardized tests applied in developing countries
show that many students fail to reach even minimum pro-
ficiency levels in reading, writing and mathematics. The
size of this basic skills gap is disturbing, and has potentially
serious implications for equity and overall development
(Hanushek & Woessmann, 2007). And while there is gen-
eral agreement that many schools are failing to provide an
adequate learning environment, articulating an effective
policy response to this problem remains a challenge. On the
one hand, on-going efforts to expand access into secondary
education in developing countries deflect some attention
away from school quality issues. But even when policy-
makers are focused on improving skills in basic education
the empirical evidence they have available to guide their
actions is restricted. This unfortunate reality, despite more
than 30 years of production function analyses of student
∗RAND Corporation, 1776 Main Street, P.O. Box 2138, Santa Monica, CA
90407-2138, USA. Tel.: +1 310 393 0411.
E-mail address: jmarshal@rand.org.
achievement from all over the developing world, is in part
attributable to the limited utility of many of the inde-
pendent variables used in these studies (Fuller & Clarke,
1994).
Concerns about production function work go beyond
data limitations, as best demonstrated by the increasing use
of randomized trials. Nevertheless, the framework remains
a popular one for testing ideas in education, and a group of
“exceptional” studies combining original data and methods
provides a useful precedent to build on (Glewwe, 2002).
This paper continues in this vein using unusually detailed,
longitudinal data on student achievement from rural
Guatemala. In the first part I estimate models of student
achievement using variables that are rarely available to
researchers. These results are then contextualized through
two sets of achievement decompositions. The first extends
previous studies in Guatemala (McEwan & Trowbridge,
2007) and beyond (Psacharopoulos & Patrinos, 1994) by
testing for specific mechanisms that explain a persistent
indigenous student test score gap. The second analyzes
different production dynamics between traditional public
schools and their community school (PRONADE) counter-
parts.
0272-7757/$ – see front matter © 2008 Elsevier Ltd. All rights reserved.
doi:10.1016/j.econedurev.2007.10.009
Author's personal copy
208 J.H. Marshall / Economics of Education Review 28 (2009) 207–216
The results push forward the production function liter-
ature on student achievement and school policy in several
areas. First, the significant effects of total school days
and teacher knowledge confirm the importance of two
frequently cited – but rarely analyzed – influences on
achievement. However, these variables explain little of the
achievement gap between indigenous and non-indigenous
students in rural Guatemala, which in turn points to other
(unmeasured) aspects of schools or communities. Finally,
there is evidence that PRONADE community schools are
realizing efficiency gains vis-à-vis the public sector in
things like teacher attendance. But these gains are largely
offset by lower levels of capacity among the teachers
recruited to work in these communities.
The paper proceeds as follows. Section 2includes a brief
review of production function work in developing coun-
tries, focusing on some of the harder to measure elements
of school quality. Section 3introduces the Guatemalan con-
text, the data and methodology. Section 4details the results
from the various statistical analyses, and Section 5con-
cludes.
2. A brief review of some relevant evidence
This review focuses on elements of schooling that have
received little attention in previous analyses of student
achievement in developing countries. For example, surpris-
ingly little is known about how variation in days offered or
teacher attendance affects student achievement in devel-
oping countries. Measuring the length of the school year
is complicated by unreliable official records, and in rural
areas especially there is little supervision. The evidence that
does exist–coming largely from surprise visits–is troubling.
Kremer, Chaudhury, Rogers, Muralidharan, and Hammer
(2005) and Alcazar et al. (2005) find teacher absence rates
of between 15 and 40% (daily) based on visits to Indian
and Peruvian classrooms, respectively. Bedi and Marshall
(2002) show that “non-official” school closings in Honduras
add up to 19 days on average per year—twice as many days
the average student is absent.
For the teacher’s work the performance of the “usual
suspects” (teacher experience, education, etc.) has been
underwhelming. One glaring omission in the literature
is teacher knowledge, which is often reduced to proxies
based on education levels or classes taken. Only a handful
of studies use actual measures of teacher content knowl-
edge based on an exit exam (Mullens, Murnane, & Willett,
1996), their overall score on a battery of performance tests
(Santiba˜
nez, 2006), their primary level content knowledge
(Harbison & Hanushek, 1992) or their “pedagogical content
knowledge” (PCK) using real-life teaching situations (Hill,
Rowan, & Ball, 2005;Marshall & Sorto, 2007). In each case
the teacher’s knowledge is a significant predictor of stu-
dent achievement. For teaching methodology a handful of
studies have used limited classroom observations, detailed
teacher questionnaires or student interview responses to
gather clues about effective strategies (Fuller et al., 1999;
Glewwe, Grosh, Jacoby, & Lockheed, 1995).
There are clear limits in conceptualizing school quality
solely in terms of days worked or teacher qualifications.
One policy initiative in the developing world that provides
a natural arena for analyzing more dynamic influences
on quality is the community school. Community schools
empower principal agents through the use of parent coun-
cils that are responsible for making decisions locally, such
as the hiring and firing of teachers. In Central Amer-
ica the EDUCO and PRONADE systems in El Salvador
and Guatemala, respectively, have been associated with
marginally higher student achievement levels (Jimenez &
Sawada, 1999). One mechanism appears to be a better use
of existing capacity, as community school teachers are less
frequently absent (DiGropello & Marshall, 2005). But there
is also evidence that efficiency gains in some areas are being
offset by capacity limitations in others. A recent review of
community schooling in Central America finds no evidence
that teachers in decentralized programs are more effec-
tive than their public school counterparts. There is also a
tendency for community schools to have fewer students in
the classroom, which could reflect parental concerns about
the teacher’s ability to work with many students at once
(DiGropello, Marshall, & Rápalo et al., 2004). In other words,
there appear to be limits to the community school impact
when parents are unable – or unwilling – to attract high
quality teachers, or unable to instigate in-service training
activities to upgrade capacity.
3. Analytical framework: explaining academic
achievement in rural Guatemala
3.1. Guatemala context, data collection and variable
measurement
Guatemala is one of the poorest countries in Latin Amer-
ica. 34% of the population lives on less than two dollars per
day, and 70% of rural Guatemalans live below the poverty
line (World Bank, 2004). The inhabitants are a mixture of
indigenous peoples, many of whom speak Mayan dialects,
and Spanish-speaking non-indigenous (“ladinos”). There
have been efforts in recent years to improve educational
opportunities in rural areas. Access has been expanded
through the PRONADE community schoolprogram. Teacher
capacity is being improved through an in-service training
program. And the country’s linguistic diversity is receiv-
ing more direct recognition through the DIGEBI (Dirección
General de Educación Bilingüe Intercultural) program.
The Human Capital in Rural Guatemala (HCRG) data
were collected by the author throughout the 2002 school
year in 58 rural schools. The selected students were part
of a cohort that was originally tested in grade three in
2001 by the PRONERE assessment project (De Baessa,
2002). Because of resource limitations it was not pos-
sible to replicate the nationally representative PRONERE
sampling framework in all 21 states (“departamentos”), so
instead all of the schools from only three states were re-
visited.1Comparisons of 2001 data between the HCRG and
PRONERE samples show similar averages for achievement,
1The states were chosen based primarily on their ethnic makeup, and
include one state almost entirely inhabited by ladinos (Escuintla), another
made up mainly of indigenous residents (Alta Verapaz), and a third that
includes many mixed communities (Chimaltenango).
Author's personal copy
J.H. Marshall / Economics of Education Review 28 (2009) 207–216 209
poverty and ethnic makeup. Nevertheless, the HCRG sam-
ple is only loosely representative beyond these three parts
of the country.
Of the original 2001 cohort a little over 80% of the
students were in grade four in 2002, 9% were repeat-
ing grade three, and 9% had deserted by the end of
2002. Enrolled students were given the same test (using
a different form) at the end of the 2002 school year.
The tests included 40 multiple choice questions designed
by curriculum experts in Guatemala specifically for a
rural population. Student interviews and questionnaires
for teachers and directors provide the rest of the vari-
ables. In the analysis that follows the only information
from 2001 is the incoming student test score; all other
variables are based on the 2002 questionnaires and inter-
views.
Table 1 summarizes the variables. The information on
the number of days of instruction is especially detailed.
Based on information collected from school directors, most
schools began classes by the middle of January 2002, as dic-
tated by the official calendar. But a significant percentage
began a week or more later. There is also information on
how many days classes were held according to the teacher
grade book for grade four.2Classes were (apparently) not
given about 20% of the time during the attendance-taking
period, with the biggest category being for unknown rea-
sons where the teacher did not take attendance and did
not write in a reason for why. The total days of class time
from the first day of class until the application of the stan-
dardized exams is roughly 100 days, which is considerably
lower than the official number (roughly 140) for this same
period. Students were in attendance about 93% of the avail-
able days.
A quantitative classroom observation instrument was
applied in one Spanish and mathematics class in fourth
grade. This uses time segments to divide the total time
of each class into various categories, such as student
seatwork, student–teacher interaction, etc. There are also
observations about whether or not the teacher checked
everyone’s work during class, and the physical condi-
tion of the classroom. The teacher’s content knowledge
in Spanish is based on their responses to 12 items drawn
from the student test. Mathematics content knowledge
is based on lower order items from the student test
and 16 questions drawn from the middle school curricu-
lum. Finally, the teacher’s pedagogical content knowledge
(PCK) in mathematics is measured using three open-ended
questions.3
2When teachers did not take attendance it was assumed no class was
given. Follow-up investigation during the fieldwork and discussions with
fieldwork personnel supports this strategy, although it is impossible to
verify in every case that no class was given on these days.
3For example, teachers were asked to diagnose the source of the error
when students give the incorrect answer of 2806 to the multiplication
problem of “352 ×8”. Teacher responses such as “the student does not
know how to count” receive low marks, while specific diagnoses of the
problem (i.e. “the student is not regroupingor carrying a ten”) score higher.
The PCK items were created by Alejandra Sorto, and together we analyze
their impact on achievment in this same sample in specific content areas
(Marshall & Sorto, 2007).
3.2. Methodology
3.2.1. Student achievement production functions
In theory student achievement in rural Guatemala is
determined by a production function:
A=f(X, S, C;˛) (1)
where achievement Ais a product of home background (X),
school resources (S), community characteristics (C) and an
efficiency parameter measuring capacity utilization in the
school (˛). This general function does not specify levels of
knowledge or knowledge growth. In policy circles a higher
premium is usually placed on understanding the dynamics
of growth (or gains), although in circumstances where the
independent variables do not change much over time the
analysis of levels will give very similar results (Glewwe,
2002). Function 1 can be expressed in reduced form as a
linear estimating equation:
Aijt =ˇ#
XXi+ˇ#
SSj+ˇ#
CCi+εi(2)
where achievement Afor student iin school jin time tis
a function of vectors of variables corresponding to home
background, schools and communities. By adding previous
achievement (Ai(t−1)) Eq. (2) is extended to measure learn-
ing gains. In this paper I use both the level and gain scores
for the production function work (see Table 2).
Interpretation of the point estimates in Eq. (2) can
be affected by the presence of selection bias. One of the
unique features of the data used in this study is informa-
tion on three potential forms of selection bias. I begin by
including a Heckman-style parameter (") based on a pro-
bit estimation of dropping out between test applications
in 2001 and 2002. This requires excluding the student’s
attendance during the 2002 school year and the grade
control from the first stage equation. Three measures of
the mother’s work participation are used as instruments.
The identification strategy is supported empirically by the
significance of the instrument set in the selection equa-
tion and the lack of power of these same variables in
the achievement production functions, with and with-
out the predicted hazard of non-selection (see bottom of
Table 2). There are also different rates of attrition prior
to where this cohort began in grade three in 2001, which
is controlled using the average grades completed for a
randomly drawn group of grade one students taken from
enrollment rolls in each school in 1999. Finally, the data
include the percentage of students in attendance on the
day the exams were applied in 2002, which makes it pos-
sible to address “day-of-test selection” (Glewwe et al.,
1995).
3.2.2. Achievement decompositions
Another potential problem is omitted variable bias,
especially for the school and teacher characteristics. For
example, the capacity utilization parameter (˛) from
the theoretical production function (1) does not appear
in the actual estimation (2). However, since the sam-
ple includes schools from the PRONADE community
school project it is possible to test for different pro-
duction dynamics in one specific case. The community
Author's personal copy
210 J.H. Marshall / Economics of Education Review 28 (2009) 207–216
Table 1
Descriptive statistics: dependent and independent variables
Characteristic Definition Whole sample PRONADE Ethnicity
Ladino Indigenous
Achievement dependent variables
Spanish score 2002 Percent correct on
same 40 item multiple
choice test applied in
2001, with different
form
59.3 (19.2) 46.6** (20.6) 73.7 (14.0) 51.1** (15.7)
Math score 2002 Percent correct on
same 40 item multiple
choice test applied in
2001, with different
form
57.5 (15.8) 48.8** (15.7) 63.9 (14.9) 53.8** (15.2)
Spanish gain score Percentage difference
between 2001 and
2002 result
8.8 (11.3) 6.8* (10.6) 10.8 (11.4) 7.6** (11.1)
Math gain score Percentage difference
between 2001 and
2002 result
13.1 (11.7) 11.5 (11.6) 14.8 (11.5) 12.1** (11.7)
Student-family background
Spanish test 2001 Percent correct on
same 40 item multiple
choice test
50.5 (18.2) 39.8** (20.7) 62.9 (14.9) 43.5** (16.0)
Math test 2001 Percent correct on
same 40 item multiple
choice test
44.4 (15.3) 37.4** (15.2) 49.2 (15.5) 41.7** (14.6)
Student age Student’s age in years 11.7 (1.6) 11.5 (1.6) 11.1 (1.5) 12.0** (1.5)
Female Student is female (pct) 49.0 53.0 50.0 48.0
Indigenous Student reports
speaking a Mayan
language (pct)
64.0 84.5** – –
Parental education Average years of
parental education
2.3 (2.1) 1.5** (1.8) 3.0 (2.3) 1.9** (1.9)
SES Principal component
factor of household
possessions and
services (non-rotated)
0.10 (1.1) −0.61** (1.1) 1.2 (1.1) −0.4** (1.2)
Grade 4 Student is in grade 4
(pct)
86.0 92.0** 89.0 85.0**
Student attendance Percentage of total days
offered in school
during 2002 that
student was in
attendance
93.5 (6.9) 93.6 (5.0) 94.4 (6.1) 93.0** (7.3)
Selection correction
Cohort control Average grades
completed for cohort of
students chosen from
grade 1 in 1999
2.6 (0.5) 2.9** (2.5) 2.6 (0.4) 2.6 (0.6)
Day of test control Percentage of grade 4
class in attendance on
day of exam
application
90.1 85.1** 89.3 91.1**
Non-selection hazard (") Hazard that student is
enrolled in school in
2002 (mills ratio
transformation)
0.77 0.75** 0.79 0.76**
School characteristics
Class size Number of students in
classroom
32.9 (10.0) 35.0** (6.1) 33.1 (11.5) 32.4 (9.3)
Student-reported fighting (school average) Students report
fighting with other
students in school
(1 = None, 2 = Some,
3 = Many)
1.33 (0.15) 1.42** (0.17) 1.35 (0.18) 1.33 (0.14)
Author's personal copy
J.H. Marshall / Economics of Education Review 28 (2009) 207–216 211
Table 1 (Continued)
Characteristic Definition Whole sample PRONADE Ethnicity
Ladino Indigenous
Teacher in university Teacher currently
taking university
classes (pct)
18.0 17.1 15.3 19.5
Teacher experience Years of experience in
this school
4.9 (5.6) 2.3** (1.5) 6.1 (7.7) 4.2** (4.1)
Teacher is Indigenous Teacher describes
his/herself as
Indigenous (pct)
41.4 56.8** 22.0 51.5**
Teacher content knowledge (Spanish) Percentage correct of
12 items from student
Spanish test
94.2 (3.8) 94.2 (7.0) 94.1 (3.8) 94.4 (3.9)
Teacher content knowledge (math) Percentage correct of
10 items from student
Math test and 16 items
from middle school
level
69.7 (11.7) 66.0** (10.4) 70.5 (12.2) 69.4 (11.6)
Teacher pedagogical content knowledge (PCK) Sum of points (out of 8)
for three activities
designed to measure
specialized teaching
knowledge in grade 3
mathematics
5.2 (2.0) 4.3** (1.6) 5.7 (1.7) 4.2** (1.9)
Teaching segments
Seat work Percent of class
devoted to students
working individually
40.6 31.5** 45.2 37.8**
Teacher–student interaction Percent of class teacher
asks individual
questions, group
responses or students
work at board
26.6 27.4 25.3 28.1**
Group work Percent of class
devoted to students
working in groups
4.4 0.0** 10.3 1.6**
Teacher-centered Percent of class
devoted to teacher
giving instructions,
resolving problems, etc.
24.0 34.2** 15.9 27.9**
Transition/discipline Percent of class
devoted to transitions
in between activities or
stopped time
4.4 6.9** 3.3 4.6**
Teacher checks all work Teacher checked all
students work during
lesson (pct)
25.1 18.5** 33.3 22.4**
Physical condition of classroom Average condition of
classroom based on
observations of space,
lighting, noise and
desks
2.6 (0.6) 2.9** (0.6) 2.6 (0.6) 2.6 (0.6)
School days Total days of class from
first day of class until
day of test
111.2 (9.6) 112.8* (6.7) 112.5 (8.3) 110.5** (10.2)
Total enrollment Total grade 1–6
enrollment
249.4 (131.8) 204.4** (145.2) 254.4 (146.4) 242.3 (126.4)
PRONADE school School is part of
PRONADE system (pct)
10.6 – 4.8 14.6**
Distance to middle school Kilometers to nearest
middle school
5.0 (6.0) 7.2** (8.9) 3.5 (3.8) 6.1** (6.9)
Source: HCRG Databases, 2003. Notes: Means are presented with standard deviations (when appropriate) in parentheses. Asterisks for PRONADE schools
and Indigenous, students refer to significant differences compared with whole sample average. *Difference p< 0.10 level; **Difference p< 0.05.
school/traditional school comparison is especially entic-
ing given the stated purpose of community schooling to
improve capacity utilization. I use a simplified decom-
position framework that pools the data by school type
and divides the achievement differences between PRON-
ADE and traditional public schools into two sources:
explainable differences based on endowments of inde-
pendent variables, and an unexplainable fixed or “direct”
effect (Blinder, 1973; Neumark, 1988; Oaxaca, 1973). The
gain score production function estimates are used for
Author's personal copy
212 J.H. Marshall / Economics of Education Review 28 (2009) 207–216
this activity, and the variation is divided into four cate-
gories:
¯
A1−¯
A2=ˆ
ˇPRONADE +(¯
X1−¯
X2)ˆ
ˇ#
X
+(¯
S1−¯
S2)ˆ
ˇ#
S+(¯
C1−¯
C2)ˆ
ˇ#
C(3)
where the achievement difference between group 1 (tra-
ditional schools) and group 2 (PRONADE) is a function
of the direct effect of PRONADE in (1) and differences
in endowments for family background, schools and com-
munities multiplied by the coefficients for each variable,
all taken from Eq. (2) (with incoming achievement) from
above.
Table 1 shows that indigenous students score much
lower than their ladino counterparts. This is consistent with
previous standardized testing in Guatemala and elsewhere,
although the language gap in this sample of rural schools
is larger than in any other previous study in the region. The
same decomposition framework (3) is applied to explain
the indigenous test score gap. The basic interpretation is
the same except for the treatment of the “direct” effect
for indigenous students, which represents the difference
in achievement between indigenous and ladino students
with equal family background endowments who are study-
ing in the same school. All of the decomposition results are
presented in Table 3 in the next section.
4. Results
4.1. The covariates of student achievement
The production function results for Spanish and
mathematics are presented in Table 2. In each subject
achievement is analyzed in both level and gain score form.
Dependent variables are measured in standard deviations.
T-statistics are based on robust standard errorsthat account
for student clustering, and the models also include state
(departamento) dummy variables (Alta Verapaz excluded).
Upwards of 52% of the variation is explained in level
form, and between 23 and 31% of the gain score varia-
tion. As expected, indigenous students score significantly
lower than ladinos. In Spanish language the unexplained
difference – measured by the indigenous student dummy
variable – is 0.25 standard deviations in the gain score
model. The corresponding effect in mathematics is much
smaller and insignificant. The very different results by sub-
ject cast some doubt on differential treatment inside the
classroom. The stronger impact in language points instead
to linguistic factors. Nevertheless, the size of this effect in
Spanish is still troubling given the extensive controls.
With information on student attendance and days of
class the HCRG data make it possible to analyze the
attendance–achievement link with unusual detail. A stan-
dard deviation increase in days of class (about 10 days)
predicts 0.20 standard deviations higher gain scores in
Spanish, and a 0.11 higher level score in mathematics. The
student attendance measure is only significant in the math-
ematics level model, although when combined with days
of class the total attendance effect in standardized terms
is roughly 0.22 standard deviations. However, what if stu-
dents are attending less frequently as a response to the
school being closed more often? The question is important
enough to warrant further exploration, and in separate esti-
mations (not presented) using attendance as the dependent
variable I find that student attendance is higher in schools
that work more days. The only category of school closings
that significantly predicts less frequent student attendance
is the unexplained category where the teacher did not write
in the reason for no class. These kinds of linkages are only
suggestive, but the most serious implicationis that students
attend less frequently when schools have more unofficial –
or unannounced – closings.
In rural Guatemala the teacher’s ethnicity and language
use are potentially important elements of quality. Table 2
shows that students studying with (self-described) indige-
nous teachers have higher Spanish and mathematics scores.
However, for this prima facie evidence of bilingual educa-
tion effectiveness to be convincing it needs to be shown
that indigenous students fare especially well when study-
ing with indigenous teachers. Additional estimations using
interaction terms show evidence of positive interaction
only in the mathematics gain score estimation. A better test
of the impact of bilingual education is based on actual lan-
guage use in the classroom. Teachers were asked if they
ever use indigenous languages in instruction, and during
the classroom observations the enumerators noted the lan-
guage being used during class. The teacher’s reported use
of Mayan languages (yes–no) is significant in mathemat-
ics only, and only when it is included without the teacher
ethnicity control. So overall the results are inconclusive
with regards to the effectiveness of bilingual education pro-
grams in rural Guatemala.
Other variables touch on actual abilities and still more
specific elements of the teacher’s work. As expected,
teacher content knowledge is associated with higher levels
of student achievement. For mathematics a standard devi-
ation increase in content knowledge predicts 0.11 standard
deviations of gains. For pedagogical content knowledge
(PCK) the point estimate is significant, but the effect size is
modest—only 0.03 standard deviations in student gains for
each point increase in teacher PCK. The results for the teach-
ing segments show that the most effective classes appear to
be those that limit group work and stress teacher–student
interaction and teacher-centered lecture and explanation
activities (in mathematics). Student achievement is also
lower in classes that have more “down time” due to transi-
tions between activities or disciplinary actions. More direct
instruction methods have been associated with learning
in the United States (Goldhaber & Brewer, 1997), while
the negative coefficient for group work is corroborated by
recent work with the TIMSS international mathematics test
score data (Carnoy, Marshall, & Socias et al., 2007). Finally,
when teachers are observed checking all students work the
average achievement is significantly higher. The effect size
is quite large – upwards of 0.35 standard deviations – which
provides powerful evidence of the effectiveness of this par-
ticular strategy.
The remaining variables in Table 2 cover the commu-
nity characteristics and controls for selection and school
type. The results for selection bias are uneven, although
they do point to a negative effect on average achievement
when the school does a better job of retaining students. This
Author's personal copy
J.H. Marshall / Economics of Education Review 28 (2009) 207–216 213
Table 2
OLS estimates of student achievement, by subject
Variables Spanish Mathematics
Level Gains Level Gains
Student-family variables
Spanish test 2001 – −0.03*** (−9.67) – –
Math test 2001 – – – −0.03*** (−13.82)
Student age −0.02 (−0.74) −0.03 (−0.94) −0.001 (−0.01) −0.04 (−1.53)
Female −0.14*** (−2.73) −0.02 (−0.28) −0.32*** (−5.30) −0.15*** (−2.66)
Indigenous −0.39*** (−3.04) −0.25** (−2.05) −0.14 (−1.13) −0.02 (−0.19)
Parental education 0.04** (2.56) 0.03 (1.26) 0.03** (2.04) 0.02 (1.07)
SES 0.05*** (2.62) 0.03 (1.61) −0.003 (−0.12) 0.002 (0.08)
Attendance
Student attendance 0.003 (0.68) 0.001 (0.39) 0.01* (1.83) 0.01 (1.19)
School days 0.02*** (5.83) 0.02*** (4.67) 0.01** (2.06) 0.006 (0.80)
Teacher and classroom characteristics
Class size 0.004 (0.98) 0.01* (1.69) −0.0 04 (−0.06) −0.001 (−0.01)
Student-reported fighting (school average) −0.01 (−0.37) −0.01 (−0.32) 0.03 (0.68) 0.04 (1.28)
Teacher in university 0.09 (0.93) 0.14 (1.46) 0.10 (0.70) −0.05 (−0.58)
Teacher experience 0.01 (0.42) 0.02** (2.25) −0.01 (−0.67) −0.004 (−0.43)
Teacher is Indigenous 0.15* (1.76) 0.11 (1.30) 0.21 (1.62) 0.23*** (2.68)
Teacher content knowledge 0.001 (0.13) 0.01 (1.06) 0.01* (1.74) 0.01** (2.15)
Teacher pedagogical content knowledge (PCK) – – −0.01 (−0.29) 0.03* (1.79)
Teaching segmentsa
Teacher–student interaction −0.001 (−0.22) 0.003 (1.30) 0.005 (1.43) 0.01*** (4.22)
Group work −0.008*** (−3.71) −0.01*** (−5.21) −0.001 (−0.38) −0.006 (−1.48)
Teacher-centered −0.005 (−1.47) −0.002 (−0.28) 0.01** (2.02) 0.01*** (3.21)
Transition/discipline −0.02*** (−2.56) −0.03** (−1.99) −0.03** (−2.26) −0.05*** (−4.44)
Teacher checks all work 0.35*** (3.80) 0.20** (2.32) 0.28* (1.90) 0.34** (2.27)
Physical condition of classroom 0.07 (1.41) 0.09* (1.79) 0.06 (0.65) 0.11 (1.38)
School and community characteristics
Average parental education 0.03 (0.61) 0.07 (1.11) −0.01 (−0.08) −0.08 (−0.97)
Distance to middle school −0.05*** (−4.82) −0.03*** (−2.91) −0.008 (−0.47) 0.004 (0.37)
Total enrollment −0.001** (−2.31) −0.001** (−1.86) −0.001 (−0.47) −0.001 (−0.41)
PRONADE school −0.30** (−2.26) −0.23 (−1.92) −0.44** (−2.46) −0.17 (−0.97)
State controlsb
Chimaltenango 0.23 (1.51) 0.26* (1.78) 0.08 (0.39) 0.14 (0.83)
Escuintla 0.50*** (2.97) 0.57*** (3.41) 0.83*** (3.15) 0.98*** (3.90)
Selection correction
Cohort control −0.22** (−2.10) −0.26*** (−3.09) 0.06 (0.38) −0.22** (−2.21)
Day of test control 0.004 (1.07) 0.008 (1.58) −0.01 (−1.37) −0.01*** (−3.00)
Non-selection hazard (")−1.89*** (−2.75) −1.10 (−1.20) −0.30 (−0.29) 0.08 (0.09)
LR test for instrument power (chi-square) 9.86** 9.15** 5.67 5.61
Wald test for instrument exclusion (F-test) 0.51 1.37 0.45 0.91
Wald test excluding lambda (F-test) 1.07 1.22 0.50 0.94
Sample size 839 839 803 803
R20.517 0.233 0.276 0.310
Source: HCRG, 2003. Notes: Dependent variables measured in standard deviations. Additional predictors include the number of older and younger siblings,
textbooks, teacher gender and the distance to Guatemala City (the national capital). “Level” models use 2002 score as dependent variable; “Gains” refers
to difference between 2002 and 2001 scores. The LR Test compares first stage selection estimations with and without identifying instruments to assess
instrument power (as a group). Wald tests are used to test excludability of instruments from second stage achievement estimations, first with the predicted
non-selection hazard (") and then without. T-statistics (in parentheses) are based on robust standard errors that correct for clustering within schools. See
Table 1 for variable measurement specifics.
aExcluded category for time segment analysis is individual seatwork.
bExcluded category for state controls is Alta Verapaz.
and the very large point estimates for the all-ladino state of
Escuintla will be returned to in the decomposition activity.
As for the PRONADE schools there is some evidence that
achievement is lower when all else is equal.
4.2. Academic achievement decompositions
Table 3 presents the decompositions for student
achievement. In each comparison the raw difference (in
standard deviations) is presented for the gain scores
only. Negative coefficients favor the lower scoring schools,
meaning indigenous schools and PRONADE. The overall
flavor of each set of comparisons – especially forthe (poten-
tial) policy levers – is little changed by model specification
or decomposition technique.
Based on the fixed effects analysis of McEwan and
Trowbridge (2007), and the more detailed decomposition
in Hernandez-Zavala, Patrinos, Sakellariou, and Shapiro
Author's personal copy
214 J.H. Marshall / Economics of Education Review 28 (2009) 207–216
Table 3
Academic achievement decompositions
Variables Indigenous test score gap PRONADE test score gap
Spanish gains Math gains Spanish gains Math gains
Raw difference in standard deviations 0.21 0.25 0.14 0.07
By variable category
Student/family background −0.47 −0.11 −0.31 −0.24
Community characteristics 0.40 0.46 0.09 0.13
Selection controls −0.01 0.04 0.11 −0.02
School characteristics 0.04 −0.16 0.02 0.04
Unexplained “direct” parameter 0.25 0.02 0.23 0.17
Select school quality differences
School days 0.04 0.01 −0.04 −0.01
Teacher is Indigenous −0.02 −0.06 −0.02 −0.04
Teacher content knowledge 0.00 0.01 0.00 0.05
Teacher PCK – 0.02 – 0.02
Teaching segments
Group work −0.10 −0.05 −0.06 −0.03
Teacher-centered 0.03 −0.15 0.02 −0.10
Transition/discipline 0.05 0.07 0.07 0.21
Teacher checks all work 0.03 0.04 0.03 0.05
Source: HCRG, 2003. Notes: Decompositions based on achievement levels are available on request from author. Gains refer to difference between 2002 and
2001 scores. Students are classified as indigenous when they report speaking a Mayan language in the home. The top row refers to the raw difference in
average achievement between each group (in standard deviations). This total difference is decomposed into the five variable categories listed in the top half
(Student-family background, community, etc.). The incoming (2001) test score is included in the student-family background category. The decomposition
uses a single estimation equation and focuses only on endowment differences between the two groups. All coefficients refer to standard deviation changes
in achievement. Negative coefficients refer to areas where the low scoring group (indigenous and PRONADE) have more favorable endowments. See text
for more information, Table 2 for the coefficients and Table 1 for the means for each group.
(2006), we know that school and community characteris-
tics figure somewhat prominently in explaining test score
differences by ethnicity. With the HCRG data this activity
can be taken one step further using multiple features of
schools and classrooms that are significant predictors of
achievement gains. As argued above, the largerunexplained
(or “within school”) result for Spanish suggests that this
subject is more affected by the linguistic limitations that
indigenous students bring to school. But we cannot rule out
a pedagogical component to this, especially if differential
treatment in the classroom revolves around language use.
The remaining family background variables play a rela-
tively minor role in determining the indigenous test score
gap. The school variables as a group also do little to explain
why indigenous students have lower scores. This does not
mean that all endowments are equal, however, and the
detailed results at the bottom of Table 3 provide some clues
into areas of schooling that can be improved in indigenous
communities. For example, the non-indigenous schools
gain as much as 0.04 standard deviations by working more
days. The classes are also more orderly and the ladino school
teachers are more likely to check all student work. But
there are offsetting differences where the predominantly
indigenous schools have more favorable endowments. One
advantage is they have more indigenous teachers. Indige-
nous classrooms also rely less on group work and use more
teacher-centered instruction, each of which provide size-
able gains.
The largest gains for ladinos come instead from com-
munity characteristics. The biggest advantage is their
concentration in the state of Escuintla, which accounts for
upwards of 0.50 standard deviations. Indigenous students
also fare worse due to their schools being located farther
away from middle schools. The critical question therefore
is the extent to which these community and state controls
are capturing unmeasured elements of school and teacher
quality. This does not seem likely given the scope of infor-
mation that is available for days worked, teacher quality
and classroom processes. The centralized nature of policy-
making in Guatemala also argues against inter-state quality
effects of the magnitude suggested by the Escuintla dummy
variable parameters. The more likely explanation takes us
back to the kinds of unmeasured local influences identified
in previous studies, such as peer effects, cultural attitudes
towards schooling or variation in labor market dynamics.
The right hand side of Table 3 includes the decom-
positions by school type. As hypothesized, there is some
evidence that community schools realize achievement
gains as a result of more efficient capacity utilization. They
report more days of class, which accounts for a small advan-
tage in achievement. There is also less reliance on student
group work and more use of teacher-centered instruction,
two methodological choices that could – in theory – result
from having more motivated teachers. However, the uti-
lization argument is weakened by the greater tendency of
non-PRONADE school teachers to check all of the student’s
work, which in turn predicts upwardsof 0.05 standard devi-
ations of achievement gain advantages.
For Spanish achievement there is one aspect where
greater PRONADE efficiency has a detrimental effect on
average test scores: grade completion. Average grade com-
pletion is significantly higher in PRONADE schools among
the randomly drawn students who entered grade one in
1999 (Table 1). This greater efficiency in getting children
through school does appear to come with a tradeoff in
achievement gains. In the decomposition the effect of this
Author's personal copy
J.H. Marshall / Economics of Education Review 28 (2009) 207–216 215
variable is 0.11 standard deviations. This is a tentative link-
age given the different results for mathematics. But the
important point is that community schools may be espe-
cially focused on getting children from grade to grade and
reducing dropout, which in turn is likely to handicap their
ability to maximize achievement.
Finally, PRONADE teacher endowments are also gener-
ally inferior, at least in several aspects that are related to
student achievement. In mathematics their lower levels
of content knowledge result in a 0.05 standard deviation
disadvantage. The classes are also more disorderly, which
in turn predicts upwards of 0.21 standard deviations in
achievement differences. This disorder must be qualified
somewhat, since the PRONADE teachers are more likely to
work in a multigrade setting, which in turn requires more
frequent transitions between activities. Overall the results
are consistent with the general research contours that are
forming related to community schooling in Central Amer-
ica. There are some things that these schools do well, and
these advantages in general predict more student achieve-
ment. But capacity utilization has limits, especially when
the schools are either intentionally (to save money) or unin-
tentionally (because of access) making greater use of lower
quality teachers.
5. Conclusion
This paper uses unusually rich data from rural
Guatemalan primary schools to analyze student achieve-
ment in a poor, developing country context. The detailed
data make it possible to help fill in missing elements
in several existing literatures. For example, a handful of
previous studies have found that teachers are frequently
absent in developing country schools; the results here
corroborate this finding in rural Guatemala while demon-
strating its consequences for student achievement. The
results linking two different forms of teacher mathemat-
ics knowledge with student achievement gains also help
break some new ground for a frequently mentioned – but
rarely analyzed – feature of teacher quality. For policy-
makers in Guatemala and beyond these kinds of linkages
serve as useful reminders of the potential impact of well-
designed interventions in school management and teacher
preparation. For researchers the results provide one more
example of how exceptional data collections can help
inform research questions that are frequently left unad-
dressed in the production function framework.
The analysis also addresses one of the most pressing
social questions in contemporary Guatemalan society: the
poor performance of indigenous students on standardized
tests. The results suggest no simple policy prescription for
equalizing test scores among indigenous and ladino stu-
dents in rural areas. Most importantly, the schools attended
by each group are very similar, at least based on an exten-
sive list of school and teacher characteristics. This is itself an
important finding, since previous decomposition activities
in Guatemala have had few school quality mechanisms to
consider. There is some evidence that indigenous teachers
are more effective, and math gain scores are significantly
higher for indigenous students when they are paired with
indigenous teachers. But assessing the overall effectiveness
of this intervention – let alone identifying the most impor-
tant mechanisms – will have to wait for better data on
implementation.
If indigenous and non-indigenous schools in rural
Guatemala have similar measured endowments, then what
can policymakers do to redress the historical legacy of
massive inequalities in educational opportunities? The
significant marginal differences in achievement between
indigenous and ladino, when all else is equal, raise the
possibility of different treatment inside the classroom. But
additional cultural and institutional factors also appear to
be at work, especially when considering the large commu-
nity effects in these comparisons. Indigenous students and
their families may be less sanguine about the future pay-
offs to schooling, perhaps because cultural and/or physical
isolation reduces access to urban labor markets. Or these
communities may be neglected in other institutional areas,
such as health care access. Each points to aspects of the
larger opportunity structure that go beyond simply target-
ing these communities for more educational resources.
The results also shed some light on decentralization
dynamics in a poor, rural context. Expectations that PRON-
ADE community schools realize efficiency gains that are
offset by endowment deficiencies in critical areas of teacher
competence are largely confirmed. Once again the evi-
dence does not point to dramatic differences in schools by
program, and more qualitative evidence about how PRON-
ADE communities conceptualize school quality is clearly
needed. Given the tendency of PRONADE to enroll indige-
nous students this question once again touches on the
need to understand the impact of underlying motivations
and access to information on schooling behaviors. Capac-
ity deficiencies in PRONADE schools may be a product
of parents having low expectations about future returns
to schooling, or community councils made may be hard
pressed to instigate changes to improve teaching quality.
Acknowledgements
Useful comments were provided on earlier versions
by Martin Carnoy, Susanna Loeb, Eric Hanushek, Miguel
Socias, Arjun Bedi, Jo Boaler, Nancy Tuma, and two anony-
mous referees. All remaining errors are of course my own.
In Guatemala, Yetilu de Baessa and her research team at
the PRONERE evaluation project provided crucial logistical
assistance with the data collection. Alejandra Sorto pro-
vided the items for math teachers, which are part of a larger
study we are working on together. Partial funding was pro-
vided by the Spencer Foundation. The views expressed here
do not reflect those of the RAND Corporation. The HCRG
data are available at http://www.sapere.org.
References
Alcazar, L., Rogers, F. H., Chaudhury, N., Hammer, J., Kremer, M., & Muralid-
haran, K. (2005). Why are teachers absent? Probing service delivery in
Peruvian primary schools. Washington, DC: The World Bank.
Bedi, A. S., & Marshall, J. H. (2002). Primary school attendance in Honduras.
Journal of Development Economics,69, 129–153.
Blinder, A. S. (1973). Wage discrimination: Reduced form and structural
estimates. The Journal of Human Resources,8, 436–455.
Carnoy, M., Marshall, J. H., & Socias, M. (2007). Explaining differences in
academic achievement using international test data. Working paper.
Author's personal copy
216 J.H. Marshall / Economics of Education Review 28 (2009) 207–216
De Baessa, Y. (2002). Summary of 2001 PRONERE test application. Universi-
dad del Valle, Guatemala. Working paper.
DiGropello, E., & Marshall, J. H. (2005). Teacher effort and schooling
outcomes in rural Honduras. In E. Vega (Ed.), Incentives to improve
teaching: Lessons from Latin America. Washington,DC: The World Bank.
DiGropello, E., Marshall, J. H., & Rápalo, R. (2004). The community based
school management study: Evidence on progress and the impact of the
models. Washington, DC: The World Bank.
Fuller, B., & Clarke, P. (1994). Raising school effects while ignor-
ing culture? Local conditions and the influence of classroom
tools, rules and pedagogy. Review of Educational Research,64,
119–157.
Fuller, B., Dellagnelo, L., Strath, A., Bastos, E. S. B., Maia, M. H., Lopes de
Matos, K. S., Portelaand, A. L., & Vieira, S. L. (1999). How to raise chil-
dren’s early literacy? The influence of family, teacher, and classroom
in northeast Brazil. Comparative Education Review,43, 1–35.
Glewwe, P. (2002). Schools and skills in developing countries: Education
policies and socioeconomic outcomes. Journal of Economic Literature,
90, 436–482.
Glewwe, P., Grosh, M., Jacoby, H., & Lockheed, M. (1995). An eclectic
approach to estimating the determinants of achievement in Jamaican
primary education. World Bank Economic Review,9, 231–258.
Goldhaber, D. D., & Brewer, D. J. (1997). Why don’t schools and teachers
seem to matter? Assessing the impact of unobservableson e ducational
productivity. The Journal of Human Resources,32(3), 505–523.
Hanushek, E. A., & Woessmann, L. (2007). The role of education quality in
economic growth. World Bank Policy Research Working Paper No. 4122.
Washington, DC: The World Bank.
Harbison, R., & Hanushek, E. A. (1992). Educational performance of the poor:
Lessons from rural northeast Brazil. New York: Oxford University Press.
Hernandez-Zavala, M., Patrinos, H. A., Sakellariou, C., & Shapiro, J. (2006).
Quality of schooling and quality of schools for indigenous students in
Guatemala, Mexico and Peru. Washington, DC: The World Bank.
Hill, H., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical
knowledge for teachingon student achievement. American Educational
Research Journal,42, 371–406.
Jimenez, E., & Sawada, Y. (1999). Do community-managed schools work?
An evaluation of El Salvador’s EDUCO program. World Bank Economic
Review,13, 415–441.
Kremer, M., Chaudhury, N., Rogers, F. H., Muralidharan, K., & Hammer, J.
(2005). Teacher absence in India: A snapshot. Journal of the European
Economic Association,3, 658–667.
Marshall, J. H., & Sorto, A. M. (2007). Teaching what you know or know-
ing how to teach? The effects of different forms of teacher mathematics
knowledge on student achievement in rural Guatemala. Working paper.
http://www.sapere.org.
McEwan, P. J., & Trowbridge, M. (2007). The achievement of indigenous
students in Guatemalan primary schools. International Journal of Edu-
cational Development,27, 61–76.
Mullens, J. E., Murnane, R. J., & Willett, J. B. (1996). The contribution of
training and subject matter knowledge to teaching effectiveness: A
multilevel analysis of longitudinal evidence from Belize. Comparative
Education Review,40, 139–157.
Neumark, D. (1988). Employers’ discriminatory behavior and the esti-
mation of wage discrimination. The Journal of Human Resources,23,
279–295.
Oaxaca, R. (1973). Male–female wage differentials in urban labor markets.
International Economic Review,14, 693–709.
Psacharopoulos, G., & Patrinos, H. A. (Eds.). (1994). Indigenous people and
poverty in Latin America: An empirical analysis. Washington, DC: The
World Bank.
Santiba˜
nez, L. (2006). Why we should care if teachers get A’s: Teacher test
scores and student achievement in Mexico. Economics of Education
Review,25, 510–520.
World Bank. (2004). Poverty in Guatemala. Washington, DC: The World
Bank.
