Content uploaded by Katharina Michaelowa
Author content
All content in this area was uploaded by Katharina Michaelowa
Content may be subject to copyright.
The cost-effectiveness of inputs in primary education:
Insights from recent student surveys for sub-Saharan Africa
Sebastian Fehrler, Katharina Michaelowa and Annika Wechtler
Abstract
With SACMEQ and PASEC there are now two large data bases available on
student achievement, socio-economic background and school and teacher
characteristics in both anglophone and francophone Sub-Saharan Africa.
A joint analysis of PASEC and SACMEQ in a common education production
function framework allows us to estimate the impact of educational inputs on student
achievement in 22 sub-Saharan African countries, and to compare our results with those
of earlier empirical studies for education systems in Africa and other world regions. Our
empirical results based on a traditional retrospective analysis of student achievement in
PASEC and SACMEQ countries are broadly in line with earlier analyses. In addition, we
find some evidence for the effectiveness of incentive based policy options. Some
differences between francophone and anglophone education systems can be observed
with respect to the impact of the teachers' own education and training if differences in
the sampling methodology are duly taken into account. These differences may
themselves be related to education quality. Other inputs show similar effects in both
linguistic areas.
1
Introduction
The increasing availability of student survey data, the development of new
statistical and econometric methods and the expansion of computing capacities has led
to a huge increase in scientific evaluations of the determinants of education quality in
recent years. Education quality is thereby measured in terms of student achievement on
standardized tests, which reflects the cognitive knowledge acquired through the
education process. In line with international policy priorities as codified in the Education
for All (EFA) objectives and the Dakar Framework for Action, for sub-Saharan Africa,
evaluation efforts currently concentrate on the primary level. In addition to various
national level evaluations, three programs have been launched on a larger scale: The
UNESCO/UNICEF Monitoring Learning Achievement (MLA), the Southern and Eastern
Africa Consortium for Monitoring Educational Quality (SACMEQ) and the Programme
d'Analyse des Systèmes Éducatifs de la CONFEMEN (PASEC) now jointly cover most
countries on the continent. General information on these programs is available from
Chinapah (1997) for MLA, Ross (1998) and Murimba (2000a,b) for SACMEQ, and
PASEC (1999) and CONFEMEN (2005) for PASEC.
SACMEQ and PASEC are of particular interest because they use comparable (or
identical) tests in all their countries, which allows us to jointly analyse different country
cases as well as to draw comparisons across countries. While the comparison of test
items and thus a direct comparison of achievement levels across programs is not yet
possible, the relationship between inputs and outcomes can be compared.
A joint analysis of PASEC and SACMEQ data in a common education production
function framework allows us to estimate the impact of educational inputs on student
achievement in 22 sub-Saharan African countries, and to compare our results with those
of earlier empirical studies for education systems in Africa and other world regions.
2
As SACMEQ data have only recently become publicly available, to our knowledge, this
study presents the first attempt to jointly explore results for francophone and anglophone
Africa in a common education production function framework.
The Impact of traditional school resources on student learning
There is a considerable number of studies on the impact of traditional school
resources on student learning including excellent literature reviews such as UNESCO
(2004), Hanushek (2003), Glewwe and Kremer (2006). As outlined in most of the
literature reviews, generally, the results of different empirical studies are highly
inconsistent, and the overall picture is rather bleak in terms of truly promising policy
options. In fact, many of the studies raise doubts about the relevance of traditional inputs
in the schooling production function all together (Hanushek 2003, Glewwe and Kremer
2006, Glewwe, Kremer, Moulin and Zitzewitz 2004). Although there have been large
improvements in the levels of school resources around the world, no corresponding
improvement of student learning could be observed. As Hanushek puts it:
‘Class sizes have fallen, qualifications of teachers have risen, and
expenditures have increased. Unfortunately, little evidence exists to suggest that any
significant changes in student outcomes have accompanied this growth in resources
devoted to schools.’ (Hanushek, 2003: F67)
This is especially true for countries in which the level of school resources
is already high. One should on the other hand expect the relationship between resources
and outcomes to be much clearer for developing countries as the low initial level of
resources makes it more likely that additional inputs have a significant effect. Indeed,
looking at 96 production function estimates in less developed countries reveals a
somewhat stronger support for the expected positive relationship between inputs and
3
achievement (Hanushek, 2003: F84). Analysing 60 studies of education in developing
countries, Fuller (1987) also finds that resources were more important determinants of
students' achievement in developing countries than in industrialized countries. Fuller and
Clarke (1994) reinforce this conclusion taking into account the cross-country differences
in socio-economic and cultural settings even within developing countries.
We conclude that despite rather discouraging evidence on the international level,
for developing countries in general, and for most of the very poor sub-Saharan African
countries in particular, school resources still play an important role in improving
education quality. However, even for these countries, the estimated relationship between
school resources and student achievement is far from consistent across studies, so that
there is no easy recipe for successful policy interventions.
Data and econometric methods
In our paper we will examine the evidence from PASEC and SACMEQ data,
using a common education production function framework, to assess whether the results
from the literature are consistent with results from this unique dataset covering a large
part of sub-Saharan Africa. To start with, let us discuss the data coverage and sampling
methods as well as our econometric approach.
Data
The SACMEQ data base includes more than 40 000 sixth grade students from 13
countries: Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia,
Seychelles, South Africa, Swaziland, Tanzania (main land and Zanzibar), Uganda and
Zambia. The PASEC data used here includes more than 17 000 fifth grade students and
the same number of second grade students from eight countries: Burkina Faso,
Cameroon, Côte d'Ivoire, Madagascar, Mali, Niger, Senegal and Togo. All surveys were
4
carried out between 1995/96 and 2001/02. For both sets of countries, we estimate the
effect of various policy options on student test scores in literacy and mathematics. The
policy options discussed include the provision of better learning materials (for example
textbooks, teacher manuals), school equipment (for example electricity), teacher
qualification and the organisation of student flows (for example multi-grade teaching,
repetition, effective teaching time). Moreover, they include some institutional variables
that relate to incentive based reform options. All effects are calculated after controlling
for the influence of student socio-economic background, for example possessions at
home, mothers' and fathers' education, language spoken at home and so forth.
In PASEC, students have been tested twice, once at the beginning, and once at
the end of the academic term (pre-test and post-test). This allows us to also control for
student performance before the school that is before teacher and classroom related
influences measured for the corresponding term actually start to become relevant. As a
comparable variable is not available for SACMEQ, PASEC results are computed twice,
with and without the pre-test variable. This procedure ensures that real differences
between the two country groups can be distinguished from differences which are merely
induced by the introduction of the pre-test variable.
Regression methodology
For both sets of countries, the dependent variable used in our regressions is the
test score in literacy and mathematics. This test score is coded on a scale with mean
500 and standard deviation 100 for SACMEQ. For PASEC, the simple percentage of
correct answers (0-100 per cent) is used with a cross-country average of: 37 per cent
(French) and 39 per cent (maths) and a standard deviation of 20.5 per cent for fifth
grade in both subjects; and an average of 47.2 per cent (French) and 45 per cent
5
(maths) and a standard deviation of 27 per cent and 26 per cent respectively for second
grade.
All countries within each country group are considered jointly in a single
regression. This has the considerable advantage that, due to the high total number of
observations, even very small effects can be distinguished. Country differences are
captured by country fixed effects.
We use two different econometric models to estimate the education production
functions. For both SACMEQ and PASEC, model type A is the usual hierarchical linear
(or multi-level) model with school random effects (for textbook expositions see for
example Raudenbush and Bryk, 2002 or Goldstein, 2003). Estimations are carried out
with generalized least squares (GLS) with the exception of SACMEQ regressions
because the availability of sampling weights makes maximum likelihood estimation
(MLE) computationally more attractive in the multi-level framework.
Model A has the advantage of providing a clear distinction between the
explanations of the variance within and between schools. However, the true standard
errors may be underestimated if sub-clusters exist (such as classes within schools for
SACMEQ - see below; or groups of students living in the same area or doing their
homework together), which lead to a variance structure different from the one explicitly
specified. As a robustness check, we therefore introduce a model type B using the Stata
survey sampling command. The underlying estimation procedure takes into account the
nested structure of the data without separate computations of the variances at the
different levels. However, it has the advantage of being robust with respect to any type
of sub-clustering, as it uses the Huber-White sandwich estimator for the estimation of the
variance-covariance matrix. For details, see the Stata handbook on survey sampling
6
(StataCorp., 2003: 38). For a comparison of the different methodologies and their
results, see Brown and Micklewright (2004).
Differences in PASEC and SACMEQ evaluation design
Some conceptional differences in PASEC and SACMEQ evaluation methodology
and survey design may have a non-negligible impact on estimates of regression
coefficients and standard errors, as well as on their interpretation. We will discuss them
one by one in the following.
Pre-test scores
As mentioned, only PASEC tests students twice, once at the beginning and once
at the end of the year. The inclusion of a pre-test score in a regression function has
important implications. First, it is a relevant control variable for general ability and the
influence of student background which might not have been fully captured otherwise. Its
inclusion can avoid (or reduce) omitted variable bias when estimating the effects of
relevant policy measures. Second, it changes the interpretation of all coefficients as the
control for the score at the beginning of the term implies that the coefficients of all other
variables reflect the influence on students' progress over the year, rather than on
students' final skills. This is why econometric models including a pre-test score are also
known as "added value models". And third, many teacher and classroom related
variables change over the years, so that a precise estimation of their impact is only
possible for the ongoing term. For example, the student may have got a high performing
teacher for the current term, but had bad teachers before. Now since the overall skills of
this student are influenced by all these teachers the positive influence of the last teacher
will be blurred in any model in which initial student skills (before they got this teacher)
cannot be taken into account.
7
Student weights
Only SACMEQ includes student weights, which can be used in the regression in
order to ensure that the overall results are truly representative. Most PASEC surveys are
designed to be representative surveys of schools, but it is not taken into account that the
probability of any particular student to be part of the sample also depends on the size of
the school. For Togo, Mali and Niger, they are not representative for schools, either,
because they were designed to study specific policy measures (that is contract teachers
and double shift teaching). This may result in some selection bias, and there are no
weights to adjust for the non-random selection ex-post.
Target population and exclusions
An obvious difference is that PASEC and SACMEQ focus on different grades.
Clearly, differences must be expected between students' learning in the early grades
(like PASEC second grade) and later grades. However, the differences between PASEC
fifth grade and SACMEQ sixth grade appear to be less substantial. While drop-out
increases from year to year, overall completion rates are higher in SACMEQ than in
PASEC countries, so that the effect of sampling students from a one-year higher grade
should be about compensated. Another concern could be that in many countries, sixth
grade is the last year of primary education, which may make it an atypical year, difficult
to compare with other years. However, it turns out that in most SACMEQ countries,
primary schooling includes one more year and ends only after 7th grade. Thus in this
respect, there does not seem to be a major problem for comparisons between PASEC
and SACMEQ.
A more relevant issue for our current analysis is that PASEC is sampling
students within a single class for each school while SACMEQ is randomly drawing
students from the overall sixth grade population within each school in the sample. This
8
implies that for a given number of students drawn in each school and grade (typically 20
students in both surveys), in SACMEQ, we have more variation between teacher and
classroom environments, but with only few students to whom this information can be
directly related. Conversely, in PASEC we have information on the students actually
taught by the same teacher in exactly the same environment. These differences lead to
different degrees of precision for our econometric estimates at the different levels
(schools, teachers/ classrooms, and students).
In SACMEQ regressions, schools are the only level explicitly considered in the
hierarchical models, and the primary sampling units in the survey regressions. In
PASEC, the hierarchical level and the primary sampling unit considered is the
classroom. The overall impact is difficult to predict. In any case, for SACMEQ, simple
two-level hierarchical estimation models which do not take into account any sub-group
clustering within schools appear to be problematic. This is the reason for the introduction
of an alternative specification using Stata's survey sampling procedures which are robust
to any correlations within primary sampling units.
Finally, neither in SACMEQ nor in PASEC all schools are included in the defined
target population. In PASEC, sampling relies on school mappings available at the
ministries of education, which, in some countries, exclude private schools. In SACMEQ,
small schools with less than 15 or 20 students, schools for students with special needs
and, in some cases, "inaccessible" schools were removed from the initial target
population. While in SACMEQ countries, these exclusions never went beyond 5 per
cent, their exclusion may still have an impact on the estimated role of certain variables
such as class size, teachers' absence and so forth.
For further details on sample design procedures for SACMEQ, see SACMEQ
9
(2004: section F). For PASEC, a similar brochure is in process and should be available
in 2007.
Interpretation of regression results
Finally, without being related to different sampling procedures, one more
difference between our data for SACMEQ and PASEC should be kept in mind when
interpreting regression results: Overall sample size is quite different for the two country
groups. In SACMEQ, 14 countries are covered while only 8 countries are covered by
PASEC (other country data are available since recently, but could not yet be integrated
here). In terms of observations for individual students, this leads to a total sample size
for SACMEQ which is more than twice as high as in PASEC. Obviously, this influences
the precision of coefficient estimates in our regressions.
Econometric evidence for francophone and anglophone Africa
(PASEC and SACMEQ)
The appendix includes two detailed tables with regression results for literacy and
mathematics respectively (Tables 1 and 2). The following discussion concentrates on the
most relevant results. Most results are in line with the findings for developing countries in
general.
Traditional policy options
Textbooks, wall charts, other equipment
We find significant effects of textbook possession for maths in all grades and for
French in second grade in the PASEC sample. The magnitude of the estimated effects
ranges from 1 per cent of a standard deviation to 15 per cent. The higher coefficient for
textbook in French in second grade can mean that it is more important to personally
10
possess a book in lower grades. One could imagine that in lower grades, being able to
take the book back home for first reading practice is more relevant than in higher grades.
However, high coefficient estimates and significant results for individual textbook
possession may also be an artifact of the lack of two relevant control variables -parents'
literacy and books at home- which were not included in the questionnaire for second
grade students. As the expected correlation between these variables and textbook
availability is positive, second grade coefficients for textbooks are likely to be biased
upwards. Moreover, the generally lower level of initial textbook availability in earlier
grades may lead to higher coefficients if there are diminishing returns to overall textbook
coverage (for a more general discussion of such nonlinearities see Frölich and
Michaelowa, 2005). Thus the distinction between grade levels is more complex here
than it might seem at first glance.
Textbooks are also considered for spill-over effects, that is the effect of the share
of pupils with a textbook in the class. The coefficient for the variable which indicates with
how many others a student has to share a textbook in the SACMEQ sample is significant
and positive at the 5 per cent level. The insignificant coefficients for share of classmates
with textbook for the PASEC sample are rather surprising taking into account the strong
peer-effects estimated in an earlier non-parametric study on a smaller sample from
Burkina Faso, Cameroon, Senegal, Ivory Coast and Madagascar (Frölich and
Michaelowa, 2005). Interaction of class share and personal possession might be a
reason for these results as both variables are included, although the specification
chosen seems to be a good approximation of the functional form found in the mentioned
study. Nevertheless the importance of availability of reading books is clearly shown.
A question about wall charts was asked only in SACMEQ countries. The
coefficient estimate is positive, as expected, but remains insignificant. Teacher manuals
11
are significant in some regressions (only for SACMEQ) and then lead to a positive effect
of up to about 7 per cent of a standard deviation in test scores. For PASEC, they
become significant in different regression specifications with a lower number of general
equipment variables and for a different set of countries (not shown here). But results are
clearly less robust than for textbooks.
All in all, these results appear to be consistent with earlier studies for developing
countries, which show a somewhat positive, but moderate impact of learning materials,
especially textbooks (for a review, see for example Mingat, 2003). We spent much time
trying to find appropriate indicators using different combinations of classroom
furnishings, school facilities and basic equipment, such as chalk and blackboards. The
final specification presented in Tables 1and 2 includes a variety of separate indicators
for individual items and facilities, a joint indicator for higher technology equipment, such
as computers, television and video projectors, an indicator for the availability of electricity
and an indicator for the general condition of the school building.
At first glance, looking at SACMEQ regressions, our results seem to present a
strong evidence for the relevance of expensive electric equipment. The indicator for
higher technical equipment is strongly significant and indicates that adding any high tech
item to the existing equipment of a school raises student achievement by more than 12
per cent of a standard deviation. However, this variable must be considered with caution,
as it may well suffer from an endogeneity problem: As high tech equipment is an easily
visible signal of a rich school environment, wealthy parents and parents with particularly
talented children may select these schools in the first place. As most parents can be
expected to make their school choice only once (that is at the beginning of primary
education), controlling for the initial score at the beginning of the year, as possible with
PASEC data, will eliminate at least part of this selection effect. Unfortunately, the high
12
tech indicator is not available in PASEC, but electricity, a strongly correlated variable, is.
In PASEC, the effect of electricity is close to significant at the 10 per cent level at second
and fifth grade in French, but only as long as the pre-test score is not included into the
regression. Controlling for the pre-test scores leads to a jump of all p-values from below
0.2 to over 0.9.
In SACMEQ, the availability of a school or classroom library also appears to be
significant, whereby the existence of the library in the classroom itself seems to be more
directly beneficial. Not surprisingly, results for reading are higher than for maths and
make up almost 10 per cent of a standard deviation in literacy scores. The library result
is also reflected in two of the PASEC regressions (grade 2). One might take this as yet
another indication of the relevance of books in the learning process. Note that libraries
also offer a compensation for a scarcity of reading material at home. The variable "books
at home", which is introduced as one of the control variables for students' family
background, is strongly significant in all literacy and most mathematics regressions. This
reinforces the potential relevance of libraries in general, be it at classroom or school
level, in the village or town, or in the more flexible form of a "rolling library", which
appears to be most cost effective in scarcely populated rural areas.
Nevertheless, it should be noted that coefficients for school libraries shrink
considerably and become insignificant when the pre-test scores are controlled for. This
suggests that, just as in the case of technical equipment, a self-selection process of
good performers into well-equipped schools may bias the results.
A similar argument applies to the interpretation of the coefficient for the condition
of school buildings. The condition of school buildings - only included in the SACMEQ
analysis - reveals a strong and statistically significant positive effect: A change from
extremely bad to extremely good conditions leads to an increase of about 10 per cent of
13
a standard deviation of student achievement. However, just as technical equipment and
school libraries, the condition of the school building is one of the easily observable
characteristics parents may base their school choice on. As the variable is not included
in the PASEC analysis, it could not be tested whether the coefficient estimates remain
significant when initial knowledge is controlled for. When a related variables providing
information about the material the classroom is built with are included in individual
PASEC country studies, results generally do not show any relevant positive role of
concrete relative to other materials (see for example PASEC, 2005).
Otherwise, very few significant effects can be reported. A certain positive effect of
the availability of blackboard and chalk can be observed for mathematics, but not in all
regressions and only weakly significant in the case of PASEC. In French, the estimates
are insignificant and/or even show a negative sign. Toilets, health equipment and fresh
water do not show a significant positive effect, either. All in all, evidence for relevant
effects of school equipment is rather weak, especially when considering potential
selection bias and the more reliable estimates controlling for pre-test scores.
Class size, teacher qualification, knowledge and in-service training
Results with respect to class size show the typical insignificant or very small
impact on student achievement (for a literature review and discussion of studies on class
size see for example Hanushek, 1998). In order to take into account possible threshold
effects or other non-linearities, the variable is entered into the regression in a quadratic
form. In the case of SACMEQ, where the coefficients are significant, the analysis
indicates that negative effects start to become evident beyond a class size of 60
students. This result corresponds exactly to earlier results for PASEC in a regression
specification for five countries (Michaelowa, 2001). In the regressions specified here,
class size is insignificant for the PASEC countries. Another study based on PASEC
14
panel data for students in Senegal, controlling for student fixed effects, does not find any
negatively significant effect either (Fehrler, 2005).
Teacher qualification is a different issue. For PASEC, neither the indicator of
teachers' educational attainment (academic qualification), nor the indicator for
professional training is significant at the 5 per cent level. Only for fifth grade maths
teachers can some positive effect on educational attainment be discerned, which is
significant at the 10 per cent level in one regression, and close to significant in others. In
SACMEQ, however, the academic qualification is clearly significant and the professional
qualification is significant in all but one regression. Coefficients for academic qualification
indicate that the students gain between 2 and 4 per cent of a standard deviation in
scores when the teacher has attained a one step higher level of education, for example
lower secondary attainment instead of primary attainment only, or some tertiary instead
of simply upper secondary.
It is interesting to note the differences between SACMEQ and PASEC countries
here. Although the indicator used is almost identical in both surveys, in PASEC, it is
much more difficult to find the expected positive results. The problem appears to be that
the indicators of both professional training and educational attainment only capture
duration while no information is available on quality. Obviously, depending on quality and
practical relevance, two different courses of the same duration may have a totally
different impact on actual teaching skills. It can be shown that in PASEC, there is no
significant positive correlation between the duration of teachers' educational attainment
and teachers' knowledge of the subject matter. This implies that the low coefficient
estimates for attainment should not be interpreted as an indication of a low impact of
increased subject matter knowledge, but rather as an indication of the low quality of the
15
education the teachers themselves received when they attended school (Michaelowa,
2003).
To measure actual teacher knowledge, PASEC uses an exercise for teachers in
which they have to count the mistakes in a fictitious student dictation. In SACMEQ, a
different and exceptionally precise indicator of relevant teacher knowledge is available:
Teachers were themselves asked to take the students' tests and marked on the same
scale. The average teacher score in literacy is more than two standard deviations above
average student scores and is reached only by about 2 per cent of the students.
As opposed to PASEC, it can be shown that for SACMEQ countries the
correlation between educational attainment and teacher test scores is significant, albeit
even here, less pronounced than one might have expected. Estimated correlation
coefficients are 0.21 for literacy, and 0.32 for maths. Since we can find a significant
correlation only for SACMEQ countries, this may indicate that, on average, the quality of
secondary and tertiary educational institutions attended by (future) teachers is better in
anglophone than in francophone Africa, at least in the core subjects of literacy and
mathematics. This could explain the differing results on the relevance of the academic
qualifications. One should be cautious, however, when interpreting these results,
because the indicator of teachers' subject matter knowledge in PASEC is much less
reliable than the one used in SACMEQ. Moreover, neither in PASEC, nor in SACMEQ
are the indicators for teachers' subject matter knowledge available for all countries. This
is also the reason why these indicators have not been included directly in our
regressions in Tables 1 and 2.
In any case, it should be noted that the coefficient estimates of 2-4 per cent of a
standard deviation for a full level of education (like the whole upper secondary cycle) are
not very high when compared to the cost incurred for this additional education, including
16
the opportunity cost of having the teachers start effectively teaching much later, and the
higher pay they can expect with a higher academic qualification. While the linear
specification of educational duration used here does not indicate any optimal cut off
point, some prior research on PASEC indicates that this may be below the A-levels or
baccalauréat (successful upper secondary completion) (Bernard, Tiyab and Vianou,
2004).
It has been shown that teachers holding a baccalauréat are often less motivated
than their peers with lower educational attainment, possibly because their higher
expectations with regard to their future jobs are not met by the reality of their situation
(Michaelowa, 2007). Moreover, even if there was a linear increase in the impact of
educational attainment on student achievement, costs in terms of salaries would
increase over-proportionately, with a strong jump related to the completion of the upper
secondary final examination. We can thus state, that raising entry requirements for the
teaching profession to include the successful completion of upper secondary education
or beyond should not become a policy priority.
As mentioned above, the differences in the significance (or lack of significance)
of SACMEQ and PASEC can be observed not only for teachers' academic qualification,
but also - in a similar way - for their professional training. In this context, there is no way
to directly show from the data that this may be related to a different quality of the courses
offered. The correlation between teachers' professional training and subject matter
knowledge is not very strong, even in SACMEQ countries, but this is plausible even for
very good training modules since professional training could focus on pedagogical rather
than academic skills. Most probably, the reason for difficulties in finding significant
results in overall PASEC regressions is that professional qualifications vary widely
17
across countries (even within the francophone education systems) and are more or less
effective, so that it is very difficult to capture their overall effect.
Individual country estimates for PASEC have often shown the relevance of
professional training for student achievement (see, in particular, PASEC, 2004). In their
individual country regressions for SACMEQ, Lee, Zuzu and Ross (2005) construct a joint
estimate for academic and professional qualification, so that results are not directly
comparable. Nevertheless, they also find that the effect varies widely between countries.
A positively significant impact is only found for about one third of the countries covered
(and insignificant effects otherwise). In this context, it may be argued that duration (the
only available measure for professional training) is less relevant than content
(Michaelowa, 2003; Bourdon, Frölich and Michaelowa, 2006). If the latter could be
adequately measured, we would probably face much less variation of results between
individual countries and between country groups.
Similar reasoning applies to in-service training (see for example Nguyen, Wu and
Gillis, 2005: 40). The latter is negatively significant in SACMEQ. This is a counter-
intuitive result also found for individual country cases in francophone Africa, and often
related to training sessions during class hours which then reduce effective teaching time
(Bernard and Michaelowa, 2005). It should also be noted, however, that in SACMEQ, the
in-service training variable is based only on teachers' own subjective assessment of the
efficacy of these courses. In PASEC regressions, the variable reflects the number of
courses attended per year, and teacher absence can be directly controlled for (in
SACMEQ, only an indirect school level variable is available). In this setting, in-service
training has a positive coefficient, which is significant for fifth grade French and implies
an improvement of up to 5 per cent of a standard deviation in students' scores for each
additional training seminar the teacher has attended per year (during the last five years).
18
Student flow organization, repetition, effective teaching time
Coming to the organization of student flows, our analysis confirms the negative
effect of double shift teaching known from other studies (for example Michaelowa, 2001).
As the control for pre-test scores generally reduces the overall effect (and makes it
statistically insignificant in some regressions), parts of the effect seem to be related to a
selection of bad performers in double-shift classes. However, after controlling for initial
knowledge, the negative coefficients remain and still indicate losses of often more than
10 per cent of a standard deviation in student test scores for double-shift classes. As
opposed to earlier analysis, we do not find any evidence that this effect is weaker in
second grade. In fact, second grade mathematics shows the most significant negative
results among all PASEC regressions.
SACMEQ regressions for sixth grade only indicate losses of up to 6 per cent of a
standard deviation in the case of double-shift organization, and the results are significant
only in one regression (even at the 10 per cent level). However, if we look again at the
individual country regressions carried out by Lee et al. (2005), we find that in some
countries, this variable does not seem to be relevant in current education practice. In
fact, the authors include it only in 9 out of 14 regressions, 4 of which show the expected
significant negative effect, sometimes with extremely high coefficients corresponding to
up to about 30 per cent of a standard deviation of (international) student scores (Kenya
and Zambia).
As opposed to double-shifts, no significant effect in either direction can be
discerned for multi-grade teaching. Unfortunately, this variable does not exist in the
19
SACMEQ database. The reason might be that in SACMEQ, very small schools for which
this system is generally most relevant have been excluded from the target population.
Grade repetition, introduced to help students to catch up with their peers, can be
shown to be ineffective. The coefficient in all regressions is clearly negative and
significant, except for the initial year, in which the repeating student appears to benefit
from a short-term advance on his new classmates, at least in grade 5. As indicated by a
comparison of regressions with and without control for the pre-test score, this short-term
advance is primarily due to the students' higher initial knowledge. In the SACMEQ
regressions and most of the PASEC regressions for second grade, even the coefficient
for current class repetition on test scores is negative. In many cases, repeaters can be
shown to do much worse than their classmates, with an overall performance reduced by
15-20 per cent. Results are robustly negative and highly significant, even in individual
country regressions. Coefficient estimates for some countries correspond to student
achievement reduced by more than half of a standard deviation of student scores (for
Botswana and Mauritius, see Lee et al., 2005).
Caution is required, however, when interpreting these results. These figures
cannot be interpreted as a causal relationship, in particular in models without control for
initial ability,. A bad student repeating his grade is not necessarily doing badly because
he repeats, but rather repeats because he has been doing badly. Nevertheless, as
strongly negative results can also be established in PASEC regressions in which initial
knowledge is controlled for and, even more convincing in panel regressions on students
from Burkina Faso, Côte d'Ivoire and Senegal (Bernard, Simon and Vianou, 2005)
followed through several years, the results cannot be considered as a mere issue of
reverse causation.
20
This confirms the conclusions of a detailed PASEC analysis on the specific issue of
repetition based on information for students followed through several years of their
primary education in Burkina Faso, Côte d'Ivoire and Senegal (Bernard et al., 2005).
Finally, considering the issue of effective teaching time, various indicators would
have to be taken into account simultaneously, and only some of them are available in
the data. SACMEQ data reveal a strongly negative impact of student absence from
class. The SACMEQ indicator for teachers being late shows a strongly negative effect as
well, while the effect of teachers' absence is more difficult to capture. The latter is
measured either as indicated by teachers themselves (PASEC), or in a more general but
also more reliable way, as indicated by school principals (SACMEQ). While overall,
coefficients show the expected negative sign, for PASEC second grade we get some
counter-intuitive results, which may be related to teachers not reporting their absences
truthfully.
The effective use of time in the classroom (for teaching the subject matter rather
than, for example, try to establish discipline) was covered by the PASEC teacher
questionnaires as well, but is based on an extremely subjective appreciation, and
therefore difficult to explore in quantitative analysis. Nevertheless, SACMEQ regressions
show that the cumulative negative effect of those variables for which reliable evidence is
available is already considerable.
Institutions and incentive structures: What about non-physical inputs
in educational production?
While the traditional discussion of school inputs focuses on physical goods such
as teachers, books, buildings, desks and benches, the "second generation" educational
production function literature focuses on more subtle inputs such as accountability, effort
and motivation. The idea is that much of the unexplained variation in student
21
achievement may be brought about by differences in these inputs that have previously
been largely neglected by the economic literature. Obviously, their relevance has been
widely discussed by educational scientists, sociologists and psychologists, but only in
recent years these discussions started to influence the input effectiveness literature.
To a certain extent, these institutional features can also be analysed empirically
along with the physical inputs of the education production function. It should be noted,
however, that it is often difficult to find appropriate indicators, that many of these
indicators do not belong to the standard set of variables covered by student surveys, and
that the concrete forms of implementation vary so much between countries that very
detailed information is required to make valid comparisons.
Private-sector participation
Looking at our data for sub-Saharan Africa, only very crude information is
available on whether schools are private or public. This information is derived from a
simple yes-or-no question to principals. It is available for all SACMEQ countries, so that
it could be included in the regressions presented in Tables 1and 2. The regression
tables show a relatively high coefficient for literacy, indicating that students in private
schools outscore students in public schools by 7-10 per cent of a standard deviation in
achievement scores. However, this relationship is significant only in one of the two
regressions and not significant at all for mathematics. Moreover, if we try to further
reduce sample selection bias by considering only (remote) rural areas where self-
selection into specific schools is impossible, no impact can be found either (regressions
not shown).
In PASEC, information on private-sector participation is available only for Togo.
Both for fifth and for second grade we find similar results (not shown): Students in private
schools show higher overall performance, but this performance advantage vanishes
22
when socio-economic background and initial knowledge as measured in the pre-test
scores is adequately controlled for. Moreover, even without any controls, in rural areas
where the possibility of school choice is considerably reduced, private schools do not
show any advantage.
In contrast, in a more elaborate study, Lassibile and Tan (2003) complement the
PASEC data base for Madagascar with external information on school types. The
authors control for self-selection using the Heckman two-step procedure and do find
some positive impact of private schools.
Decentralization of responsibilities
In SACMEQ and PASEC, community or parental involvement is measured
primarily in terms of contributions to school equipment (SACMEQ), or the director's
appreciation of how easily they could be mobilised for such purposes (PASEC). These
variables are positively significant in all SACMEQ regressions as well as in PASEC fifth
grade mathematics regressions, as long as the initial knowledge of students is not
controlled for. However, as the variable's significance cannot be shown in any of the
regressions including the pre-test score, our results might again simply reflect sample
selection bias. Students with better scores tend to have parents who can be more easily
mobilized for school issues and who also have the resources required.
SACMEQ further includes a variable indicating whether parents and / or the
community are involved in the payment of exam fees, additional teacher salaries or
bonuses. This variable is fully insignificant, probably because the realities reflected in a
yes or no to this question can vary considerably - from a veritable influence on teacher
pay thereby creating accountability, to obligatory payments for certain services.
23
Finally, in earlier PASEC regressions on five countries, the existence of an active
parent-teacher organization was considered as an additional variable for parental
involvement. Results were similarly weak and insignificant (Michaelowa, 2000: 31). As
there is no other, more convincing indicator to analyse the issues of decentralization and
school or teacher autonomy, our data do not allow us to draw any firm conclusions with
respect to this topic.
Teacher contracts
In the regressions presented in Tables 1and 2, teachers' employment on a
contractual (non-civil servant) basis has been simply introduced as a dummy variable.
For SACMEQ countries this information was not available. The pooled effect for the eight
PASEC countries is not conclusive: In second grade, it appears to be negative, but
significant only at the margin and only in one regression; and in fifth grade, it seems to
be positive, but significant only in those regressions in which the pre-test score is not
controlled for. The magnitude of the effects is similar, with opposite signs.
School inspection
In our data for eight PASEC countries, the positive effect of inspections is found
significant for fifth grade mathematics, and close to significant (or, in one case,
significant at the 10 per cent level) for fifth grade French. The magnitude of the effect is
not huge but non-negligible, as it corresponds to up to about 10 per cent of a standard
deviation in student scores. Just as in Bernard (1999), this result cannot be replicated for
second grade.
For SACMEQ countries, the inspection variable is insignificant. To a certain
extent, this may be explained by the fact that, asking for the frequency of visits within the
previous calendar year, the measure is not necessarily related to the academic year
24
concerned. Other teachers may have been concerned, so that the indicator for effective
teacher control becomes less precise. In Malawi and Mauritius where evaluations took
place later than in other SACMEQ countries, this problem might be particularly strong.
However, it should also be noted that the roles of inspections in francophone and
anglophone Africa are not the same, so that it could be worthwhile to examine in some
more detail the concrete incentives related to this control mechanism in the different
regions.
Overall, it appears that incentive based approaches may have a relevant impact
in sub-Saharan Africa. While the specific conditions under which they become most
effective still remain to be evaluated, certain ways of monitoring and control, and direct
incentives for teachers based on accountability towards parents and local communities
appear to be promising ways ahead. The political appeal of these measures is that they
may bring along considerable improvement in student learning without relevant direct
financial implications. In some cases, budgetary implications can even be positive.
Conclusions
The optimal policy package reconsidered
Investment in pedagogical resources, especially textbooks for the core subjects
of reading and maths, can still be considered as an efficient policy measure. If budget
constraints are very strong, one book may be provided only to every second student,
especially in higher grades where taking the book back home does not seem to be as
important as for very young students.
Another priority should be the reduction of repetition rates. Repetition induces
high cost because the system has to cope with an increased overall number of students.
Moreover, repetition increases early drop-out. And finally, the effects of repetition on
25
student learning have consistently been shown to be negative, rather than positive, at
least in the long run.
With respect to teacher education and training, the focus should be on quality
rather than duration. In anglophone Africa, where the duration of formal education and
teachers' subject matter knowledge are much more clearly correlated than in
francophone Africa, longer education for teachers significantly enhances student
learning. However, the effect is only moderate in size and has to be carefully weighed
against the equally high cost generally involved with salaries for teachers with higher
educational attainment. Similar considerations are in order with respect to pre service
and in service training. From a cost-benefit perspective, short but well designed and
practice oriented programs appear to be most promising.
Finally, it appears highly relevant to ensure the maximum use of formal
instruction time for effective teaching. Double shift teaching seems to have a detrimental
impact in this respect. As there is ample evidence for a rather modest negative impact of
high student teacher ratios, double shift teaching should generally be avoided.
Effective teaching time can also be increased by improving students' attendance.
Apart from the well-known requirement of adjusting the academic year to harvesting
seasons, attendance can be increase by simple health care measures. In this context,
de-worming has been shown to be particularly cost effective (see for example Kremer
and Miguel, 2001).
And last but not least, effective teaching time can be increased by reducing
teachers' absences. In some cases, simple administrative measures like the
reorganization of teacher remuneration (so that teachers do not need to collect their pay
from a far away district officer) may be very effective. In general, however, more effective
control mechanisms seem to be required.
26
This creates the link to the relevance of functioning incentive systems. Notably, in
several countries, teachers on non civil servant fixed term contracts have been shown to
miss their classes significantly less often then their colleagues (Michaelowa, 2007).
While contract teacher programs combine various features with partly contradicting
consequences for student achievement, preliminary evidence suggests that the incentive
effect works best if these teachers are employed by parents and local communities,
rather than by public authorities. Indeed, this should enhance teachers' accountability
and parents' incentive for effective monitoring. Theoretically, this system could be
generalized by channelling public funds for teacher remuneration via local communities
and parents' associations.
Other aspects of decentralization and increased local autonomy (both for
parents, and for schools and teachers) may also be beneficial for student learning. In
particular, any kind of measures to enhance transparency about resource flows and
learning outcomes appears to be valuable. This could also be a first step towards even
more comprehensive institutional change.
Appendix
Table 1 displays the results for literacy and table 2 presents the results for
mathematics. Each of the two tables includes ten regressions, two for SACMEQ (sixth
grade only, model A and B), four for PASEC fifth grade (model A and B, with and without
pre-test) and four for PASEC second grade.
27
Table 1: Literacy
28
Table 1 continued
29
Table 2: Mathematics
30
Table 2 continued
31
References
Bernard, J.-M. (1999) Les enseignants du primaire dans cinq pays du
programme d'analyse des systèmes Éducatifs de la confemen: Le role du
maître dans le processus d'acquisition des élèleves, report for the ADEA
working group on the teaching profession, french section, Paris: ADEA.
Bernard, J.-M. and Michaelowa, K. (2005) Managing the impact of
pasec projects in francophone sub-saharan africa, in K. Ross and
I. Jürgen-Genevois (eds), Cross-National Studies of the quality of Edu-
cation: Planning their design and managing their impact, Paris:
IIEPUNESCO, forthcoming document.
Bernard, J.-M., Simon, O. and Vianou, K. (2005) Le redoublement: mirage
de l'école africaine?, Dakar: PASEC/CONFEMEN.
Bernard, J.-M., Tiyab, B. K. and Vianou, K. (2004) Profils enseignants et
qualité de l'éducation primaire en afrique subsaharienne francophone:
Bilan et perspectives de dix années de recherche du pasec, Dakar:
PASEC/CONFEMEN.
Bourdon, J., Frölich, M. and Michaelowa, K. (2006) Broadening access to
primary education: contract teacher programs and their impact on
education outcomes in africa-an econometric evaluation for the republic
of niger, in L. Menkhoff (ed.), Pro-Poor Growth: Issues, Policies, and
Evidence, Berlin: Duncker & Humblot. Forthcoming document.
32
Brown, G. and Micklewright, J. (2004) Using international surveys of achievement
itute for
Chinap
Fehrle
boldt Universität zu Berlin
/fehrl
ry
raise achievement in the third world?,
Fuller,
Gleww utcomes
in developing countries, Handbook on the Economics of Education
(forthcoming, second draft as of April 2005), Elsevier.
and literacy: A view from the outside, Montreal: UNESCO Inst
Statistics.
ah, V. (1997) Monitoring learning achievement, UNESCO-UNICEF
follow up to Jomptien.
CONFEMEN (2005) Accueil pasec, downloaded on 15/11/05 at
http://www.confemen.org/rubrique.php3?idrubrique = 3.
r, S. (2005) Erklärung von schulischem (Miss-) Erfolg im Senegal
mit Hilfe von fixed effects Modellen, Bachelor thesis, Center for
Applied Statistics and Economics, Hum
http://edoc.hu−berlin.de/master/fehrler−sebastian−2005−09−01/PDF
er.pdf.
Frölich, M. and Michaelowa, K. (2005) Peer effects and textbooks in prima
education: evidence from francophone sub-saharan africa, HWWA Dis-
cussion paper 311.
Fuller, B. (1987) What school factors
Review of Educational Research 57 (3), 255-292.
B. and Clarke, P. (1994) Raising school effects while ignoring culture?
local conditions and the influence of classroom tools, rules, and
pedagogy, Review of Educational Research 64 (1), 119-157.
e, P. and Kremer, M. (2006) Schools, teachers, and education o
33
Glewwe, P., Kremer, M., Moulin, S. and Zitzewitz, E. (2004) Retrospective
vs. prospective analysis of flip charts in kenya, Journal of D velopmen
Economics 74 (1
e t
), 251-268.
ard
er 98 (1),
conomic
Kreme ties
, J. (2003) Student learning in public and private
Lee, V., Zuzu, T. and Ross, K. (2005) School effectiveness in 14 sub-Saharan
ies
t
ment
iciency
Goldstein, H. (2003) Multilevel Statistical models, 3rd edition, London: Edw
Arnold.
Hanushek, E. (1998) The evidence on class size, Occasional Pap
University of Rochester, Institute of Political Economy.
Hanushek, E. (2003) The failure of input-based schooling policies, The E
Journal 113(485), F64-F98.
r, M. and Miguel, E. (2001) Worms: Education and health externali
in Kenya, Poverty action lab paper 6, 1-64.
Lassibile, G. and Tan
primary schools in Madagascar, Economic Development and cultural
Change 51 (3), 699-717.
African countries: Links with sixth graders' reading achievement, Stud
in Educational Evaluation 31 (2-3), 207-246.
Michaelowa, K. (2000) Dépenses d'éducation, qualité de l'éducation et pauvre
é: l'exemple de cinq pays d'Afrique francophone, OECD Develop
Centere Technical paper 157.
Michaelowa, K. (2001) Primary education quality in francophone sub-Saharan
Africa: Determinants of learning achievement and eff
considerations, World Development 10, 1699-1716.
34
Michaelowa, K. (2003) Determinants of primary education quality: What can
pared for the ADEA Biennial Meeting
Michaelowa, K. & Wittmann, E. (2007) Teacher job satisfaction, student
from
eveloping Areas,
tual elements for a quality policy for
A biennial meeting 2003,
Murim ing
q), Prospects 35 (1), 91-108.
ternational invitational education policy
PASEC
we learn from PASEC for francophone Africa in the context of education
for all, Background Paper pre
2003, Paris: ADEA.
achievement and the cost of primary education. Evidence
francophone sub-Saharan Africa, Journal of D
forthcoming.
Mingat, A. (2003) Analytical and fac
primary education in sub-Saharan Africa in the context of education for
all, Background paper prepared for the ADE
Paris: ADEA.
ba, S. (2005a) The southern and eastern Africa consortium for monitor
educational quality (sacme
Murimba, S. (2005b) The southern and eastern Africa consortium for monitoring
educational quality (sacmeq):mission, approach and prospects,
Prospects 35 (1), 75-89.
Nguyen, K., Wu, M. and Gillis, S. (2005) Factors influencing pupil achievement
in sacmeq2-botswana: An application of structural equation mod-
elling, paper presented at the In
research conference, Paris: IIEP-UNESCO.
(1999) Les facteurs de l'efficacité dans l'enseignement primaire: Les
35
résultats du programme pasec sur neuf pays d'afrique et de l'océan indien,
fs
PASEC t et formation des enseignants du premier degré
PASEC
ocument.
ions
Ross,
ions based on a survey of schools (various country reports),
SACM ts, CD-ROM
StataC
UNESCO (2004) Education for all (EFA) global monitoring report 2005,
Rapport de synthese du Programme d'Analyse des Systemes Educati
de la CONFEMEN (PASEC), Dakar: CONFEMEN.
(2004) Recrutemen
au togo: Quelles priorités?, Dakar: CONFEMEN.
(2005) Impact du statut enseignant sur les acquisitions dans le premier
cycle l'enseignement fondamental public au mali, Dakar: CONFEMEN.
forthcoming d
Raudenbush, S. and Bryk, A. (2002) Hierarchical linear models: Applicat
and Analysis Methods, second edition, Sage: Thousand Oaks.
K. (1998) SACMEQ Policy Research, The Quality of Education: Some
policy suggest
IIEP/UNESCO.
EQ (2004) Data archive for sacmeq1 and sacmeq2 projec
(IIEP-UNESCO).
orp. (2003) Survey Data, Statistical Software, release 8.
education for all, the quality imperative, Paris: UNESCO. Available:
http://portal.unesco.org.
36
