Technical ReportPDF Available

Mexico Monitoring and Evaluation System: Scaling up from the Sectoral to the National Level

Authors:

Figures

Content may be subject to copyright.
ECD Working Paper Series ● No. 20
Mexico’s M&E System:
Scaling Up from the Sectoral to the National Level
Manuel Fernando Castro
Gladys Lopez-Acevedo
Gita Beker Busjeet
Ximena Fernandez Ordonez
www.worldbank.org/ieg/ecd
September 2009
The World Bank
Washington, DC
ii
© 2009
Independent Evaluation Group
Communication, Learning, and Strategy (IEGCS)
Email: ieg@worldbank.org
Telephone: 202-458-4497
Facsimile: 202-522-3125
Building monitoring and evaluation systems helps strengthen governance in countries—
by improving transparency, by strengthening accountability relationships, and by
building a performance culture within governments to support better policymaking,
budget decision-making, and management. A related area of focus is civil society, which
can play a catalytic role through provision of assessments of government performance.
IEG aims to identify and help develop good-practice approaches in countries, and to
share the growing body of experience with such work.
The IEG Working Paper series disseminates the findings of work in progress to
encourage the exchange of ideas about enhancing development effectiveness through
evaluation. An objective of the series is to get the findings out quickly, even if the
presentations are somewhat informal.
The findings, interpretations, opinions, and conclusions expressed in this paper are
entirely those of the authors. They do not necessarily represent the views of the
Independent Evaluation Group or any other unit of the World Bank, its Executive
Directors, or the countries they represent.
ISBN-10: 1-60244-121-9
ISBN-13: 978-1-60244-121-7
iii
CONTENTS
Abbreviations ........................................................................................................................... iv
Foreword ....................................................................................................................................v
Executive Summary ................................................................................................................. vi
Introduction ..............................................................................................................................1
1. Mexico’s Move to a Results Orientation ....................................................................2
1.1 Disparate Efforts for M&E from the 1970s to Early 1990s ..................................2
1.2 Establishing a Systematic Role for Evaluation: 1997–2003 .................................2
1.3 Progress toward a Governmentwide M&E System since 2004 ............................3
2. CONEVAL and the Institutionalization of Evaluation ............................................5
2.1 Objectives .............................................................................................................5
2.2 Organizational Structure, Roles, and Responsibilities of Key Stakeholders ........5
2.3 Main M&E Tools and Methodologies ..................................................................7
2.3.1 General Guidelines for the Evaluation of Federal Programs ....................8
2.3.2 Matrix Indicators.......................................................................................9
2.3.3 Program Evaluation Methodologies .......................................................10
2.3.4 Annual Evaluation Program and Sector Evaluation Units......................12
2.3.5 Poverty Reduction Evaluation Methods .................................................13
2.4 Utilization of Evaluative Information Produced by CONEVAL .........................14
3. The Performance Evaluation System .......................................................................17
4. Mexico’s M&E System: Strengths and Main Challenges ......................................19
4.1 Managing Change ...............................................................................................19
4.2 Developing Evaluation Capacity ........................................................................20
4.3 Institutional Coordination ...................................................................................21
4.4 Strengthening the Role of Planning ....................................................................22
4.5 Strengthening the Role of Monitoring ................................................................23
4.6 Strengthening the Quality of Performance Information .....................................23
5. Conclusions and Lessons Learned ............................................................................25
Bibliography ...........................................................................................................................28
Annex 1: Chronology of Key Initiatives and Events ..........................................................30
Annex 2: Annual Evaluation Program ................................................................................31
iv
ABBREVIATIONS
ASF Federal Superior Auditing Office
(Auditoria Superior de la Federación)
CEPAL Economic Commission for Latin America
CONEVAL National Council for the Evaluation of Social Policy
CTMP Technical Committee for Poverty Measurement
(Comité Técnico para la Medición de la Pobreza)
DF Federal District
(Distrito Federal)
DOF Official Diary of the Federation
IEG Independent Evaluation Group
ILPES Latin America Institute for Economic and Social Planning
M&E Monitoring and evaluation
MI Matrix Indicator
MML Logic Framework Matrix
NDP National Development Plan
OECD Organization for Economic Cooperation and Development
OMB Office of Management and Budget
PAE Annual Evaluation Program
PART Program Assessment Rating Tool
PIDER Public Investment Program for Rural Development
PMG Management Improvement Program
(Programa de Mejoramiento de la Gestión)
PRI Institutional Revolutionary Party
(Partido Revolucionario Institucional)
PROMAP Program for the Modernization of the Public Administration
ROP Rules of operation
SAGARPA Secretariat of Agriculture, Livestock, Rural Development, Fish and
Food
SECODAM Secretariat of Control and Administrative Development
(Secretaria de la Contraloría y Desarrollo Administrativo)
SED Performance Evaluation System
(Sistema de Evaluación del Desempeño)
SEDESOL Secretariat of Social Development
SEMARNAT Secretariat of the Environment and Natural Resources
SEP Secretariat of Education
SFP Secretariat of Public Administration
SHCP Secretariat of Finance and Public Credit
TOR Terms of reference
v
FOREWORD
World Bank support to strengthen M&E systems in Latin America and the Caribbean
Region has grown substantially in the past decade. There is intense activity on M&E
issues in more than 20 countries in the region, and Mexico is one of the leading countries.
This is a joint paper produced by IEG and the Latin America Poverty Reduction Unit
(LCSPP) in PREM at the World Bank. The preparation of this paper is based on the
technical assistance coordinated by LCSPP and provided by the World Bank to Mexico’s
National Council for the Evaluation of Social Policy (CONEVAL) in 2007 and 2008. The
paper takes stock of the government’s efforts to institutionalize evaluation practices,
particularly the process by which the results orientation was strengthened in the social
sector.
Mexico was a pioneer in Latin America in the use of sound methodologies to evaluate the
performance of social policy, especially since 1997 when the Progresa/Oportunidades
program was evaluated. Within the development community, this evaluation is regarded
as one of the most influential. At the same time, the social sector has been a strong
champion of using evaluations in Mexico’s public administration and has exerted
important influence on the M&E and performance-based management reforms at the
governmentwide level.
Evaluation is a rich and dynamic activity that enhances innovation and knowledge
generation in Mexico’s public administration—to such an extent that this country has
become a potential source for identifying lessons, best practices, success factors, and
obstacles in institutionalizing evaluation in government. Accordingly, it is hoped that
disseminating knowledge of Mexico’s experience in this area will benefit officials
undertaking similar tasks in other countries.
The lead authors of this paper are Manuel Fernando Castro, Gladys Lopez-Acevedo, Gita
Beker Busjeet, and Ximena Fernandez Ordonez. The World Bank team that completed
the technical assistance also included Alejandro Medina Giopp, Gilberto Moncada Vigo,
Jonna Lundwall, Rosa Maria Fernandez-Hernandez, and Lucy Bravo. The editorial
contributions of Helen Chin are gratefully acknowledged.
The evidentiary basis of this work was obtained from several Bank missions to Mexico,
discussions and interviews with government officials and external evaluators, review of
government and World Bank documents on CONEVAL, the Secretariat of Social
Development (SEDESOL), the Secretariat of Finance and Public Credit (SHCP), and the
Secretariat of Public Administration (SFP). The authors would like to acknowledge and
thank all the government officials who provided support, especially the CONEVAL team,
including Gonzalo Hernández, Executive Secretary; Edgar Martínez, General Adjunct
Director of Coordination; Gabriela Pérez, General Adjunct Director of Evaluation; and
Shakirah Cossens, Director of Monitoring of Social Programs.
The views expressed in this document are solely those of the lead authors, and do not
necessarily represent the views of the World Bank or of the government of Mexico.
vi
EXECUTIVE SUMMARY
The enthusiasm for performance management is palpable among stakeholders in the
Mexican public administration. This enthusiasm has been fueled by important legislation
as well as a number of extremely influential impact evaluations of key social programs in
Mexico. The national and international recognition, which the impact evaluations of
Progresa/Opportunidades have garnered, have played an important role in putting
evaluation on the map in Mexico. In terms of the legislative framework, this diagnostic
indicates that Mexico has succeeded in creating a solid foundation upon which the
ambitious agenda of M&E initiatives can be built. Key legislative initiatives, including
the Fiscal Responsibility Law, Management Improvement Program legislation, and the
Social Development law have provided frameworks to strengthen preexisting evaluation
laws. This is extremely important for the sustainability of the system in the longer term
and is a clear strength for Mexico as it moves forward.
This paper provides a brief review of Mexico’s progressive movement from a sectoral to
a governmentwide M&E system. It highlights the critical institutional reforms introduced,
the policy decisions, and the most important operational steps that were taken, offering an
account of the political context within which such changes and decisions were possible.
The paper emphasizes the specific role of the National Council for the Evaluation of
Social Policy (CONEVAL) as an innovative development, which furthered
institutionalization of evaluation and a results focus at the federal government level
through the implementation of the System for Performance Evaluation (SED). A quick
review of the governmentwide system’s strengths and main challenges for the future is
offered. Some lessons for other countries and conclusions follow.
A conclusion of this paper is that after the 2005 reforms, Mexico entered into an
advanced stage of performance-based management that is grounded on strong
institutional foundations: legislation has provided a modern budget framework, sound
M&E tools and arrangements have been developed, and accountability and transparency
conditions have been created to make most of the changes operational. However, the
authors also conclude that full institutionalization is not yet complete. Sustainability of
the system and the reforms, as much as consolidation of the implementation process, will
depend on how the government addresses a number of related challenges ahead
including, among others, assimilation of the reforms, greater utilization of performance
information, capacity building, increasing knowledge capacity at the subnational level,
and harmonization and rationalization of evaluation work among different responsible
agencies.
1
INTRODUCTION
Mexico’s attempts to establish the evaluation of federal programs, as a government practice,
dates back to the mid-1970s. However, it was only at the end of the 1990s that monitoring
and evaluation (M&E) and performance-based management reforms started to take root in
the federal public administration. One of the catalysts of the institutionalization process for
evaluation was the creation of the National Council for the Evaluation of Social Policy
(CONEVAL). The unique characteristics of CONEVAL—namely its independence,
technical capacities, and mandate—allowed it to advance in the construction of the social
sector M&E system, while also influencing the development of performance management in
the federal government more broadly.
With the creation of CONEVAL in 2005, there has been increasing enthusiasm for results-
based management in Mexico. Interest from social sector secretaries, for example, from the
Secretariat of Social Development (SEDESOL) and the Secretariat of Education1
to other
key cross-cutting agencies, including the President’s Office, Secretariat of Finance, and
Secretariat of Public Administration, has grown tremendously. As a result, Mexico has
introduced standardized M&E practices and tools at the governmentwide level, as well as
mandates to increase the focus on results, particularly in the budget process. Following a
constitutional amendment in 2007, subnational governments have followed the federal
administration in reforms toward performance budgeting.
Mexico’s path to a national M&E system cannot be understood without acknowledging the
political changes sweeping the country at the end of the 20th century. Seven decades of one-
party rule had steadily been replaced by a functioning democracy and a call for greater
accountability of government. Economic and social considerations also played a role. At the
beginning of this decade Mexico saw 20 percent of its population living in extreme poverty
and still suffering from the impacts of two large-scale financial crises. Questions were voiced
about the extent to which social spending was helping to alleviate this problem.
Mexico is one case of a country that was able to change from limited use of evaluation
concentrated in specific pockets of the administration—to greater use of an increasingly
sophisticated, whole-of-government evaluation system—with differentiated methodologies,
growing capacities, and progressive input into government decision making. The case
demonstrates a gradual construction of performance-based management institutions, taking
into account the international experience but also introducing a high degree of innovation and
country-specific development. For these reasons, the Mexican case holds valuable insights
for national and international M&E practitioners.
This document provides a brief overview of Mexico’s progressive transition from a sector-
based to a governmentwide M&E system. It highlights the critical institutional reforms
introduced, the policy decisions, and the most important operational steps taken, thus
offering an account of the political context within which such changes and decisions were
possible. Special attention is given to the role of CONEVAL in paving the way for these
reforms, particularly in the emergence of the Performance Evaluation System (SED) under
the leadership of the Ministry of Finance and Public Credit (SHCP). A quick review of the
system’s strengths and challenges is given, followed by the identification of lessons of
potential interest to other countries.
1 To a lesser extent, the Secretariat of Agriculture, Livestock, Rural Development, Fish, and Food (SAGARPA)
also had an important role in this process.
2
1. MEXICO’S MOVE TO A RESULTS ORIENTATION
Mexico’s path to a results-oriented system can be divided into three periods. The first period,
from the late 1970s to 1990s, was characterized by ad hoc evaluation initiatives in particular
sectors, often supported by international agencies. The second period, beginning in the late
1990s and facilitated by the opening-up of the national political system, saw the formalization
of accountability mechanisms through a flurry of comprehensive legal initiatives.
Formalization plus the implementation of a very successful and influential social program
impact evaluation helped to change the federal administration to a performance results
orientation and the identification of government entities responsible for conducting
evaluations. The progress made during these years allowed Mexico to move to a third,
ongoing stage—the institutionalization of evaluation in the social sector under the framework
of the CONEVAL and the movement to a governmentwide SED. This system is coordinated
by the SHCP and is strongly linked to the implementation of performance budgeting.
1.1 Disparate Efforts for M&E from the 1970s to Early 1990s
Pioneering program evaluations were being undertaken in Mexico as early as the 1970s,
often supported by international organizations such as the World Bank and the Inter-
American Development Bank. One example was the Public Investment Program for Rural
Development (PIDER)—one of the first rural poverty-reduction programs in Mexico.2
More systematic initiatives were introduced in the fiscal control area during the late 1980s
and early 1990s. Integral audits—performed by the then-new Federal Audit Institution
(Contraloria)—introduced performance measurement concepts. At the beginning of the
1990s there were attempts to create formal evaluation schemes for federally funded programs
but various reasons prevented these initiatives from materializing.3
By the mid-1990s scattered evaluations -such as those of the Milk Social Supply Program
(Abasto Social de Leche - Liconsa), the Rural Supply program (Abasto Rural - Diconsa),
the National Solidarity Program (PRONASOL), and several job training programs were
being conducted in the social sector, but they remained isolated and, to a large extent, ad
hoc. Also, the Program for the Modernization of the Public Administration (PROMAP),
created in 1996, introduced performance indicators and internal evaluations. However, it
is important to emphasize that these initiatives in the first period focused solely on
strengthening internal controls and did not make results publicly available, thus limiting
incentives for external accountability.
1.2 Establishing a Systemic Role for Evaluation: 1997–2003
In the late 1990s, Mexico underwent an important change in political control and
democratization, which consequently affected the character of evaluation reform in the
country. Prior to this period there had been little incentive for transparency and
accountability in the public sector because the country had been under a single-party regime
since the 1920s. However, beginning in the late 1990s, political events led to the institutional
strengthening of government accountability.
It was during this period that Mexico implemented its first rigorously planned and conducted
program evaluation. This impact evaluation—of the Progresa/Oportunidades program in
2 See Cardozo 2006, pp. 155–157. See also IEG-World Bank 1983, PIDER I. http://www-wds.worldbank.org.
3 This seems to have been the case with the Secretariat of Budget Programming. See Hernandez 2008.
3
1997—was innovative and influential and became a successful case study. The evaluation
became internationally known as a best practice, and similar programs with corresponding
evaluations were implemented in a number of developing countries.4
The Oportunidades evaluation gained national and international visibility at a critical
political moment. Mexico’s government took notice of the important role that evaluation
could play in public administration, and the experience opened the way for a more systematic
evaluation function. SEDESOL pioneered an effort to implement a sectorwide M&E system
and progressively used evaluation information in policy decision making. This was an
important influence on other cabinet secretaries and the federal government.
In 1998, Congress introduced a requirement that all federal programs providing subsidies had
to prepare rules of operation (ROP), with basic information about a program’s design,
objectives, performance indicators, beneficiary populations, and operations. ROPs greatly
improved the availability and quality of performance data as well as the quality and ease of
M&E. In 1999, a major milestone for evaluation occurred when Congress mandated that all
programs with ROPs had to be evaluated annually by external evaluators. The mandate
covered approximately 25–30 percent of the federal budget5 for programs, and the number of
evaluations jumped from single digits to over a hundred in 2001 and in subsequent years.6
Additional changes that helped to build a foundation for performance evaluation in
government included: the introduction of a “Presidential Goals” system,7 creation of the
Auditoria Superior de la Federacion (ASF) (Audit Office),8 passage of the Transparency
and Public Information Access Law,9 and creation of the Public Administration Secretary
(SFP).10
1.3 Progress toward a Governmentwide M&E System since 2004
Further institutionalization of evaluation began in 2004, when internationally accepted M&E
standards and best practices in social policies were introduced through the Social
Development Law.11
This legislation also created CONEVAL, a technically autonomous
institution with the mission of measuring national results in poverty reduction and
coordinating the evaluation of social programs by the federal government.
4 Today, more than 30 countries in the developing world have had similar cash transfer programs with their
respective evaluation components. Some examples are Colombia (Familias en Accion), Nicaragua (Red de
Proteccion Social), Jamaica (Advancement through Health and Education), and Honduras (Asignacion Familiar).
5 Interview with Gabriela Perez, Former Adjunct Director for Evaluation in CONEVAL, at World Bank
headquarters, Washington, DC, Expert Roundtables on Performance Measurement Tools, February 3, 2009.
6 As a consequence, evaluation expenditures grew significantly, averaging $15 million annually, not including
multiannual impact evaluations. Taken from Hernández 2007b.
7 The Presidential Goals system aimed to make explicit the government priorities and to monitor performance.
Although the system achieved little, it was a sign of the new role that performance evaluation and management
for results were to play in the Mexican administration.
8 This oversight body was created in 2001 to second the legislative branch in controlling the executive.
9The Transparency and Public Information Access Law (DOF 11-06-2002) reemphasized the role of evaluation
as a source of performance information. Following this mandate, all completed external evaluations are now
publicly available on ministry websites,
10 This new body replaced the Control and Administrative Development Secretary (Secretaria de la
Contraloria y Desarrollo AdministrativoSECODAM) in 2003 and was granted responsibilities in
performance auditing.
11 DOF 2001 of 2004. The leading agency for this legislation was the Secretariat of Social Development
(SEDESOL), which through the Under Secretary for Planning, Prospect, and Evaluation had set up a technical
team for developing the social sector M&E system since 2001.
4
The first articulated vision for a governmentwide performance management system was in
the 2006 Federal Budget and Fiscal Responsibility Law,12 particularly with the creation of
the Performance Evaluation System (SED). Unlike previous efforts,13
this reform was driven
and led by high-level officials seeking to modernize the budget and to facilitate evaluation of
public expenditures, among other things. Important parts of this reform were: (i)
institutionalizing the links between strategic planning and the budget process, (ii) the launch
of a performance-based budgeting initiative, and (iii) the introduction of planning and
evaluation tools at the program level.
Instrumental to the reform was that the SED (led by the Ministry of Finance) was to be
integrated with two existing activities—the Management Improvement Program (PMG),
under the Secretariat of Public Administration (SFP); and the Social Program Evaluations,
coordinated and conducted by CONEVAL. Evaluations of other sectors were to be
undertaken by the SHCP and other agencies (see figure 1). Implementation of the integrated
SED started in 2007 with the launch of the National Evaluation Guidelines.14
It was actually
a first step toward harmonizing and unifying evaluation practices across all federal agencies.
Another important step was the setting of performance budgeting guidelines (prepared by
SHCP) for all federal agencies in 2008.
Figure 1: Mexico’s Whole-of-Government Performance-based Management System
12 DOF 30-03-2006.
13 Similar endeavors had been tried before, in 1995, in the so-called Budget System Reform, which attempted to
introduce program budgeting as well as the Performance Evaluation System (SED). The initiative was
implemented with little practical success but it did manage to introduce the seeds of the discourse on managerial
and transparency in public expenditure as well as greater clarity on the concept of performance management.
See Cardozo 2006, p. 121.
14 General Guidelines for the Evaluation of Federal Programs in the Federal Public Administration (Audit
Office), DOF 30-3-07.
Annual Budget
Performance-Based Budgeting
Program Implementation
Performance-Based Impementation
Performance Evaluation System
(SED)
Secretary of Hacienda
-SHCP
Management
Improvement
Program (PMG)
Public Administration
Secretary -SFP
Evaluation of
Policies and Programs
CONEVAL, SHCP,
Others
National Development Plan
Performance-Based Planning
Performance Management & Evaluation
Permanent Working Group
(SHCP, SFO, CONEVAL)
-Annual Evaluation Program PAE
-Follow up to recommendations
Annual Budget
Performance-Based Budgeting
Program Implementation
Performance-Based Impementation
Performance Evaluation System
(SED)
Secretary of Hacienda
-SHCP
Management
Improvement
Program (PMG)
Public Administration
Secretary -SFP
Evaluation of
Policies and Programs
CONEVAL, SHCP,
Others
National Development Plan
Performance-Based Planning
Performance Management & Evaluation
Permanent Working Group
(SHCP, SFO, CONEVAL)
-Annual Evaluation Program PAE
-Follow up to recommendations
5
2. CONEVAL AND THE INSTITUTIONALIZATION OF EVALUATION
The transition to democracy, which took place in Mexico at the end of the 1990s, brought
about important changes in the political environment and these changes helped the
development of new evaluation and results-based management institutions. The
emergence of CONEVAL was a landmark response to those changes from both a
political and administrative standpoint. Politically, the end of the single-party regime
prompted Congress to find different ways to exert control over the executive branch and
so to ensure greater political accountability over social spending. The opposition’s
mistrust of the official use of performance information (e.g., the measurement of poverty,
the selection of social programs beneficiaries) was therefore an important driver.
Administratively, Mexico had a long history of many good government programs being
halted when new administrations came into office because the new politically appointed
officials wanted to implement their own programs, without much evidence on the
benefits and costs of such decisions.15
Introducing evaluations was therefore an
opportunity for the professional administrators to strengthen and modernize government
decision-making processes, and the creation of CONEVAL was an institutional solution
that aligned both political and administrative incentives.
2.1 Objectives
The Social Development Law of 2004 and the statute that supports the creation of
CONEVAL state the objectives of the council as follows: (i). Regulate, conduct, and
coordinate the evaluation of social development policies at the national level, as well as
the programs, funds, and actions executed by public agencies; and (ii) Set up the
guidelines and criteria for identifying and measuring poverty at the national, state, and
municipal levels of government, ensuring the transparency, objectivity, and technical
rigor that such activity requires.
This very unique combination of objectives explicitly makes poverty reduction the main
performance indicator of social development policy; all social interventions should
therefore contribute to this outcome. In addition, the law also clearly points out that
poverty measurement needs to be treated as an evaluation itself, making use of
independent analyses as well as technically robust and credible methodologies. The
strategic rationale behind these objectives was to contribute—throughout the production
and progressive utilization of M&E informationtoward increasing the efficiency,
effectiveness, and accountability of social development policies and poverty reduction
strategies.
2.2 Organizational Structure, Roles, and Responsibilities of Key Stakeholders
CONEVAL is the lynchpin of the social sector M&E system in Mexico. In this capacity,
it acts as one of the leading agencies in the implementation of results-based management
in the public sector more broadly. CONEVAL coordinates cross-sectoral evaluation
activities, sets out standards and methodologies for the evaluation of social programs,
provides technical assistance to ministries and agencies,16
15 See Levy, 2007.
and undertakes or manages
some specific evaluations. It also concurs with development organizations to plan and
undertake evaluation work and to advise national entities and subnational governments in
16 Advice is regularly provided to cross agencies such as the President’s Office, the Ministry of Housing and the
Public Administration Secretariat, as much as to a number of line secretariats from social and other sectors.
6
the implementation of their M&E systems, as well as on how to conduct good-quality
program evaluations.
In addition, CONEVAL prepares and consolidates evaluation reports and disseminates
the evaluation findings among Congress, government agencies, and civil society. To
accomplish these tasks, CONEVAL has a staff of 70 professionals, including
administrative personnel, and in 2008 it had an annual budget of about 120 million
Mexican pesos (US$9.2 million).17 The institutional structure is shown in figure 2.18
One particular feature that deserves attention is the mixed nature of CONEVAL as a
relatively independent government agency with a unique governance structure. Although it is
located within the executive branch, and its executive director is appointed by the federal
government, leadership of CONEVAL relies on an independent collegiate body made up of
six academic councilors. These six individuals are democratically elected19
for a period of
four years, and they are chosen from certified academic institutions. The councilors are
involved in all of the agency’s decisions and the definition and review of evaluation projects.
They also provide general guidance on the administrative direction of the institution and play
an important role in the methodologies for poverty measurement.
A second feature is CONEVAL’s particular focus on the social sector. It was conceived
of as a sectoral institution and is legally supported by a sectoral mandate, namely the
Social Development Law.
Figure 2: Organizational Structure of CONEVAL
Another distinctive feature is the integration of social program evaluation and poverty
measurement functions under the purview of one Executive Secretariat (figure 2). In most
developing countries, these two activities are usually managed by separate institutions,
even though they share common objectives, such as the assessment of government
results, and key requirements, such as independence and credibility of its products.
17 However, a number of consultants are annually contracted out, mainly to undertake different types of
evaluations.
18 Oracle Customer Snapshot 2008.
19 The members are elected by the National Commission of Social Developmenta coordination arena for
setting up the government’s actions and investments to accomplish the objectives of social development policy.
Executive Secretary
Secretariat Academic Councilors
General Direction
for Administration
General Direction
for the Analysis of
Poverty
General
Direction for
Evaluation
Internal
Audit Unit
Unit of
Planning and
legal norms of
evaluation
Unit of
Analysis and
follow up to
results
Direction for
Monitoring of
Social
Programs
Direction for
Impact
Evaluation
Direction of
poverty norms
and measurement
methodology
Direction of
poverty analysis
and measurement
Direction of
surveys and
statistics
7
CONEVAL is charged with another important institutional mandate which is poverty
measurement in the country. CONEVAL devotes substantial work and staff to the estimation
of poverty, analysis, and the gathering of information to improve monitoring. Consistent with
international best practices, CONEVAL not only monitors income or poverty measures but is
moving toward the estimation of multidimensional poverty.
Granting CONEVAL these unique institutional characteristics and mandate has had
advantages and disadvantages.20
The advantages stem from the fact that Mexico has
created a valuable champion for the development of evaluation. The expertise and
specialization of its staff, and the participation of academics, has enabled CONEVAL to
develop methods and to conduct and commission evaluations that are technically reliable
and credible to both internal and external stakeholders. Similarly, CONEVAL has
contributed to the use of evaluation information and, arguably, its coordination and
technical support has lowered the costs of evaluation practices in all social agencies.
Integrating poverty measurement and social program evaluations provided an additional
advantage by increasing the utilization and sharing of surveys and statistical information
for measuring the performance of social interventions.
However, there are some disadvantages of using the CONEVAL approach too. On the
one hand, its social sector focus limits CONEVAL from evaluating programs in key areas
of the government (e.g., infrastructure, industry, internal security21
), some of which may
have important effects on social outcomes. There are also legal limits to CONEVAL’s
evaluation oversight of the public sector, and even limits to its providing technical
assistance to key government departments. Without specialized institutions to oversee
and support the areas of the public sector that CONEVAL cannot, the technical capacity
gap in evaluation between the social sector and other relevant areas and levels of the
public administration has been increasing.
CONEVAL’s relations with ministries have also been difficult at times, particularly when
evaluations point out implementation problems and program shortfalls. Another related
and important point is that being separate from the government’s decision-making
processes limits CONEVAL’s capacity to enforce the use of evaluation information to
directly influence policy and budget decisions (e.g., in budget and planning), even in the
social sector. Despite these drawbacks, the evaluation knowledge that CONEVAL has
been providing has significantly contributed to the improved technical quality of
evaluations in the social sector. Its products and technical support are highly visible and
appreciated as a net value added throughout the public administration. Furthermore,
CONEVAL is unique within Mexico because there is, as of yet, no other entity capable of
providing this high level of expertise.
2.3 Main M&E Tools and Methodologies
The responsibility of leading the evaluation of social programs and poverty reduction
prompted CONEVAL to develop a number of basic tools to effectively carry out its
mandate. These principally included performance evaluation guidelines, program
planning, and evaluation frameworks—a range of evaluation methodologies suited to
different performance measurement purposes. These elements have been developed or
20 Officially, CONEVAL started its operations in 2005 when its corporate structure was set in its Statute of the
National Council for the Evaluation of Social Policy and its decree (DOF 24-08-2005).
21 Except for programs involving transfers to the subnational level (Ramo 33), where CONEVAL is currently
evaluating programs related to infrastructure and public security.
8
improved since 2005 following international standards but a high degree of innovation
and customization has also been introduced in response to Mexico’s particular needs.
It is important to note that although CONEVAL’s mandate is exclusively focused on the
social sector, most of the tools it developed were eventually adopted by the federal
government to serve broader M&E purposes, such as performance-based management.
2.3.1 General Guidelines for the Evaluation of Federal Programs
Introducing a culture of evaluation to ensure the proper implementation and use of
evaluation is a complex and long-term effort. To address resistance by public servants
anxious to maintain the status quo, what is often needed is standardization in the concepts
and in the language of program results, as well as clear definitions of the steps all
agencies must follow. The assimilation of these M&E reforms often leads to the adoption
of formal regulations, which might enable sustained use of effective practices and tools
over time.
CONEVAL took the lead in conveying the importance of having a formal set of policy
guidelines. No other public agency at the time had the technical capacity and the
credibility to take the lead in the formulation and implementation of such guidelines. For
SHCP and the President’s Office, having such guidelines represented an opportunity to
make the institutionalization of links between strategic planning and the budget process
operational, as stated in the 2006 Budgeting and Fiscal Responsibility Law.
In March 2007, CONEVAL, SHPC, and SFP jointly launched the General Guidelines for
the Evaluation of Federal Programs. These guidelines introduced technical criteria for
regulating the evaluation of federal programs (e.g., mechanisms to ensure quality and
utilization), standardized procedures for improving the formulation of performance
indicators and strategic objectives, and called for the development of M&E systems at all
agencies of the federal government.
The first stage of the guidelines implementation focused on the development of
evaluative frameworks (matrix indicators) as well as consistency and results analyses for
all federal programs with ROPs. The second stage focused on annual evaluation plans,
including the whole range of evaluation methodologies designed and put into place by
CONEVAL.
Significant accomplishments brought about by the General Guidelines in Mexico’s public
sector included: having specific steps for the application of logic frameworks to all
budget programs; adopting a single, results-based, M&E language in the federal
government; providing standardized M&E methodologies; and initiating a
governmentwide, learning-by-doing M&E process among federal agencies and officials.
In conclusion, the guidelines integrated, and were an improvement over, all previous
mandates; fostered evaluation in government; contributed to the consolidation of a solid
legal base; and advanced the public sector in terms of M&E international standards.22
22 The main laws that have driven the results agenda are: (i) the Social Development Act (DOF 20-01-2004), (ii)
the Budget and Fiscal Responsibility Law (DOF 30 - 2006), (iii) the Federal Law of Transparency and Access to
Information (DOF 06-06-2006), and (iv) the Annual Budget Law for 2006, 2007, and 2008.
9
2.3.2 Matrix Indicators
The results orientation adopted by Mexico progressively created powerful demands for
performance information at different government levels. One source of such demand has
been the push by the President’s Office to have the National Development Plan (NDP)
serve as the main reference point against which to align sector plans and program goals
as well as measure their progress. Another strong source of demand has been the SHCP
movement toward performance budgeting and implementation of the SED. Owing to this
move, a need developed for a mechanism to strengthen program planning and the results
focus of the federal budget.
In response to such needs, CONEVAL, SHCP, and SFP introduced the international
standard logic framework methodology, widely known as the matrix indicators (MIs),
into the Mexican context. This matrix was, in fact, mandated and regulated by the general
guidelines for M&E.
The introduction of the matrix into all programs was challenging. It involved the review
of all program designs and their internal and external logic as well as identifying
performance indicators. This, in turn, required the development of program planning and
evaluation capacities at all agencies. Consequently, CONEVAL, with the support of its
partners, led an intense training initiative in the logic framework methodology, which
was supported by international and national institutions.23 Over 1,620 officials were
trained in 65 workshops, and MIs were set up in 389 programs (covering 70 percent of
the federal budget).24
The use of MIs was initially focused on programs with rules of operation (ROP), that is,
programs giving subsidies. However, because of high demand, MIs were rapidly rolled
out with the objective of having them in all federal programs by 2010.25
CONEVAL also
held training workshops with subnational governments, to begin transferring the
principles of the MI in performance-based government, in accordance with the mandates
of the 2006 Law of Fiscal and Budget Responsibility.
Introducing the MI as one to the main system’s tools brought multiple benefits to the
federal administration, including: (i) an operational way to facilitate the shift from
traditional management (input/activity-based) toward performance management
(output/outcome-based); (ii) a unified program and planning and evaluation framework to
standardize results measurement; and (iii) improved consistency and quality in program
design.26 In Mexico, as in many countries, as governments create and modify programs,
the original rationale of many of them tends to be weakened over time, as does their
alignment to national and sector plans. Therefore, the impact of the MIs and
accompanying changes is expected to be significant.27
23 Primarily, the Latin American Institute for Economic and Social Planning (ILPES), Economic Commission
for Latin America (CEPAL), and local academic institutions.
24 Burgos 2008.
25 Perez-Jacome 2008.
26 The review of the program’s internal logic was significantly complemented by the evaluation of consistency
and results in 2007, which also showed many of these problems.
27 A World Bank evaluation for the 2001–07 period, for instance, showed that programs rated with better logic
frameworks were satisfactory in 90 percent of the cases, whereas those programs with medium or lower quality
log frames were satisfactory in only 50 percent of the cases, thus suggesting that better results frameworks, such
as the matrix indicator, might be associated with better program design, implementation, M&E, and supervision.
See “The Role of M&E in the Results Agenda and IEG Database” at www.worldbank.org/ieg.
10
However, standardizing the quality and content of the MIs, is still needed, just as much as
a process for their periodic review is also required. Analysis of a nonrandom sample of
matrixes shows there is still uneven quality among sectors. Some matrixes present very
high standards in terms of the technical definition of assumptions, goals, indicators, and
identification of financial resources and information, while other matrixes are unclear
regarding many of these aspects and are lacking in sequencing among inputs, activities,
products, and outcomes.
Internal validation of many of the matrixes by the respective agencies’ higher-ranking
officials is also critical. Without such political endorsement, the process remains mostly
at mid-level and does not advance beyond the technical echelons of the government.
Additional training and guidance at the national and sectoral levels will also be needed to
help in changing the perception, which currently exists at different levels of the MI, that
it is yet another administrative requirement from CONEVAL, SHCP, or SFP, rather than
as a managerial tool that will actually benefit the agencies themselves.
2.3.3 Program Evaluation Methodologies
One of the main challenges of institutionalizing evaluation is the development of
technically sound and standardized methodologies. One of CONEVAL’s first steps was
therefore to develop a “menu of evaluations” that could provide specific performance
information to suit the needs of different institutions within and outside the government.
The menu included a host of evaluation techniques designed or adapted to cover all levels
of the government’s program implementation chain. The primary implicit logic of the
methodological approach and the scope of each method are presented in figure 6.
Figure 3: CONEVAL’s Main Evaluation Methods
The methodologies cover areas such as the quality of program design, indicators,
processes, and strategic planning and impact of programs. Their specific contributions to
the government’s performance information are the following:
Design evaluation is conducted on programs that are in their first year of
implementation. The evaluation assesses if a program makes a clear contribution to the
solution of a particular policy problem and if the objectives of the program are aligned
with the objectives of its ministry. This evaluation only uses secondary information and
involves no original data collection. It is mandatory for all new programs and is targeted
to inform program managers, budget officials, and other ministry staff. At the moment,
Design Inputs Activities Products Effects Impacts
Indicator Evaluation
Process Evaluation
Design
Evaluation
Evaluation of Consistency
and Results
11
an ex-ante (prior to implementation) analysis to be completed by the programs is
contemplated but this is not yet in operation. It is hoped this assessment will help identify
any potential problems or flaws in the programs, thereby increasing the likelihood of
more effective interventions. It also expected this assessment will clarify how a new
program will add to existing policies, thereby reducing the likelihood of duplication.
Process evaluation appraises the efficiency and efficacy of operational processes of a
program during its implementation and provides feedback for improvement before the
program ends. It allows program managers to identify bottlenecks in the program’s
service delivery mechanisms and leads to improvements in the program’s operation,
which then benefit the target population. As such, it should play an important role in
providing explanations for low performance. The process evaluation is implemented on a
case-by-case basis and is targeted for use by program managers, budget officials, and
other ministry staff.
Program consistency and results evaluation is a rapid assessment used to obtain a
general and comparative perspective of the design, strategic planning, coverage and
targeting mechanisms, operation, beneficiary perception, and results. It is used to
highlight specific strengths and weaknesses of programs and to motivate decision makers
to consider which of these should be evaluated in a more rigorous way, using the other
methods. This type of evaluation only uses secondary information and involves no
original data collection. It is targeted at a broad range of users, including program
managers, high-level decision makers and the general public.
Indicators evaluation is a methodology that is currently being developed and will start
as a pilot program in 2010. It will measure the quality of indicators, including in the MIs
by considering whether they are clear, relevant, adequate, and monitorable, as well as
whether their information comes from reliable data sources. The objective is to ensure
that accurate information is collected for performance-based monitoring, evaluation,
planning, and budgeting. The indicators evaluation will be implemented on a case-by-
case basis and is targeted at program managers and mid-level ministry staff.
Impact evaluation seeks to measure changes in the conditions of well being of the target
population that are attributable to a specific program. This type of evaluation provides the
most technically robust evidence about whether a program is working or not. As such, its
purpose is to inform high-level officials on the extent to which a program should be
continued or not, and if there are any potential modifications needed. This kind of
evaluation is implemented on a case-by-case basis, is usually completed at various points
in the program along with the other methods, and is targeted toward more executive
levels of government, though it does provide useful information to program managers
and all kinds of government officials.
Based on the above described methodologies, CONEVAL expanded evaluation practice
to the entire social sector.
A number of benefits have been derived from the use of standard methodologies. First,
Mexico is building a performance information database with comparable information
across its programs, which will allow CONEVAL to analyze the effectiveness of wider
policies and even institutions more accurately in the future. Second, the database will also
create a larger critical massof evaluation knowledge, particularly if public access to
this information is facilitated. CONEVAL has already started to show aggregated
12
information (by line ministry) on its website, using the results of the 2007 consistency
and results evaluation, which were applied to 106 programs. 28
In terms of these methodologies, there are several issues that will need further
consideration. One issue is that the coordination of more than 100 evaluations annually
will require capacities not only at CONEVAL but also at the responsible evaluation units
of the sectors and agencies. The added technical and operational burden will require
development of appropriate tools to manage the growing volume of performance
information and mechanisms if the quality of such evaluations is to be ensured. Another
issue is guaranteeing that existing methodologies effectively complement one another
because some current methodologies seem to exhibit slight overlaps.29
Proper solutions to these issues are needed not only to ensure adoption of the methods by
the programs, but also to satisfy demands for transparency and accountability from
groups such as the media and civil society organizations.
CONEVAL has proved that it is committed to exploring how evaluation options can be
enhanced to provide useful inputs to decision making. Nevertheless, caution should be
exercised not to overextend the choice of evaluation tools to an unmanageable number. It
is also important to ensure that the SHCP, SFP, President’s Office, and programs are
actively involved in the process so that the methods can effectively match the demands,
especially when the methods are to be implemented in other sectors through the SED.
2.3.4 Annual Evaluation Program and Sector Evaluation Units
To implement the diversified menu of evaluations, CONEVAL persuaded its partners
SHCP and SFP—to support the establishment of a national Annual Evaluation Program
(PAE) every year. The PAE is jointly prepared and published each year by three agencies
(CONEVAL, SHCP, and SFP), in coordination with line ministries and agencies of the
evaluated programs.
The PAE grew out of the need for advance planning of evaluation activity—making sure
that the programs to be evaluated are strategically selected and that appropriate
evaluation methods are chosen to assess them. The PAE also serves as a tool to match the
demand and supply sides for evaluation, and seeks to ensure that the findings manage to
influence decision-making processes. This is expected to work, in particular, through the
influence that SHCP, SFP, and CONEVAL can exert, from different perspectives, over
the program.
CONEVAL is the principal technical leader of the PAE, working closely with line
ministries, while SHCP and SFP primarily play supporting roles. The PAE is published in
the middle of the calendar year before the period it covers so that line ministries can plan
and budget ahead of time for the evaluations.
The consistency and results evaluation has played a special role in terms of informing the
PAE. Based on the findings of this evaluation, completed in 2007, CONEVAL had
constructed a baseline to inform future decisions about which programs would need
deeper evaluation. As an aid in setting priorities, the consistency and results evaluation
provided a snapshot of key areas of performance, including design, targeting, and
28 Hernández 2007a.
29 This seems to be the case, for example, with the indicators evaluation and the evaluation of consistency and
results.
13
operation, which were important in the negotiations of the 2008 and 2009 PAEs with
sector and agencies.
The administrative units that manage the evaluation menu within existing line ministries
were introduced in 2007. The main responsibility of these units is the coordination,
contracting, and supervision of evaluations. The purpose of these units is to ensure that
evaluation work within each ministry is conducted properly and with the needed human
and financial resources. Though some line ministries, such as the Secretariat of Social
Development and the Secretariat of Education, already had such units, most others did
not and had to establish them.
Before the units were set up, program managers were responsible for coordinating annual
evaluations. However, managers were usually overburdened with implementation tasks,
and lacked the capacity and time to accomplish coordination of evaluation work.
Moreover, there was the problem of conflict of interest stemming from the program
managers simultaneously being both the chief implementer and the responsible official of
the election of the external evaluators of the programs. Currently, the units are seen as the
counterparts of CONEVAL, SHCP, and SFP in the PAE.
A relevant aspect of the PAE was CONEVAL’s approach to standardizing procedures
and content for terms of reference (TORs) among all agencies. The standard TORs
provide detailed technical guidance for evaluators and government officials about what
questions the evaluation needs to answer and how, as well as details about
responsibilities, timeframes, and products to be delivered by the evaluators. The TORs
are prepared in coordination with line ministries and programs, and then distributed to the
agencies.
Prior to the technical preparation of TORs by CONEVAL, ministries and programs were
asked to contract and supervise evaluators with no application of standard criteria. TORs,
therefore, helped to reduce variability in evaluation quality across the public
administration.30
As in the case of other tools, the TOR contributed to creating a common
performance language, and enhanced transparency in conducting evaluations because
final versions of all program evaluations were made available on CONEVAL’s website.
2.3.5 Poverty Reduction Evaluation Methods
Until recently Mexico did not have an official position on how to define poverty. This is not
unusual among countries, though a few, notably India and the United States, have a long
tradition of maintaining an official poverty line. Because the definition of poverty is
necessarily a product of both technical analysis and political process, it is a major asset to
have such an official position. This can help shift debate from the number of poor people to
the really important questions of causes, trends, and what to do.
In 2000, the government of Mexico convened a committee of experts to prepare an official
position on poverty, the Technical Committee for Poverty Measurement (CTMP), which was
comprised of academics from a range of fields, along with observers from the government.
CONEVAL adopted the recommendations from the CTMP. The assessment work of the
committee was of high quality, and made full use of the international literature on poverty
30 Interviews conducted during 2008 showed that the introduction of TORs was considered extremely positive
by both line ministry staff working with evaluations and external evaluators contracted to complete the
evaluations.
14
measurement. One of the most important contributions of the CTMP was setting up a
transparent process on poverty measurement, which has been carried out by CONEVAL.
CONEVAL has a direct mandate established by the Social Development Law to measure the
results of poverty reduction and social programs because poverty reduction should be the
main and final outcome of social policy. CONEVAL recognized the multidimensional and
complex nature of poverty, and has introduced multiple indicators in assessing the structure
and trends of poverty.
For income poverty, CONEVAL uses per capita income as the indicator of well-being at the
household level. There are three official poverty lines. The first is the food-based poverty
line, which is an estimate of the income required to purchase a food basket to satisfy
minimum nutritional requirements. The second line is the capacities poverty line which
measures household expenditures above the food-based line. The third line is the assets or
patrimonio line, which is the food-based level plus other basic assets. This third poverty line
is above the previous two poverty lines.
Recently CONEVAL has been working on an approach to measuring multidimensional
poverty, which has required extensive consultation with academics in terms of which
indicators of well-being to include in the measurement as well as other specific technical
issues.
CONEVAL has gained a lot of credibility and trust in the measurement of poverty in
Mexico. This is due to the organization of CONEVAL and its transparent process of
estimating poverty and the dissemination of results. CONEVAL has used a similar
approach in the evaluation of social programs and policies. First, by defining clearly the
questions that the evaluations should address, then identifying the evaluation instruments
to answer those questions, and then carrying out the evaluations. The transparent and
technical process with which CONEVAL carries out poverty reduction and social
program evaluations has helped the federal government to position the institution as an
organization trusted by civil society.
2.4 Utilization of Evaluative Information Produced by CONEVAL
The driving force behind Mexico’s results-based management has been transparency and
accountability. Congress has been the primary recipient of evaluation reports since 1999.
Government ministries, particularly, the Ministry of Finance and the Ministry of Public
Administration, as well as the programs themselves have received evaluation reports.
More recently, the public has also gained access to the reports. During the period 2000–
06 more than 500 external evaluations were conducted but very little of that information
was effectively used. There was little awareness then of the role that evaluation could
play in improving government programs.
Evaluation activity at the time lacked the incentives and institutional arrangements to
ensure use of the findings. The 2007 Evaluation Guidelines were a specific response to
low usage of the evaluation findings. One of the purposes of issuing the guidelines was to
ensure that key evaluation results would be disseminated and effectively reach
government decision makers. A follow-up mechanism, to prioritize and implement
evaluation recommendations, was issued in October 2008—the Guidelines for the
Utilization of Evaluation Results. These guidelines identified responsible actors and
instruments, given in a four-step process (figure 4).
15
Figure 4. Follow-Up to Recommendations
Source: Official Memorandum No 301-A-1981. Follow-up mechanism to potential improvements identified by external
evaluations of federal programs, México D.F., 14 October, 2008
Once the evaluation is cleared by the program or agency, recommendations are selected
in terms of its relevance and feasibility of implementation. Recommendations are then
classified by the type of actor involved in the solution, depending on their scope of
influence (e.g., program, institutional, inter-institutional level), and prioritized in terms of
importance: high, medium, or low. Based on these steps, a two-level (program and
institution) implementation plan is requested by the programs, which should outline the
activities and timeline for the introduction of recommendations at these two levels. Once
the plan is prepared, dissemination on the ministry’s website is encouraged. Evaluation
units within line ministries are expected to be responsible for this process.31
Increased use of evaluation findings has also been boosted by the SHCP through the
implementation of performance budgeting. Currently, agencies are required to submit
budget program performance indicators and their evaluation reports to the Federal Budget
Office. SHCP is also setting up a Performance Evaluation Unit, which will be in charge
of informing the budget process on a permanent basis. This unit is expected to become
one of the most important users of performance information at the federal level.
The boost to evaluation has slowly but progressively strengthened program planning. It
has also given evaluation units a more active role within the government ministries.
These units have also become important users of evaluation information. Congress and
control agencies, while they establish a more fluid relationship with executive agencies in
charge of evaluation, have also started to increase the use of performance information.
The Public Audit Institution, which supports Congress as a check on the executive, has
received on-demand training workshops from CONEVAL on issues such as the MIs,
formulation of performance indicators, and the benefits of using these tools for purposes
of budget control. In fact, the auditing institution now revises performance indicators of
31 Oficio NO 301-A-1981, follow-up mechanism for the aspects that could be improved derived from the reports
and the external evaluation of federal programs, 2008, México D.F., October 14, 2008.
Step 1: Analysis and selection of
recommendations for improvement
Step 2: Classification and prioritization
of recommendations necessary for
improvement
Step 3: Development of a work plan
defining commitments, principal
activities, and timelines, to address
prioritized recommendations
Step 4: Dissemination of the results of
the work plan on the website of the
ministry
Selection Criteria:
Clarity
Relevance
Justification
Feasibility
Classifications:
Specific
Institutional
Interinstitutional
Intergovernmental
Levels of Priority:
High
Medium
Low
16
all programs and issues an official report on their consistency, with binding
recommendations for the programs.
A sign of the trend toward greater utilization of evaluation findings is offered by the 2009
budget process. SHCP and the President’s Office had directly requested evaluation
information from CONEVAL, with the aim of prioritizing resource allocations among
programs. CONEVAL presented a report on the 106 programs evaluated using the
consistency and results methodology, and SHCP used such evaluations as additional
information to inform the 2009 budget,32
which produced some changes in a number of
programs. Three examples are provided in table 1.
Table 1. Extent of Utilization of Evaluations
PROGRAMS
EVALUATION
FINDINGS
2009 BUDGET
1. Communitarian model
for the education of a
vulnerable population
- Need to improve
program planning
- Duplication of efforts,
a single, unified design
and matrix indicator
suggested
The programs were
merged into a single
intervention called
Initial and basic
education for rural
indigenous
population”
2. Communitarian model
for the initial and basic
education of indigenous and
migrant populations
3. Training support program
Limited perspective of
job competencies
generation
Substituted by the new
support to productivity
program
4. Efficient use of water and
energy
- Focused on same
population with similar
components
- Merging would
simplify administrative
structure and would
benefit monitoring
Integration into a new
single “Program for
the Modernization and
Technical
Improvement of
Irrigation Units
5. Full usage of hydro-
agricultural infrastructure
Source: SHCP, Expenditure Under Secretary, Office of the Budget Policy and Control Adjunct Director.
Burgos 2008.
According to the SHCP, the existing evaluations and the MIs implemented by federal
agencies improved the alignment between program objectives and national goals in the
2009 budget. For the first time, budget allocations were linked to program indicators and
goals.33
It is also important to mention that, in addition to increased use by government agencies,
the national media regularly uses the program results that CONEVAL publishes on its
websites, and there is recent evidence that civil society organizations have also begun to
use the results as a way to exert public influence too.34
32 Information can be viewed at www.coneval.gob.mx.
33 Burgos 2008.
34 As in the case of Social Management and Cooperation (GESOC), a civil society organization with the goal of
conducting research, monitoring, and evaluation to increase the accountability of public programs and civil
society organizations. See www.gesoc.org.mx.
17
3. THE PERFORMANCE EVALUATION SYSTEM
The Federal Law on Budget and Fiscal Responsibility, amended in 2007, updated the
Mexican budget and was the basis for guiding federal spending toward performance
budgeting. To that end, the law created the SED, which eventually integrated all previous
evaluation efforts into a single nationwide M&E system.
It is important to note the progressive evolution of the SED, which was first suggested in
1995 as part of a series of reforms to the budget system. Substantive efforts only
materialized after more than a decade, in 2008, when the SED began its formal operation.
It is not easy to determine all of the factors that contributed to the evolution of the SED.
It is possible, however, to point out that the establishment of CONEVAL’s M&E model
and system, the progress already achieved in setting up MIs in all programs, as well as
the existence of a pool of evaluation methods applied to more than 130 programs
annually, all created a solid foundation from which the system could take off.
The SED is operated by the SHCP, SFP, and CONEVAL and has two pillars. The first is
the monitoring of programs supported by the federal budget using the information base of
the MI. This activity takes place during program implementation and seeks to warn
budget officials about possible delays in the achievement of the outcomes. The second
pillar is the evaluation of program results (within the PAE), which, in turn, has two
subcomponents. These subcomponents are the evaluation of federal programs (including
the evaluation of social programs undertaken by CONEVAL), and the evaluation of
processes and public services, which takes place within the Management Improvement
Program (PMG) led by the SFP.35
The three operating agencies have specific roles and responsibilities in regard to the SED.
CONEVAL provides technical assistance in the realm of monitoring and evaluation,
based on its experience and technical capacities. The SFP secures management standards
within the agencies through certification methods. Finally, the SHCP leads the
performance-based budgeting initiative and ensures that the information produced by all
agencies is funneled into the budgetary decision-making process. The three agencies also
jointly provide regulatory guidance on the conduct of evaluations and the follow-up on
the implementation of recommendations (figure 5).
35 The Public Administration Secretariat (SFP) is in charge of three core responsibilities regarding M&E in
Mexico: (i) developing and managing systems for monitoring staff and managerial performance in the delivery
of the government's budgetary programs, (ii) removing systemic obstacles to improved management
performance, and (iii) providing independent quality assurance of the performance indicators selected by SHCP
for monitoring budget program performance.
18
Figure 5: SED and Institutional Coordination
The SED, making use of the logical framework methodology (that is, the MIs), aligns the
objectives of the National Development Plan, sector programs, and budget36
(see figure 5).
In order to inform the budget, the SED continually produces M&E information regarding
the extent to which goals and objectives of federal programs have been achieved. With
this evaluative information being supplied by the MIs and the PAE, it is hoped that
budget officials can provide warnings about possible delays in the development of such
goals, and thus propose actions for improvement.
To ensure institutional coordination in the implementation of performance-based
management activities, the SED established a Performance Management and Evaluation
Working Group involving the three operating agencies. This group has already worked
together effectively in the formulation of regulations (e.g., General Guidelines for
Evaluation, PAE) and in the performance of relevant cross-cutting tasks such as
dissemination events and training, among others.
When it comes to setting the budget, the SHCP, in coordination with line ministries,
decides the expenditure priorities and the allocation of resources. Based on SED’s M&E
information and the results obtained from implementing agencies in the preceding year,
the SHCP can propose improvements in programs, policies, or institutional settings.
Resource allocations can increase depending on performance and the implementation of
evaluation recommendations for program improvement. Likewise, programs can suffer
potential reductions or elimination altogether depending on M&E information and
government priorities.
36 The legislative and judiciary branches, as well as subnational governments, evaluate their performances
according to their own standards but they should also follow the federal budget and fiscal responsibility
mandates with regard to evaluation.
Presidents Office
Program implementation
Performance-Based
implementation
Federal Budget
Performance-Based Budgeting
National Development Plan
Performance-Based Planning
Performance Evaluation System –SED
Secretary of Hacienda
Monitoring
(During implementation) Evaluation
(Results focus)
Politicas - Programas
(Coneval, SHCP, Others) Institutions Performance
(SFP – PMG)
Accountability -Transparency
Performance
M&E working group
(SHCP, SFP, CONEVAL)
Presidents Office
Program implementation
Performance-Based
implementation
Federal Budget
Performance-Based Budgeting
National Development Plan
Performance-Based Planning
Performance Evaluation System –SED
Secretary of Hacienda
Monitoring
(During implementation) Evaluation
(Results focus)
Politicas - Programas
(Coneval, SHCP, Others) Institutions Performance
(SFP – PMG)
Accountability -Transparency
Performance
M&E working group
(SHCP, SFP, CONEVAL)
19
Once the budget is approved by Congress and the fiscal year begins, the SHCP monitors
the day-to-day execution of the budget (i.e., the way line ministries spend). However,
through the SED, the SHCP also monitors the achievement of performance objectives
and goals (i.e., the extent to which agencies attain their programs’ stated results).
It took Mexico at least 10 years and an enormous amount of work, carried out by many
different government agencies, to accomplish such a comprehensive M&E system. Full
and complete institutionalization of the SED has yet to be achieved, but there exist solid
foundations now. CONEVAL’s performance evaluations of programs are now being used
by the Ministry of Finance in the preparation and negotiation of the budget and the SED
has a lot of potential to become a systematic and recurrent source of information on
program performance.
4. MEXICO’S M&E SYSTEM: STRENGTHS AND MAIN CHALLENGES
Mexico has been a pioneer in Latin America, in particular with regard to the importance
placed on robust impact evaluation of social programs, as evidenced by the evaluations of
Progresa/Oportunidades. The country has also made significant progress in managing
public institutions and programs more efficiently. Governments have begun to move
away from the traditional forms of management. Instead of solely focusing on the
monitoring of inputs, there is a trend toward focusing on results and impacts.
Much of this progress was made possible by a series of M&E and performance budgeting
reforms which took place in 2005. Consolidation began in 2007 when the SED integrated
previously disparate but well-oriented M&E efforts. CONEVAL was a significant
development for institutionalizing the evaluation function within the public
administration, because it granted the system independence, credibility, and technical
capacities. CONEVAL paved the way for mainstreaming evaluation methodologies and
helped to foster broader performance management reforms. The council’s role as a key
champion of evaluation has also been unprecedented and its capacity to manage the
performance of more than 100 evaluations annually is impressive. In fact, the CONEVAL
model has become a benchmark for other developing countries undertaking M&E
reforms.
All in all, Mexico had set out strong institutional foundations for its M&E system.
Existing legislation had provided a modern framework and made the system operational.
The progressive assimilation of results-based practices, tools, and codes is taking place at
all levels of the public sector, with increasing use of the evaluation findings. Moreover,
the system relies heavily on a country-led, rather than a donor-led, strategy which
increases the likelihood of long-term sustainability. There still remain, however, a
number of important challenges to achieving full institutionalization of evaluation and
results-based management in Mexico. Some of those challenges are discussed below.
4.1 Managing Change
In a relatively short period of timebasically between 2005 and 2009—Mexico has
undergone fundamental changes in performance management. The list of institutional
reforms, new practices, roles, and agencies involved in the management of the Mexico’s
policy landscape is extensive (see annex 2). These changes were not adopted overnight
by officials and agencies. In fact, as is the case in other countries, the Mexican experience
indicates that many of these changes created confusion and even frustration in some
20
entities.37
Furthermore, substantial increases in the workload of a number of civil
servants, coupled with the long-term character of the reform—which also means slow
rewarding of benefits—has produced resistance at some levels.
A key challenge for the federal government will thus be overcoming such resistance by
ensuring the proper balance between the pacing and sequencing of the reforms. It will
also consist of an adequate adoption of, and adaptation to, the changes. International
experience indicates that Mexico will benefit greatly from a carefully planned, reform
management strategy.
Such a strategy could integrate actions taken at the highest level of government and
encourage managers and political officials to adopt a results orientation, while at the
same time creating conditions at lower levels that enable programs and officials to
respond to these changes. Some of these actions include: (i) strategically communicating
the overarching philosophy of change, (ii) maintaining high-level political commitment
both in government and Congress, (iii) strengthening the technical capacities of key
reform “champions” (e.g., SHCP, SFP), (iv) providing applied training and specific
advisory services to line ministries, and (v) developing incentive plans to reward good
performance.
Some of these actions have already been implemented or are currently taking place, but
their integration under a coherent and, above all, systematic and medium-term change
strategy could give organizations the capacities and the time to absorb the scope and
depth of the reforms. This is an important consideration, in particular, for reducing the
risks of “reform fatigue.”
4.2 Developing Evaluation Capacity
One challenge that deserves particular attention is the ongoing need for evaluation
capacity building. The good practice of widespread training, led by CONEVAL and
SHCP in 2007, should not be seen as a one-off investment but rather as the first step in a
long-term process for developing M&E capacities in government. There are still many
areas in which training is needed. If a performance orientation is to penetrate deeper into
the public administration, further capacities in planning, financial and performance
management, and in the supervision of evaluation methodologies will have to be
developed.
If a results “culture” is to be nurtured and sustained, Congressional units, control
agencies’ technical staff, the media, and key civil society and professional organizations
will need to acquire more knowledge of the scope, purpose, and potential use of
performance information and M&E tools. The participation of academic institutions in
such endeavors will be crucial. The role of the SHCP and CONEVAL, to ensure effective
partnerships and lead the way to meet this challenge, will be extremely important.
The sophisticated evaluation menu that Mexico has developed has, in turn, imposed a
challenge in ensuring that an adequate pool of qualified external evaluators is available to
complete evaluation work according to the standards set up by the SED and CONEVAL.
Though the number of contracted evaluations has grown rapidly, there are still a
relatively modest number of professional evaluators available in universities and
consulting firms to fill market needs. It is not uncommon to find that the same people
37 According to officials interviewed, different and conflicting messages received by line ministries have created
confusion in many cases.
21
have evaluated numerous programs over a number of years, sometimes even evaluating
the same program more than once. Now that evaluation demand is growing, a larger and
more qualified cadre of external evaluators is needed
An aspect closely related to these concerns is the wide variation in the quality of
evaluation work completed between 1999 and 2006. A number of calls to enhance the
quality assurance process of external evaluators have come from the executive
government, academia, and even Congress.38
CONEVAL has taken some initial steps to
regulate the quality of external evaluators by setting up a national registry of existing
professionals, but more will be needed, particularly under the SED, to address these
challenges.
In terms of expanding the supply of qualified evaluators, there is an important
opportunity for Mexico’s universities and other academic institutes to introduce more
training-related knowledge products, as well as to increase the supply of advisory
services for practitioners. CONEVAL and SHCP may consider involvement in this
agenda through accrediting university courses in the evaluation of public programs,
creating incentives for evaluations to include an explicit knowledge-transfer component
through, for example, peer-to-peer learning activities so that informal training of
practitioners takes place.39
4.3 Institutional Coordination
The performance management reforms introduced have created new roles for a number of
entities which require a host of new interactions (i) between the political and technical
levels of government agencies, and (ii) among these agencies, Congress, and the control
entities. Ensuring that M&E information effectively influences policy making and
strengthens accountability requires that these relationships be clearly defined.
However, the clear definition of roles, leadership, and authority has not necessarily been
accomplished or appropriately coordinated. In fact, the creation of new institutions, the
introduction of performance indicators, the performance evaluations, the implementation
of performance budgeting, and the increasing accountability practices are all new
activities that strain the capacities of existing institutional mechanisms leading to
challenges in terms of coordination and efficient use of resources by avoiding
duplication.
This is particularly true with respect to collaboration between the so-called, cross-cutting
agencies, globalizadoras (President’s Office, SHCP, SFP, and CONEVAL), and other
key stakeholders such as Congress and control agencies. Current regulations, for
example, have not yet established the specific roles and scope of some of these agencies
evaluation functions and, therefore, potential conflicts and duplications are starting to
emerge.40
In practice, the question of “who does or leads what?” has been managed in
relatively ad hoc ways, in order to keep the process going; there has, until now been an
implicit consensus and understanding of the division of labor. However, tensions could
increase over time, thus posing potential threats to the process.
38 Alcade 2008.
39 Something similar has been done in Colombia since 2000, and more recently in the European Union, with the
goal of training evaluators working in new member states.
40 One of these potential conflicts is the extent to which the SED will be conducting evaluations of social
programs and the role of CONEVAL in such a case.
22
A similar challenge relates to the roles, responsibilities, and mechanisms for coordination
at the sector level. The National Evaluation Guidelines and the wider SED initiative
introduced a host of new performance-oriented functions and processes, which go way
above and beyond the traditional responsibilities of the administrative units, and which
will require the cooperation of various units within the ministries (e.g., budgeting,
planning, evaluation, and intelligent technology departments). There is still a long way to
go in defining roles at these levels. For example, interviews conducted in 2008 showed
that the contracting of evaluators was still being done by both administrative units and the
programs themselves, and that the various processes connected to contracting were not
well defined, with the result that hiring evaluators is still, in many cases, an extremely
slow process.
That being said, it is important not to over-regulate how line ministries should organize their
evaluation work. At the same time, additional support could be given to line ministries as
they implement and absorb the new evaluation framework within their structures. Given that
the M&E system is now entering a period of consolidation, emphasis should be placed on
facilitating coordination within the line ministers and on empowering evaluation units in
relation to the cross-cutting agencies leading the SED. In this way, communication and
coordination between these different stakeholders can become more effective. One
opportunity exists in selecting leading sectors to showcase different models on what the basic
structure and capacities of evaluation units should look like, and even more important, on
how the lines of coordination between these and budget and planning units could work.
Making the M&E system operational at the sector level is a key institutionalization challenge
ahead.
4.4 Strengthening the Role of Planning
Strategic planning is a key pillar of the Mexican performance-based management model,
because it should provide both strategic direction to overall spending and results
frameworks (benchmarks) at different levels of the administration. It is primarily against
this framework and that of the budget that the whole-of-government performance can be
measured. The role of planning has been significantly bolstered by the initiatives
implemented between 2006 and 2007. The introduction of goals and quantifiable
indicators to the NDP, the requirement that line ministries align their sector plans with
the NDP, and the development of MIs at the program level have all contributed to
enhancing the role of performance based planning in Mexico.
Nevertheless there is still a fundamental challenge regarding planning namely, its
effective harmonization with budgeting processes. At national and sector levels these
processes currently take place separately, undertaken by different units for whom
coordination of work is made it difficult by the fact that, for example, the two processes
follow different calendars. Meeting this challenge will require a revalidation of planning
as a managerial tool at sectors and agencies so as to change the perception that the
planning function is a procedural, compliance-related activity, with relatively low value
added to the public management processes. In addition, the technical and operational
elements associated with harmonizing planning and budgeting will also require
investments.
Another area where room exists to strengthen the links between the government’s
performance and planning is the systematic evaluation of the current national and sector
planning exercises. At the moment, evaluation tools are mainly focused at the program
level and no assessment methodology for plans exists. Evaluating plans will create
23
incentives for the agencies to undertake these processes in a more managerial and less
procedural manner. The challenge for SHCP, CONEVAL, and SFP is creating the
incentives and the tools for reinvigorating the planning function, and conveying its
crucial role for performance-based management and budgeting.
4.5 Strengthening the Role of Monitoring
The Mexican model has put a premium on evaluations conducted with a high level of
sophistication. However, a standardized, systematic, and consolidated monitoring
function has yet to be developed.
A well-developed monitoring function is essential to effective planning and budgeting,
because government performance is strongly linked to services and products that need to
be managed on a daily basis (normally using administrative records produced by the
programs). This rather contrasts with the evaluation function which focuses more heavily
on outcomes and impacts, and relies more strongly on statistics and longer-term, in-depth
analyses.41
As such, a core challenge for the Mexican M&E system is to reach a proper balance
between these two different activities. Part of this challenge is ensuring that both
activities are conducted in a technically sound and complementary manner. Monitoring
tools should facilitate timely assessment of key performance parameters, allowing
managers to deal with problems before they become critical. An additional, often
overlooked, benefit is that monitoring data can be collected at relatively low cost and the
data can be later used to conduct more extensive and systematic evaluations. The
adequate informing of budgeting processes—normally conducted under rigid time and
operational constraints—strongly requires monitoring.
The SED should aim to fill the monitoring gap. However, it still faces the challenge of
setting a framework with sound technical and institutional rules to cover the whole
management cycle. This would involve producing monitoring information all the way up
and down the implementation chain, from inputs and activities at the bottom, to products,
outcomes, and impacts at the top. As previously noted in discussion of other aspects of
the system, collaboration among CONEVAL, SHCP, and SFP could have a significant
pay-off for ensuring success in this effort.
4.6 Strengthening the Quality of Performance Information
Having a dependable information infrastructure that supports performance-based management
is extremely important. This infrastructure is the single, most important foundation for the
availability and quality of the government’s M&E work. A rapid review of current practices in
line ministries shows, among other weaknesses, lack of standardization in quality protocols for
managing data, disparate information on programs’ beneficiaries, limited harmonization of
surveys and thematic coverage for priority program evaluations, and limited capacities to
manage information by management staff (see table 2).
41 This may, in turn, require surveys and/or more sophisticated estimations.
24
Table 2. Challenges to Attaining Results Information in Line Ministries
Selected Challenges Identified
Results information:
Little focus on collecting results information, emphasis given to financial and administrative
data.
Registries of social programsbeneficiaries vary in quality; some do not track beneficiaries
but intermediaries instead.
National Information Institute supply is not necessarily aligned with performance information
needs; low advantage taken of potential survey harmonization and thematic coverage for
priority program evaluations.
Databases:
Variables identified by different names lead to confusion and difficulties for aggregating
information.
Databases are not interconnected and are still not easily available to users.
Management of information:
No standardized quality protocols for managing data; each program uses different practices,
resulting in varying quality.
IT systems lacking ability to operate together.
Lack of funding in programs to strengthen information collection.
Varying capacities of information management staff.
If decision makers perceive that bad information is feeding into the system, its short-term
credibility and longer-term sustainability can be seriously damaged. Action to enhance
the quality of performance information is, therefore, an important challenge for Mexico.
CONEVAL’s decision to establish a formal partnership with the National Institute of
Geography and Information (INEGI) should increase the performance information
collected through surveys, thereby providing more data to strengthen poverty
measurement and evaluation work. Although this is a good step forward, the challenge
lies in expanding this effort to cover the entire SED and in extending collaborative work
to the other statistical institutes in Mexico.
25
5. CONCLUSIONS AND LESSONS LEARNED
This paper has reviewed the steps taken by Mexico’s federal government to create a
countrywide M&E system, as well as the principal achievements and challenges on the
path toward institutionalizing the system.42
Full institutionalization has not been
accomplished yet but there are ongoing initiatives on several fronts to strengthen, expand,
and consolidate the depth and scope of the M&E reforms in the public sector. Although
the process is not yet complete, Mexico has achieved definite successes in the last decade
and, particularly, since 2005 when in-depth performance management reforms started.
The system evolved from a sectorwide, mainly accountability-based endeavor, to a
national-level, centrally driven initiative closely linked to the annual budget process. The
principal champion of the system, owing to its main area of influence, is SHCP, with the
powerful support of SFP and the technical advice of CONEVAL. Each of these agencies
is in charge of operating key components of the system. The political support of the
President’s Office has also played a part due to its increasing support and encouragement
of performance management reforms in progress. The system’s rapid implementation has
also been the result of the combination of these three agencies’ influence, power, and
technical capacities.
The involvement of the Ministry of Finance has been central. Integrating, under one
coherent framework, previous unrelated performance evaluation efforts, required
considerable authority over other ministries and close dealings with the President’s
Office. This allowed the Ministry of Finance to combat problems, such as the lack of
standardization in program planning and evaluation frameworks, and the variations in
technical quality of the evaluation methods across public agencies. The SHCP’s
championing role also ensured the establishment of further links between the M&E work
and the budget process. These have led, in turn, to greater acceptance of evaluation, in
general, and to an increase in opportunities for line ministries to use the system’s
performance information. These have been two of the most important factors in ensuring
the progress of performance management reforms in Mexico.
The existence of a technically strong and independent specialized agency in charge of
coordinating evaluative work was paramount in the process. The model set by the
National Council of Evaluation for Social Policy (CONEVAL), which brings academia’s
objective perspectives and strengths, provided the basis to increase independence,
technical robustness, and, hence, credibility of the evaluative work. CONEVAL’s
participation as technical leader, trainer, and standard setter of the system offers an
interesting example of the potential benefits of not relying exclusively on the power of
the Ministry of Finance to enhance the commitment of line ministries. Persuasiveness,
based on technical grounds, and the provision of a range of positive support and
assistance can also be very powerful. In the case of Mexico, the participation of
CONEVAL proved to be a necessary condition—rather than a complementfor
successful implementation of the system.
This combination of implementation responsibilities is not always possible and,
whenever feasible, is definitely more challenging due to higher coordination
requirements. Most advanced M&E systems have chosen to put exclusive
implementation responsibility in the central budgeting agencies (for example, in Chile,
42 General references to the institutionalization of M&E in government often refer to the existence of a
sustainable, well-functioning system, in which good-quality performance information is produced and used
intensively. See Mackay 2007, IEG 2008.
26
Australia, and New Zealand) or have relied more heavily on evaluation capacity in the
line ministries (such as in Canada and the Netherlands) or in planning ministries (for
example, as in Colombia). In Mexico’s experience, having additional agencies, such as
CONEVAL, SFP, and the President’s Office, also championing the system, provided
synergy and enhanced the reform momentum.
Important lessons in a number of areas can be draw from Mexico’s experience in
implementing a governmentwide M&E system. These lessons are related to the
successful integration of different pieces of M&E work, the obstacles encountered in the
process, and the solutions advanced. It is hoped that these lessons will be useful to other
countries endeavoring to strengthen or to build a whole-of-government M&E system.
Some of these lessons include the following:
Adopting an opportunistic approach toward implementation was effective. Top-
down state or budget reforms often offer unique opportunities for introducing,
strengthening, or broadening the scope of the M&E practices in government. Capitalizing
on the reform momentum may require the forging of partnerships even if the lines of their
responsibilities are not entirely clear. In Mexico, even when roles seemed to overlap and
tensions among agencies emerged, interested government agencies worked to keep the
process moving forward, without stopping to formally define their respective functions.
Caution should nevertheless be taken not to leave too many roles and responsibilities of
key stakeholders without clear definition because in time, the costs could exceed the
benefits. Agreements can be useful for creating frameworks and for advancing the initial
implementation. However, long-lasting and effective operation in advanced stages (e.g.,
for ensuring utilization) can be difficult, and the risks of implementation obstructions can
be high if the functions among key actors are not clearly defined. International experience
indicates that whenever possible, making those functions clear from the start offers
substantial benefits in the long run.
The introduction of M&E practices and tools needs to respond both to political and
technical requirements. Evaluation in Mexico was an answer to, and a consequence of,
the political transition to democracy, specifically, to the increasing calls for
accountability in government. But it was also a technically planned effort in response to
Congress’ political decisions to demand performance evaluations of all government
programs. The bottom line, in this particular respect, is that the implementation of M&E
reforms should not simply be seen as a technical issue, to be solved by the executive.
Congress and the political context play a crucial role if an M&E reform is to be truly
internalized by the whole government. Unfortunately, there are no formulas to ensure that
these two requirements go hand–in-hand.
The risks of overly engineering the M&E systems, both in quantity and quality of
the methodologies and information they produce, are present in many countries. In
Mexico, the law mandates and enforces annual evaluation of all government programs,
but the capacity to accomplish such mandates—both at the government level and in the
supply of evaluatorsdoes not yet exist. CONEVAL has learned from this experience
and is adjusting its Annual Program of Evaluation to identify the sequence and types of
evaluation that need to be implemented in the years to come. This is an important step
since excessive development of evaluation methods and production of performance
information might simultaneously weaken the demand for results information itself.
27
Institutional arrangements to foster the demand for evaluation might be useful.
The Mexican case shows that proactive steps are needed to integrate performance
information in decision making and accountability processes, in order to enhance
coordination and ensure appropriate dissemination of performance information. In
countries with a legal tradition, these arrangements are normally supported by laws and
permanent regulations, while in countries with a more embedded managerial tradition,
the use of performance information will depend more on largely accepted practices.
Mexico’s momentum for the institutionalization of M&E has been effectively seized
by key government champions. The development of the system has helped to move the
modernization of government forward, bringing a large array of improvements, which
affected accountability and transparency mechanisms, the planning and budgeting
process, and the professionalization of the public service. For Mexico, the
institutionalization of M&E, in parallel with the opening up of the political system at the
beginning of this decade, has contributed toward helping the country to overcome a
legacy of intense mistrust in government.
In conclusion, the achievements of the M&E system in Mexico are impressive but still
incomplete. Planning, measuring, and reporting government results have been
widespread, and capacitiesparticularly in the federal government—have improved.
However, actual use of the results—the clearest and most potent sign of successful
institutionalizationis still a challenge, though it is greater today than it was several
years ago. The risks of information overload and agency fatigue due to increasing
demands on line ministries, and the need for further coordination, are still present. The
solid foundation already built will probably result in greater use of the system, but
expectations should be managed. Even M&E systems in the most developed countries
still face many of the same challenges.
28
Bibliography
Alcade, M. 2008. Presentation at the Internal Conference on Performance-Based
Budgeting, Mexico City, June 9–10.
Arellano, D., and J.P. Amparan. 2000. Stalled Administrative Reforms of the Mexican
State. CIDE.
Burgos, Ricardo Miranda. 2008. “Presupuesto Basado en Resultados y Sistema de
Evaluación del Desempeño.” Presentation at the Indicators Matrix workshop. Mexico
City, October 29.
Cardozo, Myriam. 2006. La evaluación de políticas y programas públicos: el caso de los
programas de desarrollo social en México. Cámara de Diputados. eds., Miguel Ángel
Porrúa.
Comisión para el Estudio y Creación, Agencia Estatal de Evaluación de los Servicios y
de las Políticas Publicas. 2004. Informe, 4 Octubre 2004, Ministerio de Administraciones
Publicas, Espana.
Feinstein, O., and G. Hernández Licona. 2008. “The Role of Evaluation in Mexico:
Achievements, Challenges, and Opportunities,” Quality of Public Expenditure, NOTE
No. 2, World Bank, June.
Garde Roca, J.A. 2007. “La Creación de la Agencia de Evaluación en España, Evaluación
de Políticas Públicas, Información Comercial Española,Mayo-Junio 2007. N.836
Hernández Licona, G. 2007a. “Evaluación de la Política Social: Un Cambio Institucional
en México.” Latin American Management for Development Center (CLAD).
———. 2007b. “Monitoring and Evaluation System in Mexico,Presentation delivered
at the World Bank, Series on M&E Case Studies in Latin America, Poverty and Gender
Unit, World Bank, June.
———. 2008. Paper presented at the 13th International Congress on State Reform and
the Public Administration, Latin American Management for Development Center
(CLAD), November.
IEG (Independent Evaluation Group). 1983. “Mexico – Integrated Rural Development
Project (PIDER I),” Loan 1110-ME. World Bank.
http://web.worldbank.org/external/projects/main?pagePK=64283627&piPK=73230&the
SitePK=40941&menuPK=228424&Projectid=P007535.
Instituto Nacional para la Evaluación de la Educación (INEE). 2003. “La calidad de la
educación básica en México.” Primer informe anual 2003. México, D.F.: INEE.
Levy, Santiago. 2007. “Progress Against Poverty: Sustaining Mexico's
Progresa/Oportunidades Program.Presentation at conference on The Global Economy and
Development, joint event of the Wolfensohn Center for Development and the Latin America
Initiative, Brookings Institution, January.
29
Mackay, K. 2007. How to Build M&E Systems to Support Better Government,
Independent Evaluation Group, World Bank.
Oracle Customer Snapshot. 2008. “CONEVAL Unifies Information and Improves
Evaluation and Monitoring of Social Programs,” April.
Parker, Susan. 2003. “Case Study: The Oportunidades Program in Mexico.” Shanghai
Poverty Conference Scaling Up Poverty Reduction. CIDE - Mexico
Perez-Jacome, Dionisio. 2008. “The Mexican Performance Budgeting System,” paper
presented at the Performance Budgeting Conference, in Mexico City, Mexico, June 9–10.
Phillip, G. 2003. “Ideas and Policy Transfer: the Mexican Civil Service Reform,” Future
Government Paper No. 13, July.
World Bank. 2004. Poverty in Mexico: An Assessment of Trends, Conditions and
Government Strategy. Washington, DC: World Bank.
———. 1999. Performance Monitoring Indicators: A Handbook for Task Managers.
Washington, DC: World Bank.
Zajda, J. 2006. Decentralization and Privatization in Education: The Role of the State.
Springer.
30
ANNEX 1 Chronology of Key Initiatives and Events
DISPARATE EFFORTS FOR M&E
1970s,
1980s,
early
1990s
Pioneering program evaluations are conducted with the support of multilateral agencies, e.g., Public Investment
Program for Rural Development (PIDER)
Introduction of “Integral Audits” and concepts of performance measurement by the new Federal Audit Institution
(Contraloria)
National Planning System and Planning Law require financial and physical indicators to be presented to Congress.
1995
Evaluation System (SED) is envisioned as part of a budget reform.
The evaluations of the Social Milk Supply Program (Liconsa), the Rural Supply program (Diconsa), the National
Solidarity Program (PRONASOL) and several job training programs are conducted.
1996
The Program for the Modernization of the Public Administration (PROMAP), introduces performance indicators,
internal evaluations and the concepts “accountability” “performance management” and “performance budgeting” to
the Federal Administration.
ESTABLISHING A SYSTEMIC ROLE FOR EVALUATION
1997
Influential evaluation of Progresa/Oportunidades program starts.
1998
Congress requires federal programs providing subsidies to prepare rules of operation (ROP) with standard
information on design, objectives, performance indicators, beneficiaries, and operation.
1999
Congress mandates all programs with ROP to be evaluated annually by external evaluators.
2000
Results of the evaluation of Progresa/Oportunidades are published.
Technical Committee for Poverty Measurement (CTMP) is set up to prepare an official methodology to measure
poverty reduction.
Implementation of a Presidential Goals System.
2001
Creation of SEDESOL’s Under Secretary for Planning, Prospect, and Evaluation.
Creation of the new Federal Superior Auditing Office (ASF)
2002
Creation of the National Institute for the Evaluation of Education.
The Federal Transparency and Public Information Access Law is passed, reinforcing evaluation’s role and its
public dissemination.
2003
The Public Administration Secretariat (SFP) is created, with responsibilities in performance auditing.
PROGRESS TOWARD A GOVERNMENTWIDE M&E SYSTEM
2004
The Social Development Law is passed, introducing monitoring and evaluation of social programs and creating the
National Evaluation Council (CONEVAL).
2005
CONEVAL’s rules of operation are enacted and the council begins operations.
2006
The Federal Law of Budget and Fiscal Responsibility is enacted, making operational the links between strategic
planning and the budget process, and creating the Performance Evaluation System (SED).
2007
General Guidelines for the Evaluation of Federal Programs are set up to establish results frameworks, procedures,
standard methods, and coordination mechanisms.
CONEVAL conducts widespread training in logic frameworks for all program officials.
The Annual Budget Decree assigns specific resources for program evaluations.
2008
The General Guidelines for the Programming and Budgeting Processes establish operational procedures and rules
for the implementation of performance budgeting and the Performance Evaluation System (SED).
SED begins formal operations.
The Annual Budget Decree introduces the National Annual Evaluation Plan.
31
ANNEX 2 Annual Evaluation Program
Source: Mexico’s Ministry of Finance and Public Credit, Budgetary Policy and Control Unit, October 2008.
Type of
Evaluation
Scope of Application
Number of Evaluations
2007
2008
2009
Total 126 192 196
Consistency
and Results
Programs that deliver goods and services and
receive subsidies
103
34
22
Process
Programs that deliver subsidies under the
rules of operation 6
Impact
Programs that deliver subsidies
2
6
6
Strategic
Policies
2
1
7
Nutrition and supply
x
x
Rural development
x
x
Health
x
Social security
x
Microcredit
x
Inclusion of a gender perspective in
the housing, health, and labor policies;
and in productive projects
x
Transfers to sub-national levels
(Item 33)
x
x
x
Transfers to sub-national levels
(Item 33)
x
Specific
Particular aspects of the programs
6
151
155
Public policy monitoring
x
Quality of services
x
Performance of programs that provide
subsidies
x
Gender perspective
x
x
Perception of the beneficiaries of the
program rural supply
x
Performance of programs with matrix
indicators
x
Perception of the beneficiaries of
programs under the rules of operation
x
Metropolitan and regional funds x
32
Other Titles in the ECD Working Paper Series
#1: Lessons from National Experience, by Keith Mackay, 1998.
#2: Zimbabwe: Issues and Opportunities, by Stephen Brushett, 1998.
#3: Indonesia’s National Evaluation System, by Alain Barberie, 1998.
#4: The Development of Australia’s Evaluation System, by Keith Mackay, 1998.
#5: Comparative Insights from Colombia, China and Indonesia, by R. Pablo Guerrero O, 1999.
#6: Evaluation Capacity Development: A Diagnostic Guide and Action Framework, by Keith
Mackay, 1999.
#7: Sub-Saharan Africa: Lessons from Experience in Supporting Sound Governance, by Mark
Schacter, 2000.
#8: Strengthening Capacity for Monitoring and Evaluation in Uganda: A Results Based
Management Perspective, by Arild Hauge, 2001.
#9: Guide to Conducting Reviews of Organizations Supplying M&E Training, by Marie-Hélène
Adrien, 2003.
#10: The Development of Monitoring and Evaluation Capacities to Improve Government
Performance in Uganda, by Arild Hauge, 2003.
#11: Two Generations of Performance Evaluation and Management System in Australia, by
Keith Mackay, 2004.
#12: An Assessment of the Impact of Bangalore Citizen Report Cards on the Performance of
Public Agencies, by Adikeshavalu Ravindra, 2004.
#13: Building Country Capacity for Monitoring and Evaluation in the Public Sector: Selected
Lessons of International Experience, by Salvatore Schiavo-Campo, 2005.
#14: Evaluation Capacity Development in the Republic of Ireland, by Richard Boyle, 2005.
#15: Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector
Management, by Keith Mackay, 2006.
#16: Experience with Institutionalizing Monitoring and Evaluation Systems In Five Latin
American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay, by Ariel
Zaltsman, 2006.
#17: A Diagnosis of Colombia’s National M&E System, SINERGIA, by Keith MacKay, 2007.
#18: Insider Insights: Building a Results-Based Management and Evaluation System in
Colombia, by Manuel Fernando Castro, revised 2009.
... One well-regarded example of this is Mexico's 2004 Social Development Law, which requires new social development policies and interventions to be formally monitored and evaluated. A World Bank report described this law as representing the 'institutionalisation of evaluation' (Castro et al. 2009), establishing an expectation of good practice to generate evidence on the effects of social interventions, with the law further establishing an autonomous National Council for the Evaluation of Social Development Policy (CONEVAL) to undertake such evaluations (Lopez 2012). Another more common example of government legislation that may address sources of bias comes in the form of freedom of information legislation. ...
... One well-regarded example of this is Mexico's 2004 Social Development Law, which requires new social development policies and interventions to be formally monitored and evaluated. A World Bank report described this law as representing the 'institutionalisation of evaluation' (Castro et al. 2009), establishing an expectation of good practice to generate evidence on the effects of social interventions, with the law further establishing an autonomous National Council for the Evaluation of Social Development Policy (CONEVAL) to undertake such evaluations (Lopez 2012). Another more common example of government legislation that may address sources of bias comes in the form of freedom of information legislation. ...
Book
Full-text available
Book on the politics and good governance of evidence. Open (free) access as e-book ***If the full text does not display below*** (file size is large and may be slow loading) Please click the pull down arrow next to the 'SHARE PUBLICATION' box above to access a pdf version (select 'full text sources') Or you can download it from here (copy and paste link): http://www.tandfebooks.com/action/showBook?doi=10.4324/9781315675008
... This task began in the middle of 1970 and took root as a government practice at the end of the 1990s. The following elements highlight the advances of the policy based on results in Mexico: the creation in 2005 of the National Council of Evaluation of Social Politics (CONEVAL, its acronym in Spanish); the evaluation of the impact of the Progresa-Oportunidades program; and the establishment o f norms that define the framework for the development of program evaluation and monitoring (Castro et al. 2009). The advances in evaluation provide a uniform framework that serves as a basis for all programs. ...
Article
Full-text available
This article reports on the methodology for setting the Mexican User Satisfaction Index for Social Programs (MUSI-SP) as tested in seven national social programs. The evaluation is based on Structural Equation Modeling (SEM). How satisfaction takes the central place of the SEM, which postulates its causes and effects, contributes to the increased validity and reliability of satisfaction indicators that allow benchmarking between social programs. The MUSI model is an adaptation of the American Customer Satisfaction Index (ACSI) model. The MUSI methodology includes qualitative and quantitative techniques. The estimation model is by the Partial Least Squares (PLS) . In each of the seven social programs, no statistical evidence was found to reject the main relationships postulated by the ACSI’s model: that Perceived Quality impacts Satisfaction, and Satisfaction impacts Trust. The improvement opportunity areas were also identified for each program. These results give valid and reliable feedback to public policies.
... Governments and advocacy groups display a growing appetite for evidence-based policy-making. In 2005, Mexico established an independent government agency to rigorously evaluate social programs, and in 2012, the U.S. Offi ce of Management and Budget advised federal agencies to present evidence from randomized program evaluations in budget requests ( 1,2). ...
Article
Full-text available
There is growing appreciation for the advantages of experimentation in the social sciences. Policy-relevant claims that in the past were backed by theoretical arguments and inconclusive correlations are now being investigated using more credible methods. Changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade. When experimentation is difficult or impossible, researchers are using quasi-experimental designs. Governments and advocacy groups display a growing appetite for evidence-based policy-making. In 2005, Mexico established an independent government agency to rigorously evaluate social programs, and in 2012, the U.S. Office of Management and Budget advised federal agencies to present evidence from randomized program evaluations in budget requests (1, 2).
Article
Full-text available
This article proposes the development of a value-based perspective of evaluation in order to improve opinions on the results of public action in highly complex contexts of governmental implementation. The discussion is centered on the scope and difficulties of using the Logical Framework Approach and indicator matrices, based on a review of forty evaluations of federal educational programs. Two perspectives of study are viewed: the levels of analysis of the Fischer policies, and a focus on complex systems based on Morell and Williams and Hummelbrunner. The second conception permits emphasizing the relevance of the agents involved in a program or policy, operation in changing contexts, and the influence of measures foreign to the methodology¿s criteria.
Article
Full-text available
Transition over the last two decades to more open politics and markets generated pressures for administrative change. Administrative reform in Mexico made little headway in the 1990s. Arellano and Guerrero argue that the key obstacle to bureaucratic modernization (particularly of the Weberian variety) was the close historical fusion between political and the administrative elites. Mexican reformers opted for a superficial managerial reform as a way to improve efficiency without upsetting one of the cornerstones of the PRI-Dominated regime. The chapter provides an overview of Mexico's public administration, a thorough survey of recent reform initiatives, and a detailed case study of the failed attempt to create a career civil service in the late 1990s. The general guidelines of the Zedillo government for administrative reform were: citizen participation and service, administrative decentralization, evaluation and measurement of public management, and professionalization and ethics (this is civil service component). The chapter offers three alternative Hypotheses: 1) the integration of party and bureaucracy thwarts reform: 2) successful reform requires horizontal accountability; and 3) if the real rules of bureaucracy are informal than merely formal reforms will fail.
Article
Full-text available
Obra que analiza la utilización, alcance, difusión, relación costos/calidad de los resultados de los métodos y técnicas aplicadas en México para evaluar los programas sociales gubernamentales. El objetivo final es ofrecer una metodología que permita verificar que los objetivos de las políticas sociales se traduzcan en acciones y que éstas se cumplan cabalmente.
Book
Decentralisation and Privatisation in Education explores the ambivalent and problematic relationship between the State, privatisation, and decentralisation in education globally. Using a number of diverse paradigms, ranging from critical theory to globalisation, the authors, by focusing on privatisation, marketisation and decentralisation, will attempt to examine critically both the reasons and outcomes of education reforms, policy change and transformation and provide a more informed critique on the Western-driven models of accountability, quality and school effectiveness. We want to demonstrate that claims of advantages in ‘efficiency’ brought about by privatisation in education are not always supported empirically as proposed by proponents. The book examines the overall interplay between privatisation, decentralisation and the role of the state. The authors draw upon recent studies in the areas of decentralisation, privatisation and the role of the state in education. By referring to Bourdieu’s call for critical policy analysts to engage in a ‘critical sociology’ of their own contexts of practice, and poststructuralist and postmodernist pedagogy, this collection of book chapters demonstrate how central discourses surrounding the debate of privatisation, decentralisation and the role of the state are formed in the contexts of dominant ideology, power, and culturally and historically derived perceptions and practices. The authors discuss the newly constructed and re-invented imperatives of privatisation, decentralisation and marketisation and show how they may well be operating as an educational model of a new global ‘master narrative’— playing a hegemonic role within the framework of economic, political and cultural hybrids of globalization.
Article
Este artículo explica, en primer lugar, el desarrollo político y gerencial en el que se inserta la creación de la Agencia Estatal de Evaluación de las Políticas Públicas y de Calidad de los Servicios como un proceso de formación de una voluntad colectiva en torno a una finalidad compartida de buen gobierno. En segundo lugar, hace referencia al alcance y enfoque de la función de evaluación que cabe prever en la práctica. Finalmente, pasa revista a las principales características de la Agencia subrayando sus principios de actuación, la composición de sus órganos de gobierno y la importancia de su contrato de gestión.
Presentation at the Internal Conference on Performance-Based Budgeting
  • M Alcade
Alcade, M. 2008. Presentation at the Internal Conference on Performance-Based Budgeting, Mexico City, June 9-10.
Presupuesto Basado en Resultados y Sistema de Evaluación del Desempeño
  • Ricardo Burgos
  • Miranda
Burgos, Ricardo Miranda. 2008. "Presupuesto Basado en Resultados y Sistema de Evaluación del Desempeño." Presentation at the Indicators Matrix workshop. Mexico City, October 29.
The Role of Evaluation in Mexico: Achievements, Challenges, and Opportunities
  • O Feinstein
  • G Hernández Licona
Feinstein, O., and G. Hernández Licona. 2008. "The Role of Evaluation in Mexico: Achievements, Challenges, and Opportunities," Quality of Public Expenditure, NOTE No. 2, World Bank, June.
Evaluación de la Política Social: Un Cambio Institucional en México
  • G Hernández Licona
Hernández Licona, G. 2007a. "Evaluación de la Política Social: Un Cambio Institucional en México." Latin American Management for Development Center (CLAD).