ResearchPDF Available

VNR reporting needs evaluation: a call for global guidance and national action

Authors:

Abstract

This briefing analyses 43 ‘Voluntary National Reviews’ (VNRs) of progress towards the Sustainable Development Goals (SDGs). Extending a 2016 analysis, it finds that monitoring is strong but evaluation systems and processes often remain missing or misunderstood. Sporadic good practice is emerging, such as: linked National Evaluation Policy and action planning (Nepal); recognition of the SDGs’ complexity when considering evaluation (Czech Republic); learning through evaluation (Ethiopia and Kenya); and drawing on findings from past evaluations (Belize). Countries still to submit their first VNR could build on these examples. We also recommend action for the UN Secretary General, the UN Evaluation Group, national governments, international organisations and professional evaluators to jointly address the significant gaps.
Policy and planning
Keywords:
Sustainable Development Goals
(SDGs), monitoring and evaluation
(M&E)
Brieng
Policy
pointers
The current Voluntary
National Review
guidelines should be
revised by the UN
Secretariat and General
Assembly to clearly outline
evaluation measures and
review processes. These
gaps should be filled in
consultation with the UN
Evaluation Group (UNEG)
and the broader evaluation
community.
National governments
should engage with
evaluation networks and
professionals early in the
Voluntary National Review
process. They should also
consider developing and
harmonising national
evaluation policies.
National evaluation
organisations should
initiate dialogues with their
governments on
evaluation’s role, and
engage with institutions in
charge of defining
evaluation processes.
There is a need to
strengthen evaluation
capacity and use in all
countries; this requires
courageous political will,
adequate resources and
evaluation expertise.
VNR reporting needs
evaluation: a call for global
guidance and national action
This briefing analyses 43 ‘Voluntary National Reviews’ (VNRs) of progress
towards the Sustainable Development Goals (SDGs). Extending a 2016
analysis, it finds that monitoring is strong but evaluation systems and processes
often remain missing or misunderstood. Sporadic good practice is emerging,
such as: linked National Evaluation Policy and action planning (Nepal);
recognition of the SDGs’ complexity when considering evaluation (Czech
Republic); learning through evaluation (Ethiopia and Kenya); and drawing on
findings from past evaluations (Belize). Countries still to submit their first VNR
could build on these examples. We also recommend action for the UN Secretary
General, the UN Evaluation Group, national governments, international
organisations and professional evaluators to jointly address the significant gaps.
Countries are beginning to document their
progress towards the Sustainable Development
Goals (SDGs) through regular Voluntary National
Reviews (VNRs) submitted to the UN’s High-Level
Political Forum on Sustainable Development.
The 2030 Agenda for Sustainable Development
sets clear expectations for VNRs to be “rigorous
and based on evidence”1, and the UN
recommends that the first VNR in particular
should describe the reviewing process and how
the ‘national follow-up and reporting’ system will
be implemented.2
Evaluation enhances monitoring’s meaning and
depth by addressing complexity in how the SDGs
are best achieved:3 so each VNR should include
up-to-date evaluation findings and an assessment
of progress on national evaluation policies and
systems.2 An earlier briefing reviewed the
22 VNRs submitted in 2016,4 and found that
evaluation was mostly overlooked.5 In this briefing,
we extend our analysis to the 43 VNRs from
2017, 6 highlighting similar oversights but also
some emerging good practice (Figure 1).
Formats and content. The 43 reviews vary
hugely. The shortest, from the Maldives, is
28 pages while the longest (Uruguay) reaches
386. Generally, longer reviews include more
statistical data. Some countries with short reviews
just refer to their existing statistical reporting
system. Countries with well-established
sustainable development monitoring systems
from the 1992–2012 ‘Rio process’7 have notably
shorter reports, whereas countries building on this
approach as part of their roadmap to the SDGs
have longer documents.
Understanding how monitoring differs from
evaluation. All reports lean heavily towards
monitoring indicators when describing monitoring
and evaluation (M&E) systems. This implies that
the SDGs’ review and follow-up process will be
Issue date
January 2018
Download the pdf at http://pubs.iied.org/17446IIED
IIED Briefing
data-rich, but weaker on integrated analysis.
24 reports mentioned evaluation a few times but
confused it with monitoring. 13 reports did show
sufficient understanding of evaluation. Six didn’t
mention evaluation at all.
Gaps are particularly
notable in VNRs from
European and Asian
countries. However, Nepal,
Azerbaijan, Indonesia, Cyprus, Italy, Denmark and
Portugal are exceptions to this trend. Reviews
from Latin America and Africa dedicate
comparatively more effort and reflection to
evaluation, and many show good understanding.
Evaluation’s role is vague. Most VNRs (even
those from Latin America and Africa) do not
entirely capture how evaluation can help report
and guide SDG implementation. Only three
countries assign a clear role to evaluation: Nepal,
Honduras and Guatemala. These describe
evaluation as a policy review mechanism. However,
Costa Rica, Guatemala, Honduras, Kenya, Nepal,
Panama and Uruguay report plans to develop
National Evaluation Policies. Since the proposal of
‘voluntary common reporting guidelines for VNRs’
did not account for evaluation’s role in national
follow up and review, it is likely that it will also be
unclear in forthcoming reports, unless new
guidance is produced.8
Existing monitoring systems provide
opportunities. All 43 countries will embed SDG
reporting into their existing statistical
infrastructure. Almost all countries (41 out of 43)
have conducted, or are planning, a gap analysis to
assess how existing monitoring systems should
be adapted to integrate SDG reporting. While
many European countries refer to systems used
since the 1990s, most African, Asian and Latin
American countries refer to monitoring systems
designed between 2000 and 2015, primarily for
the Millennium Development Goals (MDGs).
Integrating new indicators into current systems
will require substantial commitment, effort and
resources. Many VNRs from Africa and Latin
America explicitly discuss the challenges of
monitoring the SDGs, and many countries have
made strong efforts to engage numerous local
and national institutions. For example, El Salvador
has held a statistical workshop to define and
validate indicators and targets for each SDG,
involving 58 governmental institutions. This
increased focus on monitoring systems will
provide opportunities for a demand of high quality
evaluations that feed into ongoing planning and
prioritisation for SDGs implementation.
Few qualitative indicators. Most monitoring
systems lack qualitative analysis. The dominance
of quantitative indicators may be partially due to:
Little clarity in existing guidelines about how
qualitative analysis can support country
reporting on the SDGs,9 and
The existence of specific targets on statistical
capacities for SDG17.
Also, in the longer reports, national authorities tend
to overlook the available statistical information for
SDG indicators and instead emphasise new
quantitative, national, indicator-based reporting.
References to qualitative evaluation approaches
are generally missing, despite their utility when
monitoring data is hard to access. Nevertheless,
there are some notable exceptions. Azerbaijan,
Benin, Costa Rica, Cyprus, Czech Republic,
Portugal, Qatar and Sweden all stress the
importance of using qualitative data and analysis.
Governance system. Almost all countries have
developed a high-level governance structure for
the 2030 Agenda and SDG reporting. This is a
promising development that shows national
prioritisation and country ownership of the 2030
Agenda. 26 reports mention establishing
coordinating bodies tasked with overseeing
progress towards the SDGs. These are usually
committees, commissions or councils coordinating
different ministries. Some are open to wider
participation, for example including representatives
from civil society and the private sector.
Responsibility for M&E usually mirrors the national
SDG governance structure, ie the main
coordinating body is also responsible for
monitoring, evaluation and reporting. In
14 countries, M&E responsibilities for the SDGs lie
with the president/prime minister’s office. In
others, overall responsibilities sit within one
specific ministry, including those with mandates for
planning, environment, foreign affairs, trade and
economy, devolution, federal planning and finance.
In some cases, new coordinating bodies have been
given responsibility for M&E. National bureaus of
statistics are heavily involved in many cases, and in
Evaluation enhances
monitoring’s meaning
and depth
Figure 1. Countries with VNRs in 2016 and 2017, and submitting in 2018
2018
2017
2016 Compiled from:
https://sustainabledevelopment.un.org/memberstates
IIED Briefing
four countries they are the main responsible body
for M&E and reporting on the SDGs.
Good evaluation practice
Asia. In Asia overall, the 2030 Agenda is seen as
complementing current national plans and
strategies for sustainable development. Most
Asian countries reporting in 2017 are still
formulating National SDG Roadmaps. In terms of
follow-up and reporting, Asian countries tend to
choose centralised systems heavily driven by
indicators. Examples of reference to and use of
evaluation include:
Nepal has developed a five year ‘Integrated
National Evaluation Action Plan’ for
2016– 2020’ and it’s planning to develop a
national evaluation policy by July 2018.
Indonesia has presented clear evaluation
guidelines for different stages of programmes.
These mention the DAC evaluation criteria of
relevance, effectiveness, efficiency, impact
and sustainability.
Afghanistan puts significant emphasis on
multi-stakeholder approaches for reviewing
progress towards achieving the SDGs.
The Maldives intends to cross-reference
qualitative and quantitative data when reporting
progress.
Europe. In Europe, many countries have made
institutional and consultative arrangements for
implementing the SDGs, have set up national
action plans and are currently establishing follow-
up mechanisms. In addition, several European
countries have reported significant private sector
involvement as companies include the SDGs in
their core business strategy. In terms of follow-up
and review, very few European countries outline
evaluation’s role. However, some examples of good
practice and commitments include:
The Czech Republic’s VNR stresses
monitoring must be accompanied by evaluation
that addresses the SDGs’ complexity when
considering how they are achieved.
Cyprus refers to evaluation in the education
sector and how it is contributing to higher
performance. Evaluation is clearly being used to
improve and adapt programmes.
Portugal stresses that indicators cannot be the
primary driving force for development. Changes
reported by indicators should be investigated to
understand causal linkages between the SDGs
and national policies using methods that assess
attributing and contributing factors.
Italy commits to maintaining stakeholder
involvement in follow-up and review. Stakeholder
involvement will help to select the most relevant
indicators, inform decisions and foster
widespread ownership of the sustainable
development agenda. Defining measures
through stakeholder engagements will provide a
good basis for active and useful evaluation.
Malmo in Sweden has adopted the 2030
Agenda goals and has piloted an assessment of
the municipality’s long-term investments using
the three dimensions of sustainable
development (economic, social and
environmental). This is a good example of
trialling evaluative exercises that recognise
interconnectedness between the SDGs.
Africa. African VNRs have some very good
examples of integrating the 2030 Agenda and
Agenda 206310 at the local level, including using
evaluative processes. Most countries reported
involving civil society in developing and validating
the VNRs.
In Kenya, the government has studied the
period 2000–2015 to assess progress against
the MDGs and to document the experiences,
challenges and lessons learnt. This study has
informed the new Medium-Term Plan and the
Road Map for implementing the SDGs. The VNR
has been validated by different stakeholders,
including through consultations with civil society
and the private sector. Kenya’s VNR also shows
willingness to tap into new and non-traditional
data sources to complement statistics.
Botswana has highlighted the need for
multi-sectoral involvement when the statistics
office develops the reporting and national
indicators system. This will provide a good basis
for evidence-based evaluation.
Ethiopia emphasises the value of learning
through M&E and has conducted a national
review of performance against the MDGs. This
has fed into the preparation of Ethiopia’s
national agenda for the SDGs.
Some companies on the Zimbabwe stock
exchange have adopted sustainability reporting
in accordance with the Global Reporting
Initiative G4 guidelines. This is generating
evaluative evidence for the private sector’s
contribution to the SDGs.
Latin America. In Latin America, governance
models for the SDGs tend to be participatory, and
include both governmental and non-governmental
institutions. In terms of follow-up and review,
countries are investing significant energy in
improving monitoring activities and in
strengthening statistical capacities. Good
examples showing understanding of, and
commitment to, evaluation include:
Knowledge
Products
The International Institute
for Environment and
Development (IIED)
promotes sustainable
development, linking local
priorities to global
challenges.
EVALSDGs is a network of
policymakers, institutions,
and practitioners who
advocate for eective
evaluation for the SDGs.
UNDP Independent
Evaluation Oce (IEO)
helps UNDP achieve greater
accountability and
facilitates learning from
experience by conducting
evaluation and generating
objective evidence.
EvalPartners is a global
partnership that aims to
influence policymakers,
public opinion and other
stakeholders so evaluative
evidence is valued and used.
Contact
Stefano D’Errico
stefano.derrico@iied.org
80–86 Gray’s Inn Road
London, WC1X 8NH
United Kingdom
Tel: +44 (0)20 3463 7399
Fax: +44 (0)20 3514 9055
www.iied.org
IIED welcomes feedback
via: @IIED and
www.facebook.com/theiied
UNDP’s Independent
Evaluation Office supports
the organisation in its
efforts towards full
accountability; contributes
to informed decision-
making by the UNDP
Executive Board and
UNDP’s senior
management; and promotes
management for
development
results by the
organisation as a
whole.
IIED Briefing
Costa Rica acknowledges the importance of
evaluation for policy review, effectiveness,
efficiency and accountability.
Guatemala is planning periodic national and local
reviews. These will feed policy recommendations
into the cycle of public management.
Belize’s VNR utilises earlier evaluation findings
to assess the current situation for cash transfer
programmes, gender equality and
fisheries/marine management.
The Government of Panama has established a
multidimensional approach to measure welfare,
poverty and discrimination/segregation. This
approach requires specific policies and M&E
mechanisms.
Recommendations
The 2030 Agenda clearly recognises the role of
evaluation for strengthening analysis,
accountability, learning and stakeholder
engagement for the achievement of the SDGs,
particularly at the national level. Yet despite
outlining several good practices, the second round
of VNRs reveals a general lack of reflection and
understanding about how evaluation should be
included into the SDGs’ national follow-up and
review processes. To address this, we recommend:
The current VNR guidelines should be revised
by the UN Secretariat and General Assembly to
clearly outline evaluation measures and review
processes. This process should engage and be
informed by the UN Evaluation Group (UNEG)
who represent all the UN Evaluation agencies,
and the broader evaluation community, including
the International Development Evaluation
Association (IDEAS), EvalPartners, the
International Organization for Cooperation in
Evaluation (IOCE) and the Clear initiative,
among others.
Countries still preparing their first VNR should
engage with regional and national Voluntary
Organisations for Professional Evaluation
(VOPEs) early in the process and not wait until
evaluations need to be implemented. They
should develop national evaluation policies to
harmonise practices across the various levels of
government and local authorities.
The evaluation community should engage
in VNR processes at both global and national
levels:
a. International evaluation networks and
organisations should actively contribute to the
revision of VNR guidelines by providing guidance
on how to integrate evaluation into the SDGs’
follow-up and review processes.
b. At national levels, VOPEs should start high-level
dialogues with their governments about the role of
evaluation in achieving the SDGs and how to
integrate it into monitoring systems. UNEG, IOCE
and IDEAS should support VOPEs’ efforts.
c. It is key that evaluators engage with the
specific institution in charge of defining
evaluation processes, so as to ensure that
evaluative exercises are timely and that
evaluations use appropriate methodologies and
approaches. In some countries, this will be the
institution charged with coordinating overall
efforts for implementing the SDGs.
d. There is a need to strengthen evaluation
capacity and use in all countries; this requires
courageous political will, adequate resources and
evaluation expertise. Particularly in:
i) Adopting and sustaining participatory
processes, and
ii) Conducting and using evaluation, including
developing analytical skills. Bilateral and
multilateral funders should support countries that
are seeking assistance to further develop their
systems and capacities.
Wolfgang Meyer, Indran Naidoo, Stefano
D’Errico, Silke Hofer, Madeeha Bajwa,
Luisa Alejandra Tello Pérez, Kassem
El-Saddik, Dorothy Lucks, Benoit Simon and
Ilenia Piergallini
Wolfgang M eyer is the vice- director of CEva l. Indran Naid oo is the
direct or of IEO UNDP. Stefano D’E rrico is the MEL l ead at IIED and the
advoca cy lead at EVALSDG s. Silke Hofer is an in dependent
consult ant. Madeeh a Bajwa is the chief of t he Evaluation an d
Monito ring Unit at UNCTAD. Lu isa Alejandra Tello Pé rez is an
indepen dent consulta nt. Kassem El- Saddik is the vic e-chair of
EVALSDG s. Dorothy Lucks is t he co-chair of E VALSDG s. Benoit
Simon is t he director of Pla nète Publique . Ilenia Pierga llini is a
coordin ator in the Strate gy and Learning G roup at IIED.
Acknowledgements: Ada Ocampo (UNICEF), Emilie Beauchamp
(IIED), Suw o T Woibah (Liberia) , Janna Sofroni (U NCTAD), Sana Bin
Salem (Tunisia), Rima Abu Bakar (Lebanon) and Samandar Mahmoudi
(Afghanistan).
Notes
1 United Na tions (2015) Transformi ng our world: the 20 30 Agenda for Sus tainable Deve lopment. Res olution adopte d by the General As sembly
on 25 Septe mber 2015 (A/RES/70/1), page 32, 74g. / 2 The Se cretary Gen eral (2015) A/70/68 4, Report of the S ecretary G eneral Guidel ines.
United N ations. Repor t on critica l milestones t owards coherent , efficient an d inclusive follow -up and review at the g lobal level, pa ge 25–27, annex
on propos al for voluntar y common repor ting guideli nes for voluntar y national rev iews at the high-le vel politica l forum. / 3 Schwand t T et al. (2016)
Evaluati on: a crucial ingr edient for SDG suc cess. IIED, L ondon. http: //pubs.iied .org/17357IIED / 4 Simon B et al. (2017) Evaluati on: a missed
oppor tunity in the S DGs’ first set of Volunt ary Nationa l Reviews. IIED, L ondon. http: //pubs.iied .org/17423IIED / 5 We have used five crite ria for
asses sing the VNRs: use of M& E keywords, ment ion of a report ing system for mea suring progres s towards the SDG s, descripti on of a
methodo logical fra mework for M&E, an d evidence that ev aluation is inte grated into follow -up and review sys tems and into the ge neral
governance system. / 6 Some nota ble countrie s haven’t yet commi tted to prese nting their VNRs , including the US A, the UK, Ru ssia and South
Africa . / 7 The process a rising from the 1992 Un ited Nations Co nference on Enviro nment and Develo pment, held in R io, Brazil (also k nown as
the Rio de Ja neiro Eart h Summit). / 8 As presented in th e annex to A/70/684 . / 9 UNDP (2017) Guidelines to sup port countr y reportin g on the
Sustainable Development Goals. https://undg.org/wp-content/uploads/2017/03/Guidelines-to-Support-Country-Reporting-on-SDGs-1.pdf /
10 African Union Commission (2015) Agenda 2063, the Africa We Want. www.un.org/en/africa/osaa/pdf/au/agenda2063.pdf
Download the pdf at http://pubs.iied.org/17446IIED
... For the first time, there was consensus among the 193 countries that constitute the United Nations General Assembly on what the global development priorities ought to be and a roadmap for achieving them. At the same time, the articulation of these priorities brought the importance of 'transformative development' solutions -those that move beyond government-led development efforts and mobilise unconventional change agents -squarely into focus (Meyer et al. 2018). Among those solutions, innovative finance, impact investing and other market-based approaches stand out as perhaps the most promising levers for change. ...
... A study by the UN Sustainable Development Solutions Network found that if substantial progress towards the SDGs is to be made globally, an additional $1.3 trillion will be needed each year (Meyer et al. 2018; Organisation for Economic Co-operation and Development [OECD] 2017). This represents over nine times the amount that countries receive as official development assistance, an amount that peaked in 2016 at $142.6 billion and is not expected to increase further. ...
... feature prominently in contemporary discourse. Prominent organisations supporting international development evaluation, such as the International Institute for Environment and Development, EvalSDGs and EvalPartners, have posited the critical role that producing new evidence and data can play towards fostering transformative development (Meyer et al. 2018). ...
Article
Full-text available
Having invested in the architecture for the field of impact investing, and consistent with its long-standing commitment to advancing development and prosperity in Africa, The Rockefeller Foundation is proud to support this special edition of the African Evaluation Journal focused on social impact measurement in Africa.
Report on critical milestones towards coherent, efficient and inclusive follow-up and review at the global level, page 25-27, annex on proposal for voluntary common reporting guidelines for voluntary national reviews at the high-level political forum
  • T Schwandt
United Nations (2015) Transforming our world: the 2030 Agenda for Sustainable Development. Resolution adopted by the General Assembly on 25 September 2015 (A/RES/70/1), page 32, 74g. / 2 The Secretary General (2015) A/70/684, Report of the Secretary General Guidelines. United Nations. Report on critical milestones towards coherent, efficient and inclusive follow-up and review at the global level, page 25-27, annex on proposal for voluntary common reporting guidelines for voluntary national reviews at the high-level political forum. / 3 Schwandt T et al. (2016) Evaluation: a crucial ingredient for SDG success. IIED, London. http://pubs.iied.org/17357IIED / 4