Measuring the Impact of Health Systems Strengthening: A Review of the Literature

Abstract

The purpose of this literature review is to summarize current efforts in measuring health system performance and to highlight the indicators and performance benchmarks most frequently used by the global community. The review also aims to serve as a resource for health system experts working on building consensus around a core set of indicators for monitoring and evaluating health system performance. The literature review is particularly useful in the context of the Millennium Development Goals and the “Countdown to 2015” initiative, as well as in the context of current U.S. Government presidential initiatives, bilateral programs, and host-country planning processes.

Full text

Measuring the Impact
of Health Systems Strengthening
A Review of the Literature

1
Measuring the Impact
of Health Systems Strengthening
A Review of the Literature
Written by (in alphabetical order)
Soumya Alva
Eckhard Kleinau
Amanda Pomeroy
Kathy Rowan
November 2009
U.S. Agency for International Development

2
Acknowledgments
This report was prepared for the United States Agency for International Development (USAID) by the
Analysis, Information Management & Communications (AIM) Activity. Sincere thanks to the Bureaus
for Europe and Eurasia (E&E), Latin America and the Caribbean (LAC), and Global Health of USAID for
creating the opportunity and entrusting the AIM Activity to conduct this literature review. The authors
wish to thank Susanna Baker, Forest Duncan, Bob Emrey, Roshni Ghosh, Ligia Paina, Kelly Saldana and
others who reviewed and provided insightful comments to the drafts of this report. Credit is also due to
the AIM Activity communications team for its support in editing and producing this report. The AIM
Activity is funded by USAID and managed by MasiMax Resources, Inc., with ICF Macro; John Snow,
Inc.; and Insight Systems (contract no. GS-10F-0311M, order no: GPO-M-00-05-0043-00). The AIM
Activity provides the Bureau for Global Health and others with essential information, products, and
services about program needs, technologies, costs, and impacts to support accurate priority setting,
program design, management, and evaluation. The authors’ views expressed in this publication do not
necessarily reflect the views of USAID or the U.S. Government.

3
Contents
Acknowledgments........................................................................................................................... 2
Acronyms and Abbreviations ......................................................................................................... 4
Introduction .................................................................................................................................... 6
Part I: Summary of Findings ........................................................................................................ 7
Defining Health Systems ........................................................................................................................ 8
Why Do Health Systems and Their Performance Matter? ................................................................. 9
Past and Current Efforts in Measuring Health Systems Performance ........................................... 10
Health Systems Indicators and Data ................................................................................................... 15
Data and Indicators: Limitations, Uses, and Criteria ....................................................................... 21
Part II: Detailed Findings by Health System Building Block .................................................... 23
Service Delivery .................................................................................................................................... 23
Health Workforce ................................................................................................................................. 29
Information ........................................................................................................................................... 34
Medical Products, Vaccines, and Technologies ................................................................................. 42
Financing ............................................................................................................................................... 50
Leadership and Governance ................................................................................................................ 56
References .................................................................................................................................... 66
Appendix 1: Indicators by Organization ..................................................................................... 73
Guide to Appendix 1: ........................................................................................................................... 94
Appendix 2: Health Output Indicators ........................................................................................ 98
Tables
Table 1: Past and Current Efforts in Measuring Health Systems Performance ...................... 12
Table 2: Most Recommended Health Systems Indicators by Building Block (BB) – WHO &
Other Sources ............................................................................................................................... 19

4
Acronyms and Abbreviations
CGD Center for Global Development
CPI Corruption perceptions index
CPIA Country Policy and Institutional Assessment
CSO Civil society organization
DfID U.K. Department for International Development
DHS Demographic and Health Surveys
DOS Department of State
DPT Diphtheria-pertussis-tetanus
E&E Europe and Eurasia
ECHI European Community Health Indicators
ECHIM European Community Health Indicators Monitoring
EO European Observatory
EPHF Essential public health functions
EU European Union
GGE General government expenditure
GGHE General government health expenditure
HAI Health Action International
HFC Health Facility Census
HIS Health information system
HISPIX Health information system performance index
HMIS Health management information systems
HMN Health Metrics Network
HSAA Health Systems Assessment Approach
HSS Health systems strengthening
JSI John Snow, Inc.
LAPM Long-acting and permanent method
LSMS Living Standards Measurement Study
M&E Monitoring and evaluation
MCA Millennium Challenge Account
MCC Millennium Challenge Corporation
MCH Maternal and child health
MDG Millennium Development Goal
MICS Multiple Indicator Cluster Surveys
MOH Ministry of health
MSH Management Sciences for Health
NGO Nongovernmental organization
NHA National Health Account
OECD Organization for Economic Cooperation and Development
OPD Outpatient department
PAHO Pan American Health Organization
PEPFAR U.S. President’s Emergency Plan for AIDS Relief
PHRplus Partners for Health Reform Plus
QAP Quality Assurance Project
RPM Rational Pharmaceutical Management
SAM Service Availability Mapping
SBA Skilled birth attendance
SFA Strategic Framework for Foreign Assistance

5
SPA Service Provision Assessment
SWEF System-Wide Effects of the Fund
TB Tuberculosis
THE Total health expenditure
TI Transparency International
UNICEF United Nations Children’s Fund
URC University Research Corporation
USAID United States Agency for International Development
WHO World Health Organization
WHO/EMRO World Health Organization Regional Office for the Eastern Mediterranean
WHOSIS World Health Organization’s Statistical Information System
WHO/WPRO World Health Organization Regional Office for the Western Pacific
WHS World Health Surveys

6
Introduction
Funding for the health sector in developing countries has grown significantly in the last decade. While
health sector investments have increasingly focused on alleviating the impact of HIV/AIDS, tuberculosis,
and malaria, both development actors and the governments of developing countries are shifting their
priorities toward linking disease-specific interventions with longer-term investments in health systems
strengthening (HSS). With increased funding, there comes a greater need for evidence-based
decisionmaking and rigorous monitoring and evaluation of HSS programs.
The purpose of this literature review is to summarize current efforts in measuring health system
performance and to highlight the indicators and performance benchmarks most frequently used by the
global community. The review also aims to serve as a resource for health system experts working on
building consensus around a core set of indicators for monitoring and evaluating health system
performance. The literature review is particularly useful in the context of the Millennium Development
Goals and the “Countdown to 2015” initiative, as well as in the context of current U.S. Government
presidential initiatives, bilateral programs, and host-country planning processes.
The review builds on the latest available published work on health system performance indicators and
existing data sources. Part I summarizes the review findings, and part II provides a detailed overview of
each of the six health system building blocks identified by the World Health Organization.
The current version of this document has benefited from discussions with the United States Agency for
International Development technical experts and implementing partners. Because of the growing global
emphasis on monitoring and evaluating health system performance, the document will be periodically
updated to reflect the most current thinking in the field. Future versions will build on the current one and
incorporate feedback received through the online portal, as well as from published updates from academia,
multilateral organizations, nongovernmental organizations, and special initiatives.

7
Part I: Summary of Findings
Over the years, the World Health Organization (WHO) has placed increased importance on health
systems, treating them as the means to deliver effective and affordable interventions to those in need. This
is particularly relevant to meeting the Millennium Development Goals (MDGs) and to achieving better
health equity among all population sectors, especially the poor. The principal aim of health systems
strengthening (HSS) interventions is to improve three aspects of priority health services: access, quality,
and utilization. However, another crucial element of HSS is to strengthen host-country health systems in
order to ensure sustainability over time and to eventually phase out donor assistance to the health sector.
The HSS at the country level is based on high-impact and cost-effective interventions in maternal and
child health (MCH), family planning and reproductive health, HIV/AIDS, tuberculosis (TB), malaria, and
other infectious diseases.
The United States Agency for International Development (USAID) plays a key role in strengthening
health systems in developing countries by working with individual governments, nongovernmental
organizations (NGOs), and donor agencies. USAID utilizes various contract mechanisms to support
health systems through bilateral projects in individual countries as well as through centrally funded
activities that support multiple countries. The Agency partners with international organizations and other
bilateral donors to strengthen health systems to maximize the effectiveness of its efforts. USAID’s current
central project activities on HSS include Health Systems 20/20, Strengthening Pharmaceutical Systems,
the Health Care Improvement Project, Drug Quality and Information, MEASURE DHS, and MEASURE
Evaluation. The main contributions of these projects have been assessment methodologies and
instruments such as the Demographic and Health Surveys, Service Provision Assessment, and National
Health Accounts. They also have produced practical guidelines, for example, about health facility
assessment methods (MEASURE Evaluation, 2006) and performance indicators (MEASURE Evaluation,
2007). A more comprehensive list can be found in Table 1. The guidelines and tools are used in several of
USAID’s bilateral projects that support health systems, for example, in Armenia, Bolivia, Ethiopia,
Guatemala, Indonesia, Kyrgyzstan, Nepal, Nicaragua, Tanzania, Zambia, and many other countries.

8
Defining Health Systems
WHO describes a health system as consisting of all the organizations, institutions, resources, and people
whose primary purpose is to improve health. It needs staff, funds, information, supplies, transport,
communications, and overall guidance and direction. And it needs to provide services that are responsive
and financially fair, while treating people decently (WHO, 2007). Strengthening health systems involves
addressing key constraints related to health worker staffing, infrastructure, health commodities (such as
equipment and medicines), logistics, tracking progress, and effective financing.
The 2000 WHO World Health Report examines and compares aspects of health systems around the world.
It provides conceptual insights into the complex factors that explain how health systems perform and
offered practical advice on how to assess performance and achieve improvements with available
resources. The report focuses on the performance (ultimate outcomes) of health systems, described health
system functions (stewardship, resource creation, service provision, and financing), emphasizing the
stewardship role of the government. However, the key failing elements of the report are that it provides
little information on why a particular system setup yields a certain outcome, what features of that system
contribute the most to the outcome, and how one could restructure the system to achieve a better outcome.
Later WHO reports have focused on health systems performance – the 2006 World Health Report, for
example, covers the need for human resources for health (WHO, 2006a), and the 2007 report,
Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework
for Action, develops a conceptual framework for understanding how health systems operate, which also
provides the basis for a toolkit of HSS monitoring indicators (WHO, 2007).
Source: WHO, 2007.

9
The framework developed in the 2007 report depicts a health system in terms of six core building blocks,
shown in the diagram on page 8.
The main purpose of the framework is to portray what a health system is and what constitutes HSS. Using
a single framework is important for understanding where investment is necessary to provide better access,
coverage, quality, and safety, resulting in improved health.
These building blocks represent the basic functions of health systems. They cannot function
independently of one another; systems strengthening activities in one building block have repercussions
on the functions of another. This interconnectedness is particularly evident in the case of governance and
information systems. Information provides the evidence base for overall sector policies, although not
always observed in practice, and leadership and governance impacts the performance of all the other
health system blocks. Financing and human resources for health are core inputs that affect all other
building blocks. Medical products, vaccines, and technologies impact service delivery and reflect the
availability and distribution of care, which are immediate outputs of the health system. Based on this
framework, WHO developed a toolkit for monitoring HSS that contains six sections with draft indicators,
each section corresponding with a building block (WHO, 2008).
Why Do Health Systems and Their Performance Matter?
A health system is a means to an end. The main objective of a good health system is to improve people’s
lives tangibly every day. The health system is the means to achieve better health outcomes, such as better
child survival through immunization, improved maternal health through emergency obstetric care and
birth spacing, and lower levels of incidence of HIV, malaria, and other infectious diseases through
prevention.
The characteristics of a well-performing health system are greater equitability, efficiency, and
sustainability of health service outputs by delivering accessible, high-quality, and affordable curative and
preventive services (Rockefeller Foundation, 2008). As a recent review commissioned by the U.K.
Department for International Development (DfID) explains, the purpose of improved measurements of
health systems performance is threefold:
• To provide better accountability for expenditures on health

10
• To increase aid effectiveness through more efficient allocation of resources and better
performance management
• To increase interest in performance- and results-based aid, especially when aid disbursements are
related to results (Walford, 2007)
As a result, health systems performance rather than just health funding is all the more important for
enabling improvements in health outputs. Following this line of thought, strengthening health systems and
making them more equitable have been recognized as key strategies for fighting poverty and fostering
development. A review by Marchal et al. (2009) finds that global health actors implement very different
interventions in their field projects. These can be categorized as:
(1) providing inputs or resources;
(2) reinforcing capacities of health services that are directly related to implementation of disease-
control programs; and
(3) integrating program activities into general health services.
This process, whether implemented by individual governments, NGOs, or donor agencies, is under way in
many countries to better respond to their population’s needs. A growing number of WHO member states
and the world’s political and international health leaders also recognize the urgent need to make a major,
sustained commitment to strengthening health systems.
Past and Current Efforts in Measuring Health Systems Performance
With the increased emphasis on health systems performance, WHO has taken a lead role in identifying
appropriate measurement indicators that can be used for multiple purposes, such as:
• In-country planning and monitoring
• Assessment of country performance
• Results-based funding
• Making intercountry comparisons to aid funding allocations
Organizations such as the GAVI Alliance and The Global Fund to Fight AIDS, Tuberculosis and Malaria
already use indicators for these purposes. GAVI encourages proposals on HSS where the link to
immunization coverage can be established. The Global Fund uses health information systems (HIS) for
performance-based disbursements (HMN, 2006).

11
Other organizations and working groups, such as the WHO Health Metrics Network (HMN), The World
Bank, the Global Health Indicators Working Group of the Center for Global Development (CGD), the
University of Washington Institute of Health Metrics and Evaluation, and USAID, also have developed
country indicators of health system performance, with plans to test them in selected countries (Walford,
2007). The Interagency Group on Health System Metrics, which comprises WHO, The World Bank,
HMN, The Global Fund, and GAVI, are working together to compile summary indicators of health
systems performance as well as a longer list of indicators for countries to choose from, with the objective
of monitoring support to countries, provided by The Global Fund and GAVI. Beside these collaborative
efforts, organizations have devised indicators and tools to measure health systems performance to meet
their program needs. For example, The World Bank has developed indicators as part of its results-based
financing for health; USAID projects have developed health system assessment tools covering MCH
services; and the Organization for Economic Cooperation and Development (OECD) tracks health care
quality as part of its Health Care Quality Indicators project through internationally comparable indicators.
Table 1 summarizes past and current efforts in measuring health systems performance. The tools and
indicators developed by various organizations that examine the indicators proposed for each of the health
systems building blocks according to the WHO framework also will be cited in the remaining sections of
this paper.

12
Table 1: Past and Current Efforts in Measuring Health Systems Performance
Organization
Publications, Tools, and Assessment Methods (in italics)
1.
WHO
•
Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes:
WHO’s Framework for Action (WHO, 2007)
• Toolkit on Monitoring Health Systems Strengthening (WHO, 2008)
• Service Availability Mapping (SAM)
(http://www.who.int/healthinfo/systems/serviceavailabilitymapping/en/index.html)
• Prevention SAM (http://www.who.int/healthinfo/systems/sampsam/en/index.html)
• Action Program on Essential Drugs, Indicators for Monitoring National Drug
Policies: A Practical Manual (WHO, 1999)
• WHO Medicines Strategy (WHO, 2008a)
• Level I and Level II Indicators to Assess Country Pharmaceutical Situations
(WHO, 2006)
• Public Education in Rational Drug Use: A Global Survey (Fresle & Wolfheim, 1997)
• Developing Health Management Information Systems: A Practical Guide for
Developing Countries (WHO, 2004)
• Alliance for Health Policy and Systems Research (http://www.who.int/alliance-
hpsr/en/)
• Framework for Assessing Health System Governance (WHO/EMRO, 2007)
• Essential Public Health Function Measurement (WHO/PAHO, 2008)
• World Health Surveys (WHS) (http://www.who.int/healthinfo/survey/en/index.html)
2.
WHO/HMN
• Health Metrics Network (HMN) Framework and Standards For Country Health
Information Systems (HMN, 2008)
•
HMN Health Information System Assessment Tool (HMN, 2008a)
3.
UNICEF
• Multiple Indicator Cluster Surveys (MICS)
(http://www.unicef.org/statistics/index_24302.html)
4.
USAID
MEASURE DHS: (http://www.measuredhs.com/)
• Service Provision Assessment (SPA)
• Demographic and Health Surveys (DHS)
5.
USAID
MEASURE Evaluation/International Health Facility Assessment Network (IHFAN)
(http://www.cpc.unc.edu/measure/networks/ihfan)
• Profiles of Health Facility Assessment Methods
• Guidance for Selecting and Using Core Indicators for Cross-Country Comparisons of
Health Facility Readiness to Provide Services
• Facility Audit of Service Quality (FASQ)
• Signature Domain and Geographic Coordinates: A Standardized Approach for
Uniquely Identifying a Health Facility
• Quick Investigation of Quality (QIQ): A User’s Guide for Monitoring Quality of Care
in Family Planning
•
Performance of Routine Information System Management (PRISM)
6.
USAID
• Child Survival and Technical Support Plus: Rapid Health Facility Assessment
(R-HFA) (http://www.childsurvival.com/rhfa_1.cfm)
7.
USAID
• Logistics System Assessment Tool (LSAT) (DELIVER, 2009)
8.
USAID
• Strategic Pathway to Reproductive Health Commodity Security (SPARHCS)
(Hare,
Hart, Scribner, & Sheperd, 2004)
9.
USAID
• Contraceptive Security Index (DELIVER, 2006)
•
Family Planning Effort Index (Ross & Stover, 2001)
10.
USAID
ACQUIRE Project: (http://www.engenderhealth.org/our-work/major-
projects/acquire.php)
•
ACQUIRE Evaluation of LAPM Services (ELMS)
11.
USAID
• Capacity Project: Human Resource Information System (HRIS) Strengthening
(http://www.capacityproject.org/)

13
Table 1: Past and Current Efforts in Measuring Health Systems Performance
Organization
Publications, Tools, and Assessment Methods (in italics)
12.
USAID
Population Council: Population Council Health Facility Assessment (PCHFA)
(http://www.populationcouncil.org)
13.
USAID
Health Systems 20/20 (http://www.healthsystems2020.org/):
• Health Systems Assessment Approach (with Rational Pharmaceutical Management
Plus /QAP)
• National Health Accounts (NHAs)
• Health systems database
• The System-Wide Effects of the Fund (SWEF) research protocol with the SWEF
Research Network (Bennett & Fairbank, 2003)
14.
USAID
Rational Pharmaceutical Management Plus/QAP
(http://www.msh.org/projects/rpmplus/):
•
Rapid Pharmaceutical Management Assessment: An Indicator-based Approach
15.
USAID
•
Strategic Framework for Foreign Assistance (SFA) Governance Indicators
(http://www.state.gov/f/c23053.htm)
16.
USAID
• Graduation Report on Increased Health Promotion and Access to Quality Health
Care in the Europe and Eurasia (E&E) Region (internal document, developed by
E&E Bureau’s Program Objective Team for Strategic Objective 3.2) (USAID
Program Objective Team 3.2, 2001)
17.
U.S. President’s
Emergency Plan for
AIDS Relief
(PEPFAR)
• Health Facility-based Survey of Human Resource for Health in HIV/AIDS, TB,
Malaria and MCH Services
18.
Japan International
Cooperation Agency
(JICA)
• Health Facility Census
19.
The World Bank
• Statistical Capacity Indicator
• Results-based Financing in Health
• Country Policy and Institutional Assessment (CPIA) Index
• Living Standards Measurement Study (LSMS)
• Public Expenditure Review (PER)
• Quantitative Service Delivery Surveys (QSDS)
• Public Expenditure Tracking Survey (PETS)
• Performance Monitoring Indicators: A Handbook for Task Managers (Mosse &
Sontheimer, 1996)
• Benefit Incidence Analysis (BIA): BIA tools and evaluations have been made publicly
available as part of the Research Department of the World Bank.
20.
The Global Fund to
Fight HIV/AIDS,
Tuberculosis and
Malaria
• Global Fund Monitoring and Evaluation Toolkit (The Global Fund, 2006 and 2008)
21.
The Global
HIV/AIDS Initiatives
Network (GHIN)
• Country studies examine the national and subnational effects of global HIV/AIDS
initiatives (by The World Bank Global HIV/AIDS Program, Global Fund, and
PEPFAR) at the country level. (Gottret & Schieber, 2006)
22.
GAVI Alliance
• Guidelines for GAVI Alliance Health System Strengthening (HSS) Applications (GAVI
Alliance, 2007)
23.
DfID
• Future Health Systems consortium, with the aim of publishing health systems
evaluations (http://www.futurehealthsystems.org)
Health Systems Resource Centre (http://www.dfidhealthrc.org/)

14
Table 1: Past and Current Efforts in Measuring Health Systems Performance
Organization
Publications, Tools, and Assessment Methods (in italics)
24.
RAND
•
Quality of Care Assessment Tools (QA Tools)
(http://www.rand.org/health/surveys_tools/qatools/index.html)
25.
United Nations
MDGs
• MDG indicator list (http://www.un.org/millenniumgoals/)
26.
University of
Washington Institute
of Health Metrics and
Evaluation
• Evaluation criteria for health systems currently in development
(http://www.healthmetricsandevaluation.org/)
27.
Harvard University
School of Public
Health
• International Health Systems Program: Carried out small-scale health systems
evaluations (http://www.hsph.harvard.edu/ihsg/research.html)
28.
Center for Global
Development • Measuring Commitment to Health. Global Health Indicators Working Group Report
(Becker, Pickett & Levine, 2006). Recommended Millennium Challenge Account
indicators for country selection.
29.
Millennium
Challenge
Corporation (MCC)
• MCC Selection Criteria (http://www.mcc.gov/mcc/selection/index.shtml)
30.
European
Community Health
Indicators Monitoring
(ECHIM) Project
•
ECHIM Shortlist of Indicators
(http://www.healthindicators.org/healthindicators/root/o13.html)
• ECHIM Comprehensive List of Indicators
(http://www.healthindicators.org/healthindicators/object_document/o5415n28314.ht
ml)
31.
International Health
Partnership Plus (IHP
+)
• International Health Partnership Plus was launched in 2007 to respond to the MDG
challenges that called for action to scale up coverage and use of health services and
deliver improved outcomes against the health-related MDGs and universal access
commitments. The focus is on health-related MDG outcomes. The Partnership
produces monitoring and evaluation frameworks and country-level reports. A focus of
the partnership is on health systems performance.
(http://www.internationalhealthpartnership.net/en/home)
32.
European
Observatory on
Health Systems and
Policies
• The Observatory is a partnership of the WHO Regional Office for Europe; the
Governments of Belgium, Finland, Norway, Slovenia, Spain, and Sweden; the Veneto
Region of Italy; the European Investment Bank; The World Bank; the London School
of Economics and Political Science; and the London School of Hygiene and Tropical
Medicine. Its work includes the “Health Systems in Transition (HiTs)” country
profiles and other analyses of the dynamics of health care systems in Europe.
(http://www.euro.who.int/observatory)
33.
OECD
• OECD’s Health Care Quality Indicators Project’s aim is to collect internationally
comparable data reflecting health outcomes and health improvements attributable to
medical care delivered in OECD countries. It produced five technical papers on health
care quality indicators, covering cardiac care, diabetes, primary care and prevention,
mental health, and patient safety. It also participated in the ECHIM effort.
(http://www.oecd.org/document/31/0,2340,en_2649_33929_2484127_1_1_1_1,00.ht
ml
)
Note: Household surveys such as USAID’s Demographic and Health Surveys (DHS), the Multiple Indicator
Cluster Surveys (MICS) of the United Nations Children’s Fund (UNICEF), and WHO’s World Health Surveys
(WHS) are included because they contain information related to health systems performance and overall goals and
outcomes shown in the WHO framework.

15
The past and current efforts in measuring health systems performance listed in table 1 serve different
needs. Some are broad assessments of health systems performance, such as the WHO Toolkit on
Monitoring Health Systems Strengthening, while others target specific systems components, such as the
logistics of distributing essential products or the quality of care provided for a specific disease category.
The tools addressing a specific systems component usually cover performance measures in much greater
detail than the comprehensive systems assessment tools. As can be expected from such a multitude of
efforts, considerable overlap exists between various tools. While some tools complement each other (for
example, rapid assessment tools and more comprehensive health facility audits), there exists little
agreement on a standard set of indicators and assessment methods that can be used for a cross-country
comparison. The WHO Toolkit and several publications from USAID’s MEASURE Evaluation and DHS
projects are the most recent efforts toward standardization.
Indicators also have been developed to evaluate specific health outcomes, particularly in the areas of
HIV/AIDS and reproductive health. Some examples are:
Reproductive Health:
• Bertrand, J., & Escudero, G. (2002). Compendium of Indicators for Evaluating Reproductive
Health Programs. Vol. I
HIV/AIDS:
• Management Sciences for Health/World Health Organization. (2006). Tools for Planning and
Developing Human Resources for HIV/AIDS and Other Health Services
• U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). (2009). Next Generation Indicators
Reference Guide. Version 1.0
Furthermore, specific indicators listed in table 1, such as the Family Planning Effort Index, the
Contraceptive Security Index, and SPARHCS, are relevant only to specific health outcomes, such as those
related to reproductive health.
Health Systems Indicators and Data
The international health community has come to a consensus regarding the key indicators representing
health outputs and outcomes (see appendix 2). This paper summarizes past and current efforts on HSS
indicators and aims to capture broad recommendations across development organizations. The
WHO Toolkit on Monitoring Health Systems Strengthening serves as the organizing framework for the

16
HSS indicators included in this review because the Toolkit is based on a simple framework, and there
appears to be some consensus among development actors around several of its indicators. Where such a
consensus has not been reached yet, the following sections provide options for further discussion. These
options include indicators recommended by other agencies, such as the USAID Health Systems
Assessment Approach (HSAA), SWEF Research Network, MCC, MDG indicators, the ECHIM Project,
etc., which are captured to some degree in the indicators presented by the WHO Toolkit. The indicators
recommended by the Toolkit, as well as by other sources, are described in the respective sections on each
of the building blocks in this report. A summary of indicators per building block that were most often
recommended by these organizations follows in table 2. These indicators are recommended by key
organizations engaged in HSS and usually by multiple sources. Appendix 1 presents a detailed list of all
indicators by donor and other organization, and includes information on thresholds, benchmarks, and
targets.
The challenge in selecting a few indicators measuring health systems performance from a large number of
potential candidates lies in the fact that no single indicator can capture the performance of a systems
building block adequately. Instead, the selection compromises between a parsimonious set of indicators
and covering health systems core functions comprehensively. One approach to keeping the number of
indicators small is the creation of composite indices. While this effectively leads to fewer indicators,
indices pose several challenges that need to be carefully addressed. Firstly, composite indices are
constructed of several – sometimes many – indicators and therefore do not reduce the burden of data
collection. Secondly, indicators contributing to an index may not all be of equal importance and may
require some form of weighted averages; weighing the components of an index could be perceived as
arbitrary. Thirdly, interpreting indices might be difficult without breaking them into their individual
components, especially when data are intended to inform decisionmaking and actions. Lastly, different
indicators may be included in an index to reflect a higher or lower level of health system development,
which could limit comparability across countries even when indices are normalized. Table 2 shows which
indicators are composite indices (or where no consensus exists yet) and which indicators could be used to
create a new index. While these indicators are generally applicable to all countries and are useful for
cross-country comparisons, there may be other indicators that are more specific to conflict or post-conflict
settings that are beyond the scope of this review. Furthermore, they are primarily limited to indicators that
can be aggregated to the national level for easier application and analysis, although theoretically,
measures of facility- or community-level care and services may be relevant.
Some summary observations from the information presented in table 2 for each building block of the
WHO framework follow:

17
• The quality of health service delivery or service capacity is of concern to many development
organizations, including USAID, WHO, and The World Bank. Projects often develop their own
assessment instruments that measure a specific health systems component supported by them. For
example, several tools focus on HIS or family planning services. USAID and other organizations
have used numerous indicators measuring these and other health systems functions, often
combining specific health systems areas with logistics functions; the latter falls under a separate
building block in the WHO Toolkit. Consensus about quality-of-service indicators or an
appropriate composite index has not been reached yet. One reason for this might be that
measurements of quality of care are likely to differ by region and by the level of a country’s
development. Some consensus seems to exist about indicators measuring availability of services,
although, as table 2 shows, these indicators do not capture the unequal distribution of services in
many developing countries well.
• The issue of unequal distribution is addressed by indicators under health workforce, including
geographic availability by type of health worker. While important, a uniform measure of country
actions to strengthen its health workforce has not yet been agreed upon.
• The WHO Toolkit, as well as HMN publications, emphasizes health information systems
performance. A proposed HIS performance index consists of 29 individual measurements. Other
indicators require simple qualitative assessments and yes or no answers, such as the existence of
reports or certain institutional responsibilities. The indicators shown in table 2 are discussed by
WHO and other organizations. They are a subset of a larger number of indicators recommended by
WHO. The large number of indicators related to information gathering and processing in the health
sector seems somewhat disproportionate to the relatively small number of indicators proposed
under service delivery.
• While some consensus seems to exist about indicators measuring medical products, vaccines,
and technologies, these face challenges similar to those of the quality-of-service indicators under
the service delivery building block. Which commodities are considered “essential” or “tracer
products” will vary by region and a country’s level of development; this means that different items
may contribute to an index in a different context, and this may limit cross-country comparability
even when indices are normalized. The proposed index for measuring the availability of tracer
medicines and commodities can be composed of as many as 61 individual items. Some potential

18
supplemental indicators are much less demanding and require simple qualitative assessments and
yes or no answers, such as the existence of guidelines or policies.
• Health financing is covered by a relatively small number of indicators; WHO proposes three. Two
of the three, total health expenditure per capita and general government health expenditure as a
proportion of total government expenditure, are well established and reported annually by
international organizations. USAID routinely uses these indicators and variations of them. The
main challenges relate to the need to better measure inequities in health service financing and
whether resources reach the poorest and most vulnerable population groups.
• Leadership/governance (stewardship) is another building block characterized by a large number
of proposed measurements. A proposed policy index consists of 10 components that are qualitative
and relatively easy to assess and report as yes or no answers (although the latter could be refined
by assessing whether individual policy components meet minimum standards of effectiveness). In
addition to the policy index, WHO proposes six marker indicators for governance that overlap to
some extent with other health systems building blocks, such as financing, medical products, and
health workforce. The World Bank is using a country policy and institutional assessment index
composed of 16 subcomponents. As for health service financing, the challenge remains of
measuring whether the needs of the poorest and most vulnerable population groups are identified
and met.

19
Table 2: Most Recommended Health Systems Indicators by Building Block (BB) – WHO & Other Sources
BB
Health Systems Indicators (indicator in italics)
Sources
Service delivery
• Number and distribution of inpatient beds per 10,000 population (service availability)
• Number and distribution of health facilities per 10,000 (or 1,000) population (service availability)
• Proportion of health facilities that meet basic service capacity standards – a composite index of
five components with 20 items in total (service capacity)
•
Proportion of health facilities that meet basic service capacity standards/number and distribution of
health facilities with basic service capacity per 10,000 population – a composite index of a
maximum of nine program components with 15 items in total (service capacity)
• Number of outpatient department (OPD) visits per 10,000 population per year (service utilization)
• Service quality: 14 sample indicators suggested, plus 13 additional data sources
• WHO, multiple
• WHO, multiple
• WHO, USAID,
multiple
• WHO
• WHO, multiple
• WHO
Health workforce
• Number of health workers/health professionals per 10,000 population/per capita/per 1,000
population
• Distribution of health workers – by profession/specialization, region, place of work, and sex
• Annual number of graduates of health professions’ educational institutions per 100,000 population
– by cadre
• Selected indicators for monitoring country actions for strengthening the health workforce – WHO
suggests an additional 10 indicators for country monitoring.
• WHO, multiple
• WHO, multiple
• WHO
• WHO
Information
• Health information system performance index (HISPIX) – a summary measure based on 29
standardized indicators for assessing data quality and the overall performance of the health
information systems (HIS). The indicators below are part of the 29 standardized indicators.
• Existence of demographic or household surveys – an indicator proposed, in addition to five
indicators measuring health surveys
• Percent of births and deaths registered in the country – two of three indicators measuring birth and
death registration
• Percentage of districts that submit timely, complete, accurate reports to national level – one of
seven indicators measuring health facility reporting
• Completion of at least one national health account in last five years – one of four indicators
measuring health systems resource tracking
• Existence of designated mechanisms charged with analysis of health statistics – one of eight
indicators measuring the capacity for analysis, synthesis, and validation of health data
• Availability of a national summary report that contains HIS information – one of eight indicators
measuring the capacity for analysis, synthesis, and validation of health data
• WHO/HMN
• Central Statistics
Office
• UNICEF/MICS,
DHS, WHO/
WHS, Census
Bureau
• MOH
• NHA program,
Abt Associates
• MOH
• MOH
Medical products, vaccines, and technologies
• Proportion of population with access to affordable essential drugs on a sustainable basis – The
MDG indicator on health service delivery has not been monitored regularly, but WHO recommends
nine indicators to measure the structure and process components of access to essential drugs.
The availability and price of essential medicines is measured through the following three indicators:
• Percent of facilities that have all tracer medicines and commodities in stock on the day of visit,
and in the last three months – a composite index based on 61 essential medicines, commodities,
and vaccines (availability of essential medicines)
• Supplemented by median proportion of tracer drugs that are in stock on the day of visit, and in
the last three months (availability of essential medicines)
• Ratio of median local medicine price to international reference price (median price ratio) for
core list of drugs (price of essential medicines)
Supplemental indicators recommended by various sources:
• The existence and year of last update of a published national medicines policy
• Existence and year of last update of a published national list of essential medicines
• Existence of standard treatment guidelines
• Percent of drugs purchased through competitive bidding of total pharmaceutical expenditures
• Appropriate prescription practices and rational drug use
• MDG
• HFS
• SPA
• EMPP
• HFS
• WHO/HAI
• IDPG/MSH
• MOH
• WHO/HAI
• Research studies
• WHO/WHS

20
Table 2: Most Recommended Health Systems Indicators by Building Block (BB) – WHO & Other Sources
BB
Health Systems Indicators (indicator in italics)
Sources
Financing
• Total health expenditures (THE) per capita in international and US$
• THE as a percent of gross domestic product (GDP)
• General government health expenditure as a proportion of total general government expenditure
(GGHE/GGE)
• The ratio of household out-of-pocket payments for health to total health expenditures
Supplemental indicators [Walford (2007), Kruk and Freedman (2008)]:
• Amount of total donor spending on health as a percent of THE
• Percent of government health spending that reaches the poorest income quintile
• Selected indicators for monitoring country actions for strengthening health financing – WHO
suggests an additional six indicators for country monitoring.
• WHO, multiple
• USAID, multiple
• WHO, multiple
• WHO
• USAID, multiple
• MCC/MCA,
USAID
• WHO
Leadership/governance (stewardship)
• WHO Policy index – consists of 10 items, each of which would be rated as zero (adequate policy
does not exist or cannot be assessed) or one (adequate policy is available). With a maximum score
of 10, the index consists of the following components:
1. National health strategy
2. Essential medicines list
3. Policies on drug procurement
4. National strategic plan for TB
5. National malaria strategy/policy
6. United Nations General Assembly Special Session (UNGASS) national composite policy index
questionnaire for HIV/AIDS
7. Comprehensive reproductive health policy
8. Comprehensive multiyear plan for childhood immunization
9. Key health sector documents
10. Surveys for obtaining timely client input
• Marker indicators of governance (six core indicators and one supplemental):
- Human Resources for Health: Health worker absenteeism in public health facilities
- Health Financing: Proportion of government funds that reach district-level facilities
- Health Service Delivery: Stock-out rates (absence) of essential drugs in health facilities
- Health Service Delivery: Proportion of informal payments within the public health care system
- Pharmaceutical Regulation: Proportion of pharmaceutical sales that consist of counterfeit drugs
- Voice & Accountability: Existence of effective civil society organizations in countries with
mechanisms in place for citizens to express views to government bodies (social responsiveness
and accountability)
- Supplemental: Disparity in coverage between lowest- and highest-income
groups/regions/rural/urban areas
• Country Policy and Institutional Assessment (CPIA) Index – measured by The World Bank and
based on a set of criteria that are captured in 16 subcomponents
• WHO
• MOH
• WHO
• HFS
• MOH
• NHA
• HFS
• Research studies
• MOH
• Transparency
International
• WB, USAID
• WB
Key to Sources: DHS = Demographic and Health Surveys; EMPP = WHO Essential Medicines and Pharmaceutical Policies; HAI = Health Action International;
HFS = Health Facility Survey; IDPG = International Drug Price Guide; MCC/MCA = Millennium Challenge Corporation/Millennium Challenge Account; MICS
= UNICEF Multiple Indicator Cluster Surveys; MOH = Ministry of health; MSH = Management Sciences for Health; NHA = National Health Account; SPA =
Service Provision Assessment; WB = The World Bank; WHS = WHO World Health Surveys

21
Data and Indicators: Limitations, Uses, and Criteria
While identifying key health systems indicators is an important step, equally essential is the availability
of cross-national data, as well as country-specific data, for these indicators. Unfortunately, reliable data
are often not available for many indicators. Several reasons account for this paucity of data. While a
multitude of indicators and tools exist to measure various aspects of health systems performance (see
tables 1 and 2), there is hardly any consensus on a core set of indicators that are measured across
programs and countries. Furthermore, investments in monitoring health systems indicators have been
inadequate, leading to a sketchy information base.
This limited availability of data led to a multipartner effort to develop, fund, and support the future
implementation of a global survey on health systems, such as the Country Health Systems Survey
(CHeSS), that would contribute to strengthening HIS and countries’ capacities to monitor their progress.
This recent effort by WHO and HMN seeks to close the information gap by focusing on data collection
methods that will be useful at the country level. Their objective is to identify methods that provide
reliable data to assess program performance, including district-level health systems surveys and other
health facility surveys. As another part of its survey program, WHO has developed and implemented the
World Health Surveys (WHS) to compile comprehensive baseline information on the health of
populations and on the outcomes associated with the investment in health systems, including data about
the way health systems are currently functioning and the ability to monitor health systems inputs,
functions, and outcomes.
USAID also has made efforts to standardize the measurement of health systems performance, often in
collaboration with WHO and HMN. Considerable effort has already gone into the standardization of
Service Provision Assessments (SPAs), which are implemented by MEASURE DHS. The MEASURE
Evaluation project has published several guidelines and tools that include systematic reviews of health
facility assessments and commonly used performance indicators. In recent years, WHO has expanded its
efforts to encourage the use of data from facility surveys to fill gaps in information on health resources,
including infrastructure, workforce, and service delivery, in many developing countries, thus providing a
comprehensive picture of health systems in the areas of service status and availability of services. Other
health systems strengthening (HSS)-related activities examine best practices and indicators of
performance-based incentives as a means to address the problem of poor quality of services, low
productivity of health workers, dysfunctional management behaviors, and low utilization of essential

22
services by the poor; all contribute to poor health outcomes. The Center for Global Development Working
Group on Performance-Based Incentives has addressed these issues.
When reviewing these indicators, it may be helpful to keep in mind the main criteria for selection. A
review by DfID suggests that an indicator should perform well on dimensions such as data availability,
comparability, ability to collect with accuracy and independence (reliability), relevance to health systems
performance, association with final outcome indicators, possibility of aggregation with other indicators,
and the likelihood that it will distort behavior negatively (Walford, 2007). Another literature review
(including work from WHO) of the top criteria for assessing indicators identifies validity, relevance,
precision (accuracy), reliability, sensitivity, timeliness, and cost, as well as the ability to quantify the
measure, interpret it without modifiers, and collect it often enough for it to have statistical power (Hutton,
2000). Equally important is the need to balance the need for information with the time and cost of data
collection, keeping in mind the consequences on data quality when sufficient resources are not available
(Martinez, 2008). As information on all the above-mentioned criteria are not always available,
recommended indicators also are based on consensus by reviews performed by the major sources of the
indicators. While data constraints for each set of indicators are addressed in the discussion of each
building block, they also will be examined in greater details in the next step in this analysis when specific
indicators will be identified.
The following sections describe in detail health systems indicators for each of the six building blocks of
the WHO framework. Subsections for each building block identify the sources from which the indicators
were taken and summarize the key indicators recommended by these sources. The last section of this
paper presents health output indicators and relevant thresholds that can be linked to the HSS indicators.
These are taken from the MDGs, the CGD’s Global Health Indicators Working Group, and USAID.

23
Part II: Detailed Findings by Health System Building Block
Service Delivery
WHO defines service delivery as the way inputs are combined to allow the delivery of a series of
interventions or health actions (WHO, 2001a). Service delivery is the main function the health system has
to perform, and it is often thought of as the only function of a health system. Service delivery is an
immediate output of the inputs of the other building blocks, such as health workforce, medical products,
and finances (Islam, 2007).
The measurement of service delivery has a more ambiguous scope, in part due to the less quantifiable
nature of service delivery. There is also no one model of good service delivery, which allows for many
variations in different settings. However, WHO defines a list of requirements that a service delivery
system must meet (WHO, 2007). There must be efforts to increase the demand for care through public
health outreach; a package of integrated health services that is offered based on need and availability; an
organized provider network; effective management; and the infrastructure and logistical control to support
providers in supplying health care. Indicators of health service delivery need to measure these various
characteristics, as well as the adequacy of the structure of the service delivery system, while also
collecting indicators of service delivery outputs (Islam, 2007).
Measures of service delivery outputs include access, utilization, and coverage, which indicate whether
people are receiving the services they need (WHO, 2008). Access includes a wide array of measures,
including physical, financial, and sociopsychological access to services.
Physical access to services is often called the availability of services. Availability of services can be
measured based on whether services are available within a certain maximum distance or by availability
per capita. Data on the population distribution of health service resources are required to estimate physical
access, and estimates of types of services rendered need to be reported by facilities (WHO, 2008). Data
about service delivery infrastructure are easier to collect than data about other aspects of service delivery
because they are mostly durable, tangible, and less mobile. Financial affordability of services is covered
in the health finance section. The last dimension of access, sociopsychological access, is considered the
acceptability of the service. Acceptability is an area that is often included as a component of the “quality
of care” measures in service delivery indicator lists.

24
WHO, as well as many other sources cited below, focuses primarily on the physical availability of
services. Ideally, other aspects of service delivery should be covered as well, such as acceptability and
quality of care (safety, efficiency, and effectiveness of selected interventions) (WHO, 2008). Because of
the difficulty in collecting these data, however, fewer indicators are available on these dimensions.
Several sources also include coverage indicators. These types of “output” indicators are not considered by
WHO to be part of monitoring service delivery. Therefore, they are not included here under the health
systems building blocks but rather discussed in the last section, where health systems performance
indicators are linked to these health systems outputs (WHO, 2008).
Recommended Indicator Sources
The WHO Toolkit on Monitoring Health Systems Strengthening includes a list of draft indicators for
service delivery (WHO, 2008). The Toolkit focuses largely on availability and utilization among the
overall population of a country. In addition to the indicators on the number of hospital beds and inpatient
facilities described later, this source also suggests a measure of the number and distribution of health
facilities that meet basic service capacity standards. WHO notes that this measure depends on the
availability of detailed data about basic amenities (access to water, etc.), basic equipment, infection
control, human resources, and tracer drugs and diagnostics. This may be a measure better used within a
country at the facility level as a checklist, rather than aggregated for cross-country comparison because
many countries have moved toward a higher level of service delivery that requires a more sophisticated
tool for measuring differences in service capacity.
The USAID-funded Health Systems Assessment Approach (HSAA) effort of Rational Pharmaceutical
Management (RPM) Plus, Health Systems 20/20, and the Quality Assurance Project (QAP) matches the
WHO framework on several indicators but also includes a greater number of indicators intended for in-
country purposes (Islam, 2007). This tool was designed so that USAID Missions could choose a smaller
set of indicators from a larger list to better match their program activities closely. In addition to health
systems performance indicators, the approach includes many coverage and health outcome indicators,
such as mortality rates, immunization coverage, and contraceptive prevalence that WHO does not
consider part of health systems indicators.
USAID also put together a health systems indicator list for its 2001 graduation report on increased
health promotion and access to quality health care in the Europe and Eurasia (E&E) Region (USAID
Program Objective Team 3.2, 2001). This report used the indicator list to rate the health systems of
countries in the E&E region. Even for this group of relatively more developed countries, there were
significant limitations to health systems data and information. This lack of reliable data resulted in the

25
inclusion of more qualitative indicators in the report. Like the HSAA, the report uses a number of output
indicators for service delivery related to graduation criteria, but it does have one input measure related to
the quality assurance process.
Finally, the ECHIM Project is part of an effort to develop and implement health indicators in Europe and
to develop the European Union (EU) HIS as a whole. Participants in this project include EU member
states, Eurostat, OECD, and WHO. The list of indicators is geared toward evaluating health systems in
more developed countries, but many of the indicators are still relevant and overlap with the health service
delivery indicators designed for the developing world. In its focus on high-level measures of care,
ECHIM largely matches the WHO approach to service delivery indicators. It covers the availability and
utilization indicators in the WHO Toolkit but also go beyond with more precise measures of care, such as
equity of access and disease-specific outcomes (ECHIM, 2008). Many of these specific indicators may
require more data collection than is possible at present in developing countries and thus are excluded from
this review.
Regarding sources of data for the indicators suggested by these assessment tools, the USAID-supported
Service Provision Assessment (SPA), conducted by ICF Macro, is a detailed and reliable source of
service delivery data. This is a respected source for this type of data because of the relatively large
number of facilities that are included in its sample, which covers all facility types in a country. The
downside is that samples for the client exit interview are small. Moreover, SPAs are not conducted in
nearly as many countries as DHS or National Health Accounts (NHAs). In addition, client exit interviews
will only capture the portion of the population who chose to receive care and therefore miss vital
information about the reasons why people choose not to seek care when needed.
In addition to SPAs, there are a few other sources of service delivery data. WHO’s Statistical Information
System (WHOSIS) collects data on hospital beds in each region through its core indicators. Some of the
WHO regions also collect outpatient visit data, but that information is not reported in WHOSIS. Data
about health care facilities might be obtained from the ministry of health (MOH) in each country. Some
service delivery data also can be found in The World Bank’s Quantitative Service Delivery Surveys and
Public Expenditure Tracking Surveys, although these surveys have only been conducted in a few
countries.

26
Most Recommended Service Delivery Indicators
Besides a general agreement on the need for quality-of-care indicators, which are discussed below,
several specific service delivery indicators were recommended by many of the aforementioned sources.
• The number and distribution of inpatient beds per 10,000 population is the single most-
mentioned indicator by major sources. This indicator can serve as a proxy for the availability of
health services, where more direct measures are not available. It also provides information on
health care institutions’ capacities for care and their resource use (ECHIM, 2008b). It can be used
to assess the adequacy of beds in relation to the population and as a measure of whether inpatient
services are even available in the most resource-poor settings. This indicator measured relative to
population size better represents the adequacy of services available as compared to a measure of
the total number of beds in the country. There is no specific threshold for this indicator; generally,
a greater number of hospital beds suggest greater availability of inpatient health services.
• Number and distribution of health facilities per 10,000 (or 1,000) population is mentioned by
several sources, but each has its own variation on the measure. USAID’s HSAA chose this
indicator but suggested measuring only primary care facilities, not all health facilities. A 2008
literature review cites this measure in terms of 1,000 population, not 10,000 population (Kruk &
Freedman, 2008). Regardless of the permutation, the rationale behind this measure is that it
collects data on the ratio of health resources to the total population. It is more rudimentary than
the first indicator and more easily measurable. There are few benchmarks available for this
indicator; a comparison with regional or peer-country averages may be the most useful way to
gauge it.
• Measuring basic service capacity is an area highlighted by WHO in the 2008 Toolkit, suggesting
a measure based on an index of availability of basic amenities, basic equipment, infection control,
health workers, and tracer drugs and diagnostics in a facility. This idea has been echoed by other
sources, although how they define the index and what they include vary by source. USAID’s
HSAA looks only at primary care facilities and states that country standards should dictate the
minimum equipment that facilities at each level of care should have available. This tool
recommends that “the standard should be obtained directly from the MOH and may include
standards or conditions other than presence of certain equipment (e.g., materials, electricity,
running water, and laboratory services), in which case this situation should be explained.”
Walford also notes that WHO supports the use of management capacity indicators but comments
that there is limited availability and comparability across countries, due to the wide variations in

27
what can be considered basic service capacity, and suggests that it would be best to choose these
indicators in country (Walford, 2007). The SWEF Research Network does not suggest an index
but gives a specific indicator relating to the number of new services offered by type of facility. It
also asks about tracer drugs, but does so in the pharmaceutical section. Collecting data on service
capacity is important to ensure that the full range of services is available to clients. The absence
of capacity standards at the MOH would indicate lack of management capacity for the health
system.
• WHO also suggests another basic service capacity indicator, this one more specific to a disease,
such as malaria control, Integrated Management of Childhood Illness, safe motherhood, family
planning, HIV/AIDS, control of sexually transmitted diseases, TB control, or control of
noncommunicable diseases. These can be measured on their own or in any combination, given the
type of facility that is being evaluated. It suggests that a short set of indicators should be devised
to measure all the areas mentioned above and suggests indicators for the areas of services offered,
staff and training, equipment, diagnostics, and treatment/prevention for each disease. USAID’s
HSAA does not include such an indicator but does ask about the availability of updated clinical
standards for MOH-priority areas, high-burden disease areas, and/or areas responsible for high
morbidity and mortality. The modularity of this indicator would allow for other high-risk diseases
specific to a particular country, such as neglected tropical diseases, to be added. Both of the basic
service capacity indices require detailed facility-level information, which increases the difficulty
of collecting these data. A facility audit such as the SPA would be the best source for such
information.
• The only utilization measure to be cited across multiple sources is the number of outpatient
department (OPD) visits per 10,000 population per year. In several countries, OPD visit rates
significantly increase when constraints to using health services are removed, suggesting this is a
good proxy indicator for improved access to care, although only for countries that start with very
low rates (WHO, 2007). This measure also can suggest a basic level of functioning in the health
system, if poverty is not an overriding concern, because use of OPD visits would stay low despite
increased per capita income if services are poor or staff are not present. It is important to note the
interaction of this indicator with inpatient beds/visits; in some cases, OPD visits can increase
because of constricted supply of inpatient services, reflecting the substitution of ambulatory
procedures for inpatient admission. This is not necessarily desirable. However, in most
developing countries, especially those where OPD visit rates are strikingly low (such as Uganda,

28
Burundi, and others), increases in OPD visits would mean increases in primary health care use, so
rises in OPD visits would suggest positive movement (Sjönell, 1984).
• Quality of care is an area characterized by lack of clarity on indicator definitions. Many sources,
including WHO, suggest there are a wide range of indicators to choose from, and each situation
may call for different combinations. The WHO-recommended list includes a variety of indicators
to choose from, such as providers questioning patients about medications, providers’ knowledge
of hand hygiene/safety procedure, providers adhering to treatment protocols, patient success at
seeing a provider before having to leave a facility, etc. (WHO, 2008). USAID’s HSAA also
includes a secondary group of indicators that delve more deeply into quality-of-care measures for
health service delivery (Islam, 2007). Walford’s review for DfID suggests an indicator to cover
patient satisfaction and/or wait times (Walford, 2007), but further work is needed to create a
usable indicator from this guidance. Kruk and Freedman cite quality-of-care indicators most often
used in the literature: providers treating patients with respect, the quality of physician-patient
communication, length of wait for care, use of evidence-based diagnostics and therapies, and the
rate of avoidable hospitalization (Kruk & Freedman, 2008). RAND Health’s Quality of Care
Assessment Tools is the most well-documented and verified listing of indicators for this area
(McGlynn, et al., 2000), but, like ECHIM’s more specific indicators, many of these measures are
more suitable for the more advanced health and information systems of Eastern Europe and Latin
America than for those in the more resource-poor settings of sub-Saharan Africa and South Asia.
The context of each evaluation will have to be considered when choosing which quality-of-care
indicators to include.
Issues and Concerns
A major consideration with all quality-of-care and service capacity indicators is again data availability.
Particularly in the case of measuring quality of care, there is a need to consider these indicators in the
context of evidence-based guidelines. Not only are these indicators not widely collected at present, some
are also very subjective, and poor collection could result in noncomparable or, at worst, useless data.
Furthermore, little data on newly emerging diseases may be available or, when available, are limited to
endemic areas. For all of these indicators, a major stumbling block will be to get data for not only a
country’s public facilities but also its private facilities. In addition, significantly more value would be
gained by gathering these data by facility type, so that these indicators could be disaggregated. For
example, it would be very useful to know whether the majority of OPD visits are occurring in rural
primary care clinics or in hospital OPDs. Currently, few sources call for this level of detail to be reported.

29
Health Workforce
The ability of a country to meet its goals in health depends on the workforce responsible for organizing
and delivering health services (WHO, 2008). They are the gatekeepers to health. There is ample evidence
that the size and quality of the health workforce are positively associated with immunization coverage,
outreach of primary care, and infant, child, and maternal survival, among other outcomes (WHO, 2006a).
Unfortunately, there is wide variation in the type, skill, and gender mix in the health workforce across and
even within countries (WHO, 2007). This can create chasms in care where the workforce is not sufficient
to produce positive gains in health outcomes. Reasons for workforce shortages include geographic biases,
migration of health workers within and across countries, poor mix of skills, lack of domestic training
capacity, HIV/AIDS and other pandemic-related deaths, and demographic imbalances. Better knowledge
of the composition of the health workforce will help countries optimize their use of what is currently
available and also plan for changes to be made in the future.
According to WHO, the health workforce includes not only clinically trained health professionals but also
nonclinical health management and support workers. They serve in both the public and private sectors.
Each country has different workforce needs, which will have to be estimated while taking into
consideration the limitations of range, skill mix, and demographics of the currently available workforce.
Among nonclinical health workers, there has been increased interest in those working in health systems
research. Consensus among developing countries at the 2008 World Health Assembly was that health
systems financing, policy, and management experts were “extremely needed, and that there was a strong
demand in the local job markets for such workers” (Rockefeller Foundation, 2008). In order to measure
this section of the workforce, there are several hurdles to overcome. First, there is no concrete definition
of what constitutes a health system professional. Second, compared to clinical fields, there are fewer
professional associations, universities, and international groups who track these types of professions,
specifically in health. There are efforts under way to improve this, with WHO and the World Federation
of Public Health Associations working on a global mapping of public health schools, institutes, and
associations. WHO also is partnering with the University of Copenhagen on the Avicenna database, a
global directory of education institutions for health professionals. Until these tools are fully available, and
a definition of health systems workers is agreed upon, any indicators related to measuring this section of
the workforce should be interpreted with caution.
Compared with the other building blocks, fewer indicator sources deal with health workforce issues as a
separate area of health systems. Some address them under service delivery, while others are concerned

30
primarily with the financing and planning for human resources for health. Still other sources focus on
health workforce management, which is more accurately measured at the facility level. WHO expects
workforce indicators to cover whether human resources are allocated fairly and efficiently; whether
allocation is responsive to change; and whether there are sufficient resources to achieve the best health
outcomes. They also mention that the health workforce should be competent, responsive, and productive
(WHO, 2007).
Because of workforce mobility, it is sometimes difficult to collect accurate information on specific cadres
of health workers. Where there are domestic institutions for training health professionals, it may be more
feasible to collect exit data on the number of graduates, although this will not take into account
individuals who leave the country or who do not go on to practice. It also will not count health workers
who are trained in other countries but migrate after school. Data on the competency and responsiveness of
workforces in developing countries are even harder to find, in part because of the ambiguous definition of
these measures but also because it may be difficult to separate provider shortcomings from shortcomings
of facilities, the supply pipeline, etc. On this subject, there is significant overlap with the service delivery
building block.
Recommended Indicator Sources
Most sources deal with indicators pertaining to the number and distribution (density) of health
professionals rather than the quality of the workforce. The WHO Toolkit on Monitoring Health Systems
Strengthening devotes the majority of its indicators to this type of measure. Its listing includes both
general and cadre-specific worker counts per 10,000 population, along with the density of workers by
various subcategories (region, profession, sex, place of work). The suggested indicator list in a review for
DfID largely follows this approach (Walford, 2007). This is likely because density indicators are already
widely available, whereas quality indicators are harder to collect.
ECHIM’s workforce indicators are included under its health services category (ECHIM, 2008). It also
focuses on the number and distribution of the workforce. It includes subdivisions within professional
groups (for example, number of physicians practicing versus not practicing). For countries where it is
feasible to collect these data, it would be useful to do so.
USAID’s HSAA also includes density measures but devotes more time to the planning and financing of
the human resources system (Islam, 2007). Its indicators deal with topics that other sources may define as
part of information systems or finance. Its secondary component contains some indicators that are more

31
appropriate for collection at the facility or training institution level rather than the national level. For
instance, it asks about the incentives and benefits available to staff, in-service trainings, and the review
process; these may vary widely from facility to facility and between private and public employers. To
report one national-level result would be uninformative.
The SWEF Research Network is the only major source that includes indicators that could be considered
quality indicators. These indicators deal with periodic training of health workers and supervision. These
indicators tend to rely more on data collected by The Global Fund and/or by human resource information
systems, both of which have serious problems as sources of publicly available national data.
More promising sources of health workforce data include WHOSIS and the World Development
Indicators Database, both of which have many measures of density for various health professions,
although in developing countries, some may not be reported as frequently as others. USAID’s HSAA
suggests using MOH data, provider surveys, private provider groups, and United Nations agencies
in country to get rural and urban density data, but any other within-country disaggregation of data on
workforce will be more difficult to collect, with data quality suspect in more remote areas. Countries with
SPAs will have more complete information.
Most Recommended Health Workforce Indicators
There is only one indicator that is mentioned in more than two major sources: a measure of density of the
health workforce. There are, however, two other thematic groups that are often mentioned, although a
specific indicator will have to be worked out to make them measurable.
• Number of health workers/health professionals per 10,000 population/per capita/per 1,000
population is the most often cited indicator for health workforce. This indicator is a necessary,
but not sufficient, measure of coverage (HMN, 2006). There are many variations on this indicator,
so further research may be needed to decide which combination of options best fits the indicator
selection criteria. Some sources allow for all health workers, clinical and nonclinical, to be
included (WHO, 2008). More of these sources call for aggregate totals of clinical staff only, while
others require this measure by profession, i.e., physicians, nurses, midwives, etc. More data are
generally available on the supply of physicians, so they may be used as a “tracer” population, if
needed. Because very few developing countries suffer from oversupply of health professionals,
any upward movement of this indicator would be considered a positive movement. WHO has
found that a minimum of 2.3 physicians, nurses, and midwives per 1,000 population are needed to
meet 80 percent coverage of skilled birth attendance (SBA), and cites another study that found a

32
similar threshold of 2.5 per 1,000 to reach 80 percent coverage for SBA or measles vaccination
(Chen, et al., 2004; WHO, 2006a). These numbers have been cited by other sources as a general
workforce threshold, but perhaps more work is needed to test for additional basic output
indicators other than SBA and measles. Individually, countries can develop rough estimates of
optimum and minimum workforce needs using WHO’s staffing requirements model (Hall, 2001).
• Distribution of health workforce is an area that many sources touched on with various
indicators. Distribution is an important modifier to the density indicator, which can be deceiving
if distribution is not taken into account. Some suggest measuring rural versus urban densities of
health professionals. However, this may not measure geographic barriers or other impediments to
care. Another group of sources suggests measuring densities by region and place of work, which
give more detailed information but may be harder to collect. The goal would be to decrease
disparities in density per capita/per X population across measurement units.
• Domestic education of health professionals is another thematic area for which many sources
included indicators, although there is no agreement on how to measure them. The WHO Toolkit
suggests measuring the annual number of graduates of health professions’ educational institutions
per 100,000 population, by cadre. Several other sources suggest an indicator to capture the
number of graduates from health professional schools, but this type of data is sometimes hard to
capture even in developed countries and nearly impossible to collect in more resource-poor
settings (WHO, 2006a). It is also important to note that the number of graduates does not directly
translate into the number of new health workers in a country. For instance, WHO found that a 10
percent increase in the entering class of medical students will result in only a 2 percent increase in
the supply of credentialed doctors in 10 years (WHO 2007a). Another source suggests an
indicator relating to the number of health professional schools in a country. This indicator is more
easily collected but gives little useful information.
• Indicators to monitor national workforce management. The WHO Toolkit gives additional
“output” indicators related to the above input indicators. These measure the government outputs
that would produce measures of density. They note that the list is neither exhaustive nor absolute,
but they do point out the value of having indicators that can be disaggregated to monitor health
workforce management at the country level. USAID’s HSAA includes some additional national
workforce management indicators, but further research will be needed to merge the two different
groups of measures.

33
Issues and Concerns
One concern with measurement of indicators related to the health workforce is whether workers who are
not part of the formal health system should be included in the calculation of the indicator. For instance,
community health workers play an important role in influencing health outcomes but are not often
included in the measurement of the health workforce. Similarly, there may be others who are not formally
trained and hence excluded from measurement, even though they play a key role.

34
Information
Reliable and timely health information is an essential foundation of public health action and health
systems strengthening, both nationally and internationally. The main objective of health information is to
provide data to improve health service delivery and provide evidence for policy decisions that will lead to
improved health status of the population. A well-developed information system will include all
information relevant for health decisionmaking, including financial, programmatic, and geographic
information about health services. The need for sound information is especially urgent in the case of
emergent diseases and other acute health threats, where rapid awareness, investigation, and response can
save lives and prevent broader national outbreaks and even global pandemics (HMN, 2008). The
generation and strategic use of information, intelligence, and research on health and health systems are
also integral parts of the leadership and governance function (WHO, 2007). Monitoring of the information
“building block” therefore needs to capture the generation of data, the quality of the data produced, the
creation of information, and the application and synthesis of information into knowledge for
decisionmaking that improves health systems operations (Alliance for Health Policy and Systems
Research, 2008). Each of these elements can be assessed as part of information performance monitoring.
Much emphasis has been placed on information systems that produce data, while, more recently,
monitoring and evaluation (M&E) groups and academics have examined the application of information in
decisionmaking, as well as the financial and human resources allocated to information systems and
application (Hanney, et al., 2003; Islam, 2007; Stansfield, et al., 2006).
A good information system has four main functions – namely, to generate, compile, analyze and
synthesize, and communicate and use health data – to help in the decisionmaking process (WHO, 2008).
Publicly funded health programs are increasingly being asked to account for their performance, and
information systems provide a means of operationalizing this quest for accountability (Perrin, et al., 1999)
Their main objective is to produce information for improved health services and evidence-based policy
decisions that will lead to improvements in the health status of the population. Although they do not
directly improve or reduce health status, reliable and timely health information is an essential foundation
of public health action and health systems strengthening, both nationally and internationally. Information
is therefore used as a management and oversight tool to improve outcomes through the analysis of
changes in outcomes, as well as the processes and capacity being applied to achieve the outcomes (Perrin,
et al., 1999).

35
According to WHO, the key components of the information building block include the generation and use
of information; development of health information and surveillance systems; development of standardized
tools and instruments; and collation and publication of international health statistics (WHO, 2007). These
components should deliver timely and accurate reporting of population- and facility-based data that
support decisionmaking and, where relevant, be utilized at the point of collection. The information system
should also be able to detect events that threaten public health security so that authorities can investigate
and contain the threats. The data provided by information systems also can draw attention to systems that
are performing well or populations that are achieving better health as well as to areas in need.
Information systems include health information systems, which provide data on health status and
integrate data collection processing, reporting, and use of the information necessary for improving health
service delivery through better management, and health management information systems (HMIS),
which are specially designed to assist in the management and planning of health programs, as opposed to
the delivery of care. According to the WHO Toolkit, indicators of country HIS performance can be
grouped into two broad types, namely:
• Indicators related to data generation using core sources and methods (health surveys, civil
registration, census, facility reporting, health system resource tracking). These indicators reflect
country capacity to collect relevant data at appropriate intervals.
• Indicators related to country capacities for synthesis, analysis, and validation of data. These
indicators measure key dimensions of the institutional frameworks needed to ensure data quality,
including independence, transparency, and access.
Core sources of information include:
• Population-based data relating to the whole population, not only to groups using health facilities.
Such data can be gathered continuously from administrative records, such as census or birth
registries, or, periodically, through cross-sectional household surveys (HMN, 2008).
• Institution-based data that produce data from administrative and operational activities. These
include patient records, which contain clinical information obtained in the course of providing
health care, and administrative data collected as part of the operation of a program, such as billing
information or number of patients served.
Different data are needed at different levels of the health system. At a lower level, data regarding a patient,
often presented in patient charts, are needed for patient management. At the facility and district levels,

36
summary indicators are needed for management, planning, and procurement purposes. Indicators also are
needed at the district level for planning and reporting to the national level. The national summary
indicators are then used for the governance of the health system and regional or global reporting (Islam,
2007). The guiding principles of information systems are that no health data should be requested from a
service level to be reported to higher levels that do not have an actionable use at the recording level;
health data should be used to analyze and solve important health and service problems; and priority
attention should be given to improving data generation and use at the local level, to support the
enhancement of service performance at that level (WHO, 2000).
Recommended Indicator Sources
The WHO Toolkit identifies 29 indicators across six categories: country health surveys, birth and death
registration, census, health facility reporting, health system resource tracking, and capacity for analysis,
synthesis, and validation of health data. These core indicators assess the strength of the system based on
the presence and quality of the data generated using core sources and methods (health surveys, civil
registration, census, facility reporting, and databases), the reporting of data, and the country’s capacity for
analysis, use, and dissemination of health data. The Toolkit also proposes an overall health information
system performance index (HISPIX), a summary measure based on the 29 indicators that captures overall
data quality and overall health information system performance. The sources of the data for the indicators
in the Toolkit and for HISPIX are, for the most part, available within WHO databases and from other
international agencies. However, the draft Toolkit has not yet identified sources for all proposed indicators.
WHO/HMN and its partners have developed the Health Information System Assessment Tool, a self-
assessment tool that has been completed by 68 countries to date, with several more awaiting validation
(HMN, 2008a). HMN states that the self-assessment approach appears to have worked well, although it
will lead to comparability issues. The tool contains more than 125 indicators, also grouped in six areas
similar to the WHO draft Toolkit: health information system resources (policies, human resources,
financing, and infrastructure); indicators; data sources; data management; information products; and
dissemination and use. Scenarios are provided that allow for objective rating of the system. From these
ratings, an overall measure is calculated.
In its Framework and Standards for Country Health Information Systems, HMN describes a minimum set
of core health indicators that all national health information systems should be able to report (HMN,
2008). The Framework points to the indicators used for World Health Statistics 2005, which were
developed in part to use to monitor WHO’s efforts to strengthen health information systems. WHO aims
for all countries to have systems in place to be able to report on these core indicators, and the extent to

37
which a national health information system can do so is a measure of the system’s quality. The 39 core
indicators reflect determinants of health, health status, and health systems, and were selected based on the
availability and quality of data. They include a number of indicators to measure progress toward the MDGs.
The Working Group on Health System Metrics, which involves HMN, WHO, The World Bank, The
Global Fund, and the GAVI Alliance, has chosen one indicator for information systems: the percent of
births and deaths registered.
WHO also has a manual titled Developing Health Management Information Systems: A Practical Guide
for Developing Countries to help program managers monitor the establishment and functioning of HMIS.
It contains questions and indicators designed to assess data collection processes and identify issues with
the current information system, including issues with the use of information. It also demonstrates how to
identify indicators that the HIS should collect based on the national health strategy for a given area, such
as child health or HIV/AIDS (WHO/WPRO, 2004).
The Global Fund’s Monitoring and Evaluation Toolkit (second edition) provides four examples of
information system indicators. Two indicators relate directly to The Global Fund goals: 1) staff and civil
society training in M&E and 2) the presence of a nationally coordinated multiyear plan, with a schedule
for survey implementation and data analysis prepared (The Global Fund, 2006).
Health Systems Assessment Approach: A How-To Manual, developed by the USAID-funded Health
Systems 20/20, Partners for Health Reform Plus (PHRplus), RPM Plus, and QAP projects, classifies 26
HIS indicators in seven topical areas: health status indicators for both mortality and morbidity; health
system indicators; resources, policies, and regulations; data collection and quality; data analysis; and use
of information for management policymaking, governance, and accountability. It provides the definition
and rationale for each indicator, as well as the sources and caveats. The first seven indicators cover health
status and systems (different from the WHO Toolkit). These represent data collection for commonly
agreed upon indicators of health status and systems. For example, unlike other HIS assessments, it does
not measure the existence of survey or vital registry to measure mortality or HIV prevalence but rather
uses the indicator itself (under-5 mortality). Whether a given country’s HIS has collected and reported
these agreed-upon indicators of health indicators is a basic level of function and capacity, and a lack of
current data for these critical indicators would also imply serious weaknesses in the information system.
The next 19 indicators are concerned with resources and policies (5); regulation of information systems
(6); data analysis (6); and use of information for management, policymaking, governance, and
accountability (2) (Islam, 2007).

38
USAID indicators for its Strategic Framework for Foreign Assistance Operational Plan (Department of
State [DOS], 2006) include several indicators that are designed to measure progress in capacity building
of information systems within each element. However, they capture only activities conducted by the U.S.
Government and do not measure the strength of national information systems. They measure such things
as individuals trained in strategic information activities, including M&E; surveillance; the number of
host-country institutions with improved management information systems; the number of monitoring
plans or number of sector assessments conducted by the U.S. Government.
Other relevant indicators include The World Bank’s Statistical Capacity Indicator, which is a summary
measure that provides an overview of the statistical capacity of developing countries. It was developed to
assess the capacity of statistical systems using metadata generally available for most countries. The
quality of the statistical system is also important (de Vries, 1998). The Performance of Routine
Information System Management (PRISM) framework developed by MEASURE/Evaluation and John
Snow, Inc. (JSI) helps managers assess the performance of HIS in terms of data quality and continuous
information use.
No information indicators are explicitly mentioned by the SWEF Research Network, The World Bank’s
results-based financing in health project, which plans to link funding to performance on five standard
indicators, or the CGD Global Health Indicators Working Group. Countries also use a number of
measures to monitor performance-based funding schemes, such as the percent of districts with disease
surveillance reports, the proportion of districts submitting monthly reports to MOH on time, or the
proportion of research findings translated into policy and practice. However, these are extensive, vary
widely, and are country specific.
Most Recommended Information Indicators
Based on this review, several indicators are recommended by multiple organizations. It is important,
however, that indicators be chosen according to their purpose.
• HISPIX. The WHO Toolkit proposes a health HISPIX, a summary measure based on a set of 29
standardized indicators for assessing data quality and the overall performance of the health
information system. The score is calculated from information available from WHO databases and
other international agencies. Information on inputs and resources is available from country health

39
statistics reports and from the self-assessments conducted through HMN. The indicators below are
part of the 29 standard indicators.
• Existence of demographic or household surveys – two or more data points available for child
mortality in the past five years. This indicator measures the existence of demographic or
household surveys and is relevant only to countries without complete civil registration systems (less
than 90 percent coverage of births and deaths), based on whether the country has had more than one
demographic or health survey. Data quality and comparability may be an issue. Because the
indicator captures the existence of the survey and not the survey quality, it cannot be used to
compare issues across countries. The indicator has a threshold of two or more surveys within the
past five years.
• Percent of births and deaths registered in the country. This indicator measures the functioning
of the vital registration system and the availability of key indicators at the national level. The
numerator is civil registration records and the denominator is the census.
• Percentage of districts that submit timely, complete, accurate reports to national level. A
variation of this is “health facilities or districts reporting all indicators according to national
guidelines.” This is a measurement of the comprehensiveness of health information management
and capacity at the lower levels to provide the data. It also demonstrates the extent to which
information is available on a specific disease or service statistic. Sources are administrative records
aggregated at the national level.
• Completion of at least one NHA in last five years. The existence of an NHA demonstrates (to
varying degrees) health systems resource tracking, health systems governance, and decisionmaking.
NHAs provide managers with reliable national information on the sources and uses of funds for
health. Governments in countries with NHAs can use them to understand how resources are
mobilized and managed for the health system; who pays and how much is paid for health care; who
provides goods and services; and how health care funds are distributed across the different services.
• The existence of designated mechanisms charged with analysis of health statistics, synthesis of
data from different sources, and validation of data from population and facility sources. The
definition of this concept varies. The HSAA describes it as “the presence of mechanisms to review
the utility of current information system indicators for the planning, management, and evaluation
process, and to adapt and modify accordingly” (Islam, 2007). The Global Fund describes it as “a

40
nationally coordinated multiyear plan with a schedule for survey implementation and data analysis
prepared” (The Global Fund, 2008). The underlying rationales are similar: to provide some
indication of the use of data in program planning.
• Availability of a national summary report that contains HIS information, analysis, and
interpretation (most recent year). This is an indicator of the capacity for analysis, synthesis, and
validation of health data. It also provides a proxy for the national commitment to transparency and
data dissemination, and provides stakeholders the opportunity to engage and take action together.
The main difference between the HISPIX and the HMN self-assessment score is that the HISPIX
indicators can be assessed externally on the basis of information largely available in the public domain,
while the HMN is a self-assessment. However, indicators of health systems resources are not widely
available and used by most agencies. The HMN assessments are one of the few sources available that
capture the details of health information system resources and data management and quality. The WHO
Toolkit, HSAA, and HMN all recommend indicators for data use and analysis. The extent to which “use”
is measured varies widely. The Toolkit includes the indicator mentioned above (“the existence of
designated mechanisms charged with analysis of health statistics, synthesis of data from different sources
…”), as well an indicator on immunization data validation that does not actually measure data use but
rather the establishment of mechanisms to do so. The HMN assessment includes 10 indicators to measure
use of data for planning and resource allocation, but countries will vary in their assessment of
demonstrated use of data.
Other relevant indicators not widely included for monitoring information system performance are
indicators of health systems resource tracking. These indicators, where available, indicate the level of
support the government provides to the HIS. Another area that varies includes indicators concerning data
quality assessments. WHO is one of the few sources to establish a target for this, specifying that data
quality assessments should be carried out and published within the last three years and that assessment
should cover all routine data sources.
One indicator that may overlap with the medicines building block of the framework is the existence of
information on the availability of tracer medicines and commodities in facilities. Similarly, the indicator
of the existence of a national database for human resources may overlap with the human resources
building block.

41
Issues and Concerns
The lack of data availability or an information system itself is an indication of a country’s HIS situation
and may be useful in determining system needs and where capacity building should occur. Furthermore,
several HIS indicators, such as the existence of two or more data points on child mortality, are not a
continuous measurement, as in health status, but rather categorically defined as yes or no.
Information systems also are difficult to compare across countries, as rarely has the same assessment tool
been applied to all countries, and even when assessed on the same indicator, data quality may be an issue.
Often HIS indicators are suited to the local context; data needs will vary according to the epidemiological
profile and development (HMN, 2008a). HIS indicators also are difficult to include in progress
monitoring, as they lack a proximity to better health outcomes, and progress in information systems is
difficult to associate with progress toward better health or reaching MDGs (Walford, 2007).
Differences in health systems, the level of decentralization, and the various economic and social and
political contexts also affect the nature and effectiveness of the information system. The World Bank
attempts to compare the quality of national information systems through its Statistical Capacity Index.
However, the Index does not include health facility reporting and data utilization.

42
Medical Products, Vaccines, and Technologies
A well-functioning health system ensures equitable access to medical products, vaccines, and
technologies of assured quality, safety, efficacy and cost, and their sound and cost-effective use (WHO,
2007). The lack of affordable access to lifesaving medicines contributes to enormous inequities in health
between developed and developing countries and leads to preventable mortality and morbidity, episodes
of catastrophic illness that cause further impoverishment, low quality of life, and large-scale economic
and health system losses. Economic factors are frequently the most important barriers to access.
Measuring and understanding the reasons for prices of medicines and how the price affects consumption
of health care are important in helping countries develop and establish policies that ensure the
affordability of medicines. Yet cost is only part of measuring access to medicines.
According to WHO, access to and appropriate use of effective medicines require a complex and
coordinated system that has five components. It must encompass:
• National policies, standards, guidelines, and regulations that support policy and evidence-based
selection of medicines, vaccines, and technologies according to international standards
• Information on prices, international trade agreements, and capacity to set and negotiate prices
• Reliable manufacturing practices and quality assessment of priority products
• Procurement, supply, and storage and distribution systems that minimize leakage and other waste
• Support for rational use of essential medicines, commodities, and equipment through guidelines and
strategies that ensure adherence, reduce resistance, and maximize patient safety and training
Monitoring of this building block is closely intertwined with at least two other building blocks –
leadership and governance, and information – and, where rational use is concerned, the service delivery
building block.
Recommended Indicator Sources
The indicators that measure access to medicines in the WHO indicator Toolkit are constructed around the
MDG 8 indicator “Proportion of population with access to affordable essential drugs on a sustainable
basis.” This was created to monitor the goal of providing access to affordable essential drugs in
developing countries, in cooperation with pharmaceutical companies (United Nations, 2008). The target is
formulated to capture the extent to which the pharmaceutical sector is contributing to providing access.
The contribution of the pharmaceutical sector to access, affordability, and sustainability components of

43
the indicator depends on both domestic and international factors. However, the data needed to form the
MDG indicator has not been collected regularly, and the lack of comparable data is a significant problem.
Given the limitations of the MDG indicator, WHO has broken it down to two components, namely the
structure component (policy and legal provisions) and the process component (cost, supply, and use),
which contain nine indicators. The nine indicators include measures of national legislation and policies,
insurance coverage, cost, and availability of medicines. The Toolkit does not include indicators on
rational use of medicines. The Toolkit itself specifies two indicators. The first is the percent of facilities
with tracer medicines and commodities in stock on the day of visit and in the last three months. A
supplemental indicator is the median proportion of tracer medicines and commodities in stock on the day
of visit and in the last three months. Data for this indicator come primarily from facility visits. However,
comparisons across countries may vary based on the epidemiology. The second indicator compares the
median local medicine price to the international reference price for a core list of drugs, which also
requires the availability of facility data.
The only indicator included in the 2007 draft monitoring plan of the Working Group on Health System
Metrics is the availability of lifesaving medicines, including those for TB, antiretroviral HIV/AIDS
treatment, and malaria (Walford, 2007).
The SWEF Research Network included in its list two frequently cited indicators – facilities without
stockouts and price paid compared with the international price – but also includes the percent of facilities
with expired items, an indicator of quality (SWEF Research Network, 2003). The set included an
indicator on affordability (average drug cost per encounter), which would require including a comparison
to wages or purchasing power.
USAID’s Health Systems Assessment Approach: A How-To Manual contains 39 indicators related to
pharmaceutical management, including national policies regarding medicines, indicators of availability,
and procurement and financing (cost) arrangements (Islam, 2007). However, it provides more detailed
indicators of procurement, pharmaceutical registration, mechanisms for licensing providers, appropriate
use, and quality than the WHO Toolkit. It also includes indicators of private and public expenditures per
capita on pharmaceuticals and total expenditures on medicines as a percentage of total health expenditures
(THE), the latter being a measure of the significance of pharmaceutical spending relative to other
spending on health and of financial and institutional sustainability.

44
In 1999, the WHO Action Program on Essential Drugs, now the Essential Medicines and Pharmaceutical
Policies Department, issued Indicators for Monitoring National Drug Policies: A Practical Manual
which proposes a set of 31 background indicators, 50 structural indicators, 38 process indicators, and 10
outcome indicators (WHO, 1999). These indicators are intended for self-use by developing countries to
monitor their pharmaceutical systems. The manual allows a country to evaluate the performance of the
pharmaceutical sector, monitor progress in the implementation of national drug policies, and assess the
effects of changes on drug policy objectives.
The draft WHO Medicines Strategy (2008–2013) aims to improve medicine-related progress in three
areas: policy and access, quality safety and efficacy, and cost-effective use (WHO, 2008a). It measures
country progress on four indicators that are assessed from eight specific expected results. The four
country progress indicators are access to essential medical products and technologies; availability of and
median consumer price ratio for 30 selected generic essential medicines in the public, private, and
nongovernmental sectors; national regulatory capacity; and percentage of prescriptions in accordance
with current national or institutional clinical guidelines.
To monitor the progress of efforts to improve the global medicines situation and measure important
aspects of country pharmaceutical situations, WHO has developed Using Indicators to Measure Country
Pharmaceutical Situations: Fact Book on WHO Level I and Level II Monitoring Indicators (WHO,
2006). Level I indicators measure the existence and performance of key national pharmaceutical
structures and processes. Level II indicators measure key outcomes of these structures and processes in
the areas of access, product quality, and rational use. These indicators can be used to assess progress over
time, compare situations across countries, and reassess and prioritize efforts based on the results.
The Level I and Level II data are gathered through surveys. Level I is a questionnaire completed by health
officials, which leads to several data limitations. The interpretation of questions and validation of data are
issues, and respondents may not have the resources to provide accurate responses (Carandang & Pierre-
Jacques, 2005). Level II indicators are completed through facility and household surveys and have been
completed on a smaller sample of countries. It contains 13 questions on pharmaceutical-seeking behavior
and affordability. WHO conducted a survey of Level I indicators in 2003 and Level II indicator surveys
between 2002 and 2004. A key finding was that most countries have a national medicines policy, but few
monitor the policy with indicators. It also was found that many basic policies fundamental to promoting
rational use of medicines are not being implemented.

45
WHO’s Action Program on Essential Drugs conducted a global survey (Public Education in Rational
Drug Use: A Global Survey), on public education interventions in rational drug use (Fresle & Wolfheim,
1997). The rationale for these indicators is the need to evaluate such interventions by measuring
consumer’s information and education on medicines and appropriate treatment-seeking strategies.
Rapid Pharmaceutical Management Assessment: An Indicator-based Approach is a manual that
presents an indicator-based approach for rapidly assessing pharmaceutical management systems and
programs, and contains indicators similar to the WHO Level I and II indicators, although it covers a
broader range of topics (MSH/URC, 1995). It presents a set of 46 indicators of performance, grouped
under eight topics of pharmaceutical management (policy, legislation, and regulation; essential drug lists;
health budget and finance; pharmaceutical procurement; pharmaceutical logistics; patient access and drug
utilization; product quality assurance; and private sector pharmaceutical activity).
The ECHIM Project comprehensive indicator list (ECHIM, 2008a) and the European Observatory
(EO) Health Systems in Transition: Template for Analysis (Mossialos, et al., 2007) also contain
indicators that intend to provide a detailed description of a health system. Both of these sources are geared
toward evaluating more developed-country health systems and are mainly concerned with expenditures
for pharmaceutical products and the appropriateness of prescribing practices in developed countries. The
EO asks about the government’s ability to regulate providers. It includes regulatory indicators that could
be applied for more developed systems, including regulations on alternative complementary medicines
and malpractice. The EO template is one of the only sources to include indicators of patient education.
Other sources containing information on medicines include JSI’s Logistics System Assessment Tool
(LSAT) (DELIVER, 2009), which is a qualitative data collection instrument that provides a
comprehensive system-level assessment of logistics system performance for any program that manages a
health commodity. The indicators can be used to diagnose problems in drug supply systems, management,
or procurement. USAID’s Strategic Framework for Foreign Assistance Operational Plan (DOS,
2006) contains several indicators regarding medicines, particularly on the availability of drugs at U.S.
Government-supported locations and the value of drugs purchased by the U.S. Government.
Data on indicators of access to and use of medicines can be obtained from a range of surveys, including
WHS (health expenditures and insurance, conducted in 2001 and 2007); NHAs (cost and expenditure
data); WHO/Health Action International (HAI) national medicine pricing surveys (prices, availability,
affordability, and components of medicine prices in low- and middle-income countries, conducted in 45
countries in 1999, 2003, and 2007); The World Bank Living Standards Measurement Study (LSMS)

46
surveys (level and distribution of out-of-pocket payments for health care and extent to which such
payments act as barriers to health care access); and USAID-funded SPA facility surveys (availability of
essential medicines/first-line medicines, indicators for monitoring vaccine storage conditions, adherence
to standard treatment guidelines, and quality assurance activities).
Most Recommended Medical Products, Vaccines, and Technologies Indicators
From the above sources, the indicators most frequently cited are components of the MDG indicator that
collectively try to measure the proportions of national populations that have access to affordable essential
drugs on a sustainable basis. Four dimensions are largely captured by the indicators: drug prices, drug
availability, affordability (financial burden of drug costs felt by consumers and governments), and quality.
• Proportion of population with access to affordable essential drugs on a sustainable basis
(MDG indicator on health service delivery). This has not been monitored regularly, and WHO
now recommends nine indicators that are components of access to essential drugs.
• Availability and price of essential medicines. The average availability of 30 selected essential
medicines in public and private health facilities indicates the frequency of stockouts in facilities and
the degree of system function in financing, procurement, logistics, and distribution. This is a
common measure but detailed definitions vary by country as medicines vary by epidemiological
profiles and health goals. Although the WHO global target is 80 percent availability, country-
specific targets are required. Furthermore, data may not include all private sector facilities. The
median consumer price ratio of essential medicines also has a target set by WHO of not more than
four times the world market price. This is an indicator of cost, as well as of procurement efficiency,
governance, and corruption. The core list of medicines tends to vary among countries, and therefore,
results are often not comparable. The number of facilities (including pharmacies) is sometimes
small, leading to large sampling error. Data need to be presented by type of facility and by public
and private sectors. To evaluate affordability, several studies have compared medicine treatment
costs to a day’s wages of the lowest-paid unskilled government worker (WHO, 2008;
WHO/AusAID, 2006). This is a useful complement to the actual cost of the drug compared with
international prices. Drug availability and price make a composite indicator that includes the
following three subindicators:
- Percent of facilities that have all tracer medicines and commodities in stock. The availability
of critical supplies and frequency of stockouts in facilities are common measures of how well a

47
system is functioning in financing, procurement, logistics, and distribution. Alternatively, some
surveys may collect data on the percent of facilities that have all tracer medicines and
commodities in stock on the day of the visit. Data on medicine availability can be used with data
on other components of service capacity to assess the ability of facilities to provide specific
services.
- Median proportion of tracer drugs that are in stock on the day of visit and in the last three
months.
- Ratio of median local medicine price to international reference price (median price ratio)
for core list of drugs.
• The existence and year of last update of a published national medicines policy. The existence of
such a policy indicates a commitment to improving pharmaceutical management in the public and
private sectors. Data can be found at WHO and MOH Web sites, and in country studies. The WHO
target is the existence of a policy updated within the last 10 years.
• The existence and year of last update of a published national list of essential medicines. This
indicator measures a country’s commitment to rational resource allocation and containing
pharmaceutical costs. Information can be found in National Essential Medicines Program and WHO
reports. The WHO target is the existence of a list updated within the last two years.
• Standard treatment guidelines. These indicate the capacity to provide consistent treatment for
common health problems. If guidelines exist, evidence-based best practices for treatments of
common conditions are reviewed and codified. Data comparability is an issue with this indicator, as
guidelines may vary by country. The WHO target is the existence of a policy updated within the last
five years.
• Percent of drugs purchased through competitive bidding of total pharmaceutical expenditures.
A well-organized procurement unit should have this information readily available. This indicator
has a WHO target of 80 percent. An estimate of the value would be acceptable in most cases if the
question is also asked about the percentage of suppliers that are international versus national or local.
However, not all items are best procured through competitive tenders, and country or donor policies
may affect purchasing arrangements. In some cases, such as vaccines, there are few reliable
suppliers, so these products are usually procured through direct purchase.

48
• Appropriate prescription practices and rational drug use. Several publications provide
indicators that assess the quality of drug prescribing and use, which is important for monitoring the
impact of essential drugs programs on hospitals, providers, and consumers. Monitoring
pharmaceutical use is a timely issue in developed countries, but resources for this are lacking in
developing countries. As Laing and his co-authors noted in 2001, policies that affect the health
system structure and financing may have negative impacts on individuals’ use of pharmaceuticals,
with outcomes such as reductions in access or overconsumption. Policymakers should monitor the
effects of implementing heath system reforms through key patient use indicators that have already
been developed and tested (Laing, et al., 2001).
The WHO Level I and II Fact Book includes indicators on policies for rational use, prescribing practices,
public education campaigns, and the provision of independent information for providers and consumers
(WHO, 2006). The WHO Toolkit does not include indicators of use. In general, indicators of use, the
presence of a national strategy to contain antimicrobial resistance, and public education on medicines are
less frequently mentioned by agencies or countries. Other less frequently mentioned indicators are
standards and regulation of traditional medicines, and legal provisions for generic substitutions.
There are several international targets for access to medicines, such as the WHO and Joint United Nations
Program on HIV/AIDS (UNAIDS) ‘3 by 5’ indicators for access to HIV/AIDS treatment, the Amsterdam
targets for TB treatment, and the Abuja targets for malaria. These are targets for access and end goals;
they are less useful for monitoring progress toward improved access and cost.
Another issue is government expenditures on medicines as a percent of the health budget. In 1999, WHO
established a target of 20 percent to be sustained over a three-year period. This indicator is not included in
the WHO draft Toolkit, nor is it widely used by other agencies. The HSAA includes a measurement of
government expenditures on drugs but does not set a target. Also, less frequently mentioned is the
existence of quality assessment systems for products prior to procurement, such as a drug prequalification
process. WHO does not include prequalification as an indicator in the Toolkit, although it encourages the
use of the WHO/United Nations prequalification program. Regarding traditional medicines, while WHO
has put forth guidelines for the registration of herbal medicines and technical guidelines related to safety,
efficacy, and quality, most agencies do not assess progress in the use and regulation of traditional
medicines as part of HSS activities. However, as of 2007, about 50 countries had a national policy on
traditional medicines, and more than 110 had various mechanisms in place to regulate these medicines.

49
Researchers recently were able to use data from the WHO/HAI surveys to determine drug prices,
availability, and affordability in 36 low- and middle-income countries (Cameron, et al., 2008), and other
researchers have used 2002 WHS to examine health expenditures and medicine access among
respondents in eight countries (Wagner, et al., 2008). The WHS indicators were used to assess out-of-
pocket expenditures on medicines and barriers to access to medicines. The second phase of this research
is now under way and will involve analytic work to assess the key determinants of medicine access and
affordability, and the extent to which health insurance protects households from catastrophic health
expenditures, as well as the impact of insurance on cost-related barriers to medicine access (WHO/
PAHO Collaborating Center in Pharmaceutical Policy).
Issues and Concerns
Despite the existence of several global and regional price information services, including the WHO/HAI
database, data availability on access to and use of medicines is a critical issue. SPAs capture many of the
facility-based aspects of medicine availability and prescribing but only have been conducted in a few
countries. WHS capture expenditure and insurance information, but the same questions may not be asked
in all countries. Product quality is also difficult to assess (other than by expiration date), as are
appropriate prescription practices, rational drug use, and user compliance. WHO is working to improve
the accessibility of unique drug information at the country level and create one central Web site with links
to all medicine-related country information relevant for planning and measuring progress (WHO, 2008).

50
Financing
According to WHO, the purpose of health financing is to “make funding available, as well as to set the
right financial incentives to providers, to ensure that all individuals have access to effective public health
and personal health care” (WHO, 2000). This building block is essential for the operation of a national
health system – if financing were to fail, no health promotion or disease prevention would be able to
take place.
Health financing refers not only to funds coming from the government but also to funds spent by
individuals on their health care (out-of-pocket expenditures) and funds coming from and managed by the
private sector, such as employers and insurers. In developing countries, additional funds come from
donors and are either administered by donors themselves or given to the government or a private
institution to be administered as they do their own funds. Donor funding is not a sustainable source of
funding in the long term, so health system plans should include increasing domestic spending on health to
eventually bridge the gap currently covered by donor funds. In the short term, increasing the diversity of
international donors is important to protect against unexpected discontinuation of funds (WHO, 2007).
As mentioned above, the health financing system also will set financial incentives for providers. These
incentives can have positive and negative effects on how health service workers provide care
(Christianson, et al., 2007). Ideally, health systems should reward providers who give better care and
serve more patients with greater payment, but in practice these “pay for performance” schemes can result
in the creation of unintended (“perverse”) incentives. Health system planners need to take into account
what incentives they are creating when they decide on provider payment methods. These payment design
questions have been covered extensively in the health economics literature for more developed countries,
as well as for India, China, Russia, and some Latin American countries, but for less-developed countries
there has been less study of incentives and payment methods because of the general simplicity of their
health finance systems (Schneider & Hanson, 2007).
Ensuring all individuals have access to care requires that financing be available universally, whether
through public or private funding. This funding needs to be available in sufficient amounts to cover basic
care and prevent financial disaster due to a catastrophic illness (WHO, 2007). The definition of
catastrophic health costs is still under debate, but a 2003 WHO-supported study found that three key
preconditions for incurring catastrophic health costs were the availability of health services requiring
payment, low capacity to pay, and the lack of prepayment or health insurance (Xu, et al., 2003). The

51
importance of pooling financial risk through prepayment is reiterated in WHO’s 2007 Framework for
Action report (WHO, 2007) and USAID’s HSAA manual (Islam, 2007). For poor and vulnerable
populations with a low capacity to pay, prepayment may not provide full protection from financial
hardship, and WHO encourages taking additional steps to cover their health care costs.
Finally, the supply and demand of health care is dynamic, and thus health financing needs to be able to
flow efficiently, effectively, and equitably to the facilities and individuals who need them.
In order to measure whether these aspects of health financing are being met, indicators should cover
whether adequate funds are available in ways that ensure access to health care, if the system protects from
financial catastrophe, and if there are incentives for providers and users to be efficient with funds (WHO,
2007). In general, the indicators in the sources below fall neatly into these categories. Financing
indicators are generally quantifiable, making measurement less difficult. The indicators are also largely at
the national level. Other building blocks such as service delivery and information systems will have more
district- and facility-level indicators.
Recommended Indicator Sources
For finance, the WHO indicator Toolkit has just two recommended core indicators and one subindicator,
reflecting the dearth of reliable financial information (WHO, 2008). It does, however, cover both national
expenditures and personal expenditures, and there are also three optional indicators being considered to
gather greater information on resource utilization and household impoverishment due to health
expenditures.
There is significant overlap in indicators between WHO’s Toolkit and USAID’s HSAA, but the latter has
more indicators and information on the rationale behind the choice of each. It includes nearly all of the
indicators given in the Toolkit.
USAID’s graduation report on increased health promotion and access to quality health care
(USAID Program Objective Team 3.2, 2001) includes many of the recommended indicators described
below and an indicator for effective payment systems for primary and secondary health care providers.
This type of indicator relates to provider incentives and is one of the few sources that touch on this topic.
In order to operationalize such a measure, greater detail needs to be given on what counts as an effective
payment system.

52
The SWEF Research Network takes a slightly different approach to categorizing indicators but still uses
several of the same measures as the WHO Toolkit. It also includes measures relating to financial system
sustainability, the pace of disbursement of donor and MOH funds, and the verticality of the system. The
majority of these additional indicators seem to be of more relevance to funders rather than to the broader
health systems community. This is in part because The Global Fund is a disease-specific funding
organization and also because the stated purpose of the indicator list is to monitor how The Global Fund’s
financial support is being used (SWEF Research Network, 2003).
A CGD review of indicators for MCC resulted in only one finance-related input indicator on the final
MCC list – the portion of total general government expenditure (GGE) that is used for general
government health expenditure (GGHE) – but the earlier list included many other financial indicators that
were disqualified due to data availability concerns (Becker, Pickett & Levine, 2006). These may become
usable later, if data become available. It is notable that The World Bank Country Policy and
Institutional Assessment (CPIA) Index also relies solely on the GGHE/GGE ration for measuring
health financial systems (World Bank, 2007 and 2008).
MDG indicators are largely involved in measuring health outputs and outcomes, but two would collect
information on how much donor funding is going to health. The ECHIM Project includes one indicator
on health financing (ECHIM, 2008).
Data availability is possibly the most important criterion for judging health systems indicators. The main
data sources for the indicators mentioned in this section are NHAs, household surveys such as DHS and
LSMS, and SPA exit interviews. As pointed out earlier by the 2000 World Health Report, NHAs are
subject to the same quality, validity, and reliability issues as other data sources from less-developed
countries but are nonetheless an important part of estimating the efficiency of national health systems
(WHO, 2002). WHO worked from the data available in full NHAs from 67 countries to choose its
financial indicators. In some cases, additional disaggregated data can be taken from The World Bank’s
public expenditure reviews.
Most Recommended Financial Indicators
From these sources, we found a collection of indicators recommended by multiple organizations.
• THE per capita is one of the financial indictors most often cited, as it is a commonly collected
indicator that measures the availability of resources for health. While there is not a direct linear

53
relationship between expenditures and health at all expenditure levels, a 2001 study found that at
low expenditure levels, health system efficiency is positively related to health expenditure per
capita (Evans, et al., 2001). Performance sharply increases with expenditure up to about $80 per
capita per year. According to the WHO’s Commission on Macroeconomics and Health, basic
essential health care services would have required expenditures in 2007 of at least US$34
[US$30–40] per capita per year in low-income countries (WHO, 2001). Countries spending
below this threshold are likely to have poor access, a low quality of health care, or both (Kruk &
Freedman, 2008).
• THE as a percent of gross domestic product (GDP) is cited as often as THE per capita. Some
would argue that this measure is not as easily interpretable, and HMN has noted that a target
specifying a minimum threshold for this indicator is not very useful on scientific grounds (HMN,
2006). For the purposes of determining graduation countries in its E&E region, USAID also has
set a minimum threshold for this indicator (USAID, 2001).
• As mentioned earlier, another often-cited indicator is GGHE/GGE. This indicator illustrates the
level of government financial support to the health sector relative to other financial commitments.
This is one way to quantify the political support for health within a country. According to the
Abuja Declaration, 15 percent or greater GGHE/GGE would reflect that health is a high priority
in that country. However, as with THE per capita, when GGHE/GGE is already high, increasing
it will not necessarily be a positive movement. This indicator will have to be gauged using
country context.
Two other notable indicators are the amount of total donor spending on health as a percent of THE
and the percent of government health spending that reaches the poorest income quintile:
• The first is an indicator that measures how sustainable health funding is in a country. If donor
contributions to a country’s total health spending are above 10 percent, a significant burden
would be placed on private spending if donor contributions are withdrawn (Islam, 2007).
• The second of these indicators deals with equity in resource allocation and risk protection, which
is very important, but there is no easily applicable threshold that would be appropriate in every
situation, and data collection for this would be difficult (Walford, 2007). Whether or not this
particular indicator is used, there is a need for some type of measure dealing with the equity of
health care finance in order to describe a population’s exposure to catastrophic health

54
expenditures. Several similar indicators deal with the financial risk households are exposed to by
health expenditure, but there is a lack of consensus on how best to measure this. The WHO
Toolkit suggests using a ratio of out-of-pocket expenditures to total health expenditures, but it
also considers the percent of the population incurring catastrophic health expenditures, which is a
measure also suggested by DfID. Other sources approach this question by measuring not the
exposure to risk but the extent of protection against such risk. ECHIM includes a measure related
to the percent of the population covered by prepaid care/health insurance, as does an academic
literature review (Kruk & Freedman, 2008). Risk pooling through insurance and prepayment are
acknowledged to be the best ways to increase financial protection against health expenses and
equity of health financing (Gottret & Schieber, 2006). While insurance and prepayment schemes
may not be widely used currently, including an indicator to measure insurance/prepayment could
encourage the creation of such programs. Clearly, there is still considerable discussion on how to
capture these data and what types of thresholds would be appropriate. The World Bank’s work on
the idea of affordability, notably by Wagstaff and van Doorslaer, has contributed to a better
description of the difficulties of measuring this indicator (Wagstaff, 2002; Wagstaff & van
Doorslaer, 2004).
• Indicators to monitor country actions for strengthening health financing. The WHO Toolkit
gives additional “output” indicators related to the above input indicators. These measure the
government outputs that would produce measures of resources. Some of these indicators are
collected under information systems in other sources, while others, such as government
expenditures on salaries and number of people covered by health insurance, are mentioned as
stand-alone indicators in other sources under finance.
In an extensive literature review, THE per capita and donor spending on health as a percent of THE were
found to be among the most often used in evaluations of health systems. The authors discuss the data
availability for these indicators, as well as perhaps a stronger link to health outputs and outcomes (Kruk &
Freedman, 2008).
Issues and Concerns
To have cross-country comparability, it is important that the classification of health expenditures be
standardized in NHAs. Particularly in the first round of NHAs, it was not possible to disaggregate total
expenditures by function due to country-by-country variations in classifying funds (Nandakumar, et al.,
2004). Until this standardization occurs, any indicator based on this data will have to be used with caution

55
across countries. The OECD has published a very complete guide to data collection for its own work on
its System of Health Accounts, which may be a good basis for standardization (OECD, 2000).
Expenditure data in household and facility surveys also will be of concern due to the lack of monetization
of markets in many countries, so dollar amounts may not always capture the total cost of a transaction
(Kahn & Hotchkiss, 2006). It is important, however, to collect whatever information is available on the
personal cost of health care to measure household hardship due to health care.

56
Leadership and Governance
Governance in health systems is about establishing effective rules in the institutional arenas for policies,
programs, and activities related to public health functions in order to achieve health sector objectives
(USAID, 2008). These rules determine the roles and responsibilities for three sets of actors: state actors
(who include policymakers and other government officials), health service providers, and service users.
Determining the roles and responsibilities among these three sets of actors can reduce the hierarchical or
authoritarian nature of public sector bureaucracy, often a core obstacle to the practice of good governance
within health systems (Doherty & Gilson, 2006). According to WHO, this building block (also referred to
as stewardship) is the most complex but critical building block of any health system (WHO, 2007). There
is no clear blueprint for effective health leadership and governance. Although governance is ultimately
the responsibility of the government, leadership and governance functions are carried out at lower levels,
as well. Furthermore, as the concept of governance/stewardship in the health sector is relatively new,
there is little guidance for the collection or standardization of information on this aspect of the health
system (Islam, 2007).
The rationale for governance and accountability is that health systems are responsible for the
improvement of population health in an equitable manner, and those affected by decisions and policies
that affect well-being must have an understanding of and ultimate control over that system. Such control
requires reasonable accountability from the government (Daniels, et al., 2000). Good governance has
been shown to correlate with property rights, civil liberties, greater foreign investment, and increased aid
effectiveness. These clearly have relevance to health care (Lewis, 2006). Good governance also includes
evidence-informed policymaking and knowledge generation and dissemination.
Governance has been the subject of multiple definitions and interpretations that derive from different
purposes (USAID, 2008). Some definitions concentrate on technical government functions and how they
are administered. The World Bank views governance as economic policymaking and implementation,
service delivery, and accountable use of public resources and regulatory power. It divides governance into
six dimensions: voice and accountability, political stability, government effectiveness, regulatory quality,
rule of law, and control of corruption. Other definitions address how government connects with other
sectors and with citizens. For example, USAID considers health governance as the process of
“competently directing health system resources, performance, and stakeholder participation toward the
goal of saving lives and doing so in ways that are open, transparent, accountable, equitable, and
responsive to the needs of the people” (Brinkerhoff, 2007). Different United Nations agencies have

57
various descriptions of governance, while WHO prefers the concept of governance as stewardship, “the
careful and responsible management of the well-being of the population” (WHO, 2000).
Better governance influences health in various ways. For example, it can:
• Improve the policy process in the health sector by greater use of data, evidence, and policy
research, and increase the extent to which policy processes are informed by independent, valid
research and analysis, and are open and accessible to citizens
• Improve accountability and transparency, and reduce corruption, informal payments, fraudulent
billing, and employee theft by installing fee collection systems or facility-level accounting and
reporting systems and by monitoring the sale of publicly funded drugs in the public and private
sectors
• Enhance participation at local levels by providing information to the public on citizens’ rights and
duties in health sector activities and increase participation by civil society and community groups
To achieve better governance, WHO helps governments develop health sector policies and frameworks,
design regulatory frameworks, support greater accountability, generate and interpret intelligence and
research on policy options, build coalitions, and work with other partners (WHO, 2007).
According to USAID’s Health Systems 20/20 project, governance indicators are of two types:
1. Rules-based indicators, which measure whether the government has established key rules or
policies in the health sector, such as a national essential medicines list or a national policy on
malaria control. These indicators cannot be measured along a continuum and are discrete
achievements.
2. Outcome-based indicators, which measure whether rules and procedures are being effectively
implemented or enforced based on the experience of relevant stakeholders. These include
integration and responsiveness indicators that measure aspects of hospital or facility
organizational structure and functioning, such as information use and technology, coordination
among staff, and use of standardized protocols.
Recommended Indicator Sources
The WHO draft Toolkit on Monitoring Health Systems Strengthening classifies indicators into the
rules-based and outcome-based categories. There are 10 rules-based indicators that result in a Policy

58
Index for assessing overall policies, regulations, and strategies in relation to the health sector. The Toolkit
also has six select markers of governance: health worker absenteeism, government funding reaching the
district level, stockouts, informal payments, pharmaceutical regulation, and the existence of civil society
organizations. The Toolkit also uses The World Bank’s CPIA Index to provide a composite measure of
governance. Again, the indicators overlap with the medicines and HIS sectors.
The WHO Regional Office for the Eastern Mediterranean (EMRO) has developed an analytical
framework that intends to measure the governance function of the health system at the national and
subnational levels (WHO/EMRO, 2007; Siddiqi, 2008). It assesses nine principles of governance:
strategic vision; participation and consensus orientation; rule of law; transparency; responsiveness; equity
and inclusiveness; effectiveness and efficiency; accountability; and intelligence, information, and ethics.
These nine principles are divided into 100 broad and specific questions that use data from both publicly
available information and key informant interviews. The framework includes interviews with private
sector NGOs but does not include interviews with civil society organizations (CSOs) or measures of
clinical governance from interviews with health system users. While the results help identify strengths
and weaknesses, they are qualitative and not comparable between countries. The framework has been
applied in nine countries, including Afghanistan, Egypt, Pakistan, and Sudan. The questionnaire is
available, but we have not been able to find results of the country assessments.
PAHO also has developed a tool to assess government stewardship in providing “essential public health
functions” (EPHF) (PAHO, 2008). The EPHF are the fundamental set of actions that governments should
perform in order to improve the health of populations. PAHO has developed a methodology that allows
health ministries to evaluate in a comprehensive manner their public health systems, including health
infrastructure, management, and financing, and evaluate their performance in the 11 functions. It has
practical use as a framework for understanding the MOH service delivery capacity and role, and can be
used as an M&E tool by governments. It was initially conducted in 41 Latin American and Caribbean
countries in 2001 and 2002, and since then many countries have continued monitoring the status of EPHF
not only at the national level but at the subnational level, as well.
USAID’s HSAA instructs on how to use indicators to gauge the overall government context and the
capacity of the government in five dimensions (Islam, 2007). For the overall governance environment, the
HSAA uses the six World Bank indicators of government effectiveness, as well as indicators from five
dimensions of governance: information/assessment capacity, policy formulation and planning, social
participation and system responsiveness, accountability, and regulation. Several indicators overlap with
indicators for information systems, medicines, and financing. There are criteria and guidance given on

59
how to weigh the information and produce a balanced assessment, although there are few questions that
can be answered with quantitative data. The HSAA has far more indicators on regulation and social
responsiveness than WHO and other agencies; however, unlike other groups, it does not include
indicators of disparity between quintiles or regions.
USAID Missions have included health governance indicators as part of results frameworks under the
Democracy and Governance Strategic Objective. USAID/Guinea, for example, in its results framework
measured progress toward “improved effectiveness, accountability, and transparency of government
institutions in the productive and social sectors,” using the indicator “percentage of target CSOs which
have formed effective partnerships with local government delivering services.” For the objective
“Strengthened Civil Society and Advocacy,” one indicator used by the Mission was “Percent of CSOs
that effectively monitor government public services.” These are examples of indicators that could be
modified to be potentially collected across countries. The USAID Democracy and Governance Office also
developed the Handbook of Democracy and Governance Program Indicators (Hyman & Silver, 1998).
While there are no health sector indicators, it offers insight into the challenges of applying performance
measurement indicators for governance.
DfID has adopted the use of “country governance analysis” to help guide aid allocations to countries
(DfID, 2008). These analyses examine the capability (stability, regulation, trade/growth, effectiveness,
security); accountability (transparency, free media, rule of law, elections); and responsiveness
(rights/liberties, pro-poor, equality, regulation, corruption) of the government. Assessments are based on
existing national and international data. They are relatively new and have not been completed for many
countries.
The Working Group on Health System Metrics comprises HMN, WHO, The World Bank, The Global
Fund, and the GAVI Alliance uses The World Bank’s CPIA Index, which rates countries against a set of
16 criteria grouped in four clusters: (a) economic management; (b) structural policies; (c) policies for
social inclusion and equity; and (d) public sector management and institutions. In addition to
macroeconomic management, the assessment criteria include trade and financial policies; business
regulation; social sector policies; the effectiveness of the public sector; and transparency, accountability,
and corruption.
As a response to improve the health indicators of the Millennium Challenge Account (MCA), the CGD
Global Health Indicators Working Group reviewed indicators for their overall utility and their relationship
to poverty and good governance. Good governance and development policies are key considerations of

60
aid allocations from the MCA. The Group’s report, Measuring Commitment to Health, specifies
indicators that reflect a government’s commitment and allocation of resources to reach the underserved
and its selection of interventions (Becker, Pickett & Levine 2006). It chose eight proxy indicators
(complete diphtheria-pertussis-tetanus immunizations [DPT3], government public health expenditure,
under-5 mortality rate, contraceptive prevalence rate, unmet need for family planning, stunting, skilled
birth assistance, and access to water) to reflect a government’s health policies and commitment to health.
These are actually output indicators but are included here to demonstrate the different concepts of
measuring governance.
The Global Fund’s March 2008 addendum to its 2006 Monitoring and Evaluation Toolkit provides
guidance on indicators of governance, including six examples (such as number of CSOs partnering with
public/private providers to deliver services and frequency of audits or reviews of performance). Rather
than specify indicators, it strongly encourages countries to use existing in-country indicators such as those
that are part of a program-based approach (including the Sector-Wide Approach) or other national
strategic frameworks that meet Global Fund M&E requirements. Governance indicators may be indicators
of equity of access and coverage of essential services by regions quintile or vulnerable population groups.
Appropriate targets are set using country-specific baselines and resources (The Global Fund, 2008).
The SWEF Research Network working group primarily focuses on governance of the policy process,
fairness in resource allocation, and donor harmonization. It is one of the only sources to include indicators
on the number of public/private partnerships for services and the number of new private providers, which
measure private sector involvement in health care, the government’s understanding of the role of the
private sector, and ease of entry into the market. These are goals of The Global Fund, and private sector
involvement may not be a suitable indicator for all countries.
Other sources of governance indicators include “corruption evaluations” to help policymakers measure
corruption in the health sector and identify possible ways to intervene to increase accountability,
transparency, citizen voice, detection, and enforcement; control discretion; and reduce monopoly power
(Vian, 2008). Sources of data to identify corruption include corruption perception surveys, expenditure
surveys, qualitative data collection, and reviews of control systems. Other indicators that have been
identified for capturing poor governance in health service delivery mismanagement include leakage of
drugs and supplies, patients providing in-kind supplies and drugs, staff absenteeism, and informal
payments (Lewis, 2006). Benchmarks of Fairness, a generic matrix for assessing health systems reform
in developing countries (Daniels, et al., 2000), includes benchmarks for democracy, accountability, and
empowerment. The democracy benchmark has eight field-tested and measureable criteria (indicators),

61
such as the existence of procedures for evaluating services, resource allocation with transparency, and fair
grievance. Because of the adaptive framework, it is a promising tool for cross-country comparability
(Alliance for Health Policy and Systems Research, 2008). The USAID Strategic Framework for
Foreign Assistance Governance Indicators quantify how USAID Missions are improving governance
in USAID-assisted areas and at the national level (DOS, 2006). A corruption perceptions index (CPI),
public sector opinion surveys, and corruption barometers are available from Transparency International
(TI); however, they do not cover any health-specific indicators. Among the 11 institutions that provide
data for the CPI, none are health institutions or organizations. TI and a partner organization, CIET
International, have implemented public opinion surveys, institutional surveys, case studies, and social
audits that measure governance of local health institutions or health boards. However, these case studies
are typically qualitative assessments for specific purposes and are not comparable across countries.
Another indicator used by Political Risk Services in its International Country Risk Guide is a measure of
institutional/bureaucratic quality (PRS Group, 2009). This indicator captures the result of good
governance, although not necessarily in the health sector. It has been used in research to analyze the
relationship between governance and health and education. Its role, however, has not been clearly
captured in such analyses.
Most Recommended Leadership and Governance Indicators
The indicators most frequently cited are measures of accountability and social participation. Government
expenditure on health is also frequently used, although this overlaps with financing measures. Sources
varied widely in the use of governance indicators, although several include the CPIA in their overall
assessment. Often governance indicators must be established within local contexts and aligned with local
goals. Some general descriptions include:
• WHO Policy Index. WHO proposes universal use of 10 yes/no indicators that assess if a country
has adopted WHO-recommended policies, regulations, and strategies in relation to 10 areas of the
health sector (WHO, 2008). The 10 policy items are rated as zero (adequate policy does not exist
or cannot be assessed) or one (adequate policy is available). The proposed index does not aim to
assess enforcement, as this may be captured by other indicators within the health system
components. The indicators include the existence of a national health strategy; an essential
medicines list and policy on drug procurement; a national policy on child, maternal, and
reproductive health; national policies on malaria and TB; and participation in the United Nations
General Assembly Special Session (UNGASS) composite policy index questionnaire.

62
• Existence of an up-to-date national health strategy linked to national needs and priorities.
The strategy should follow internationally accepted policy standards and guidelines (e.g., the
international code of marketing of breast milk substitutes, international standards of care). These
measures indicate government stewardship and capacity to develop, implement, and monitor
legislation and guidance on public health and health system issues. Comprehensive health policy
and planning processes integrate health system information, public input, and evidence-based
recommendations for action. The target is for each health goal to conform to standard
international policies. Comparability is an issue, as the regulation and enforcement of policies,
and the processes by which they are set, vary by country. Data are found within MOH documents
or reports. This indicator is also part of the WHO Policy Index.
• Health worker absenteeism in public health facilities. This indicator is also an indicator of
regulation (see below).
• Proportion of government funds that reaches district-level facilities. The rationale for this is
that governance in health financing can be assessed by monitoring not only overall levels of
health spending but also equity in allocating budgets and efficiency in ensuring that spending
reaches health facilities and the poor. Data can be gathered from Public Expenditure Tracking
Surveys, NHAs, and/or MOH records.
• Health service delivery: Stockout rates of essential drugs in health facilities. This indicator is
also part of the medicines component of health systems (see Medical Products, Vaccines, and
Technologies section).
• Pharmaceutical regulation: Proportion of pharmaceutical sales that consist of counterfeit
drugs. This indicator is also part of the medicines component of health systems (see Medical
Products, Vaccines, and Technologies section) and one of several often-used indicators of
regulation (see below).
• Accountability indicators. These measure the government’s ability to answer questions, meet
reasonable expectations of the system, and address negligent or corrupt actions. There are many
measures of this, and researchers approach accountability using different paradigms and measure
different outcomes (Alliance for Health Policy and Systems Research, 2008). They include

63
indicators of civil society participation in decisions, although the range and impact of civil
involvement are difficult to compare. Some agencies (USAID among them) include the existence
of a free and scientific press, watchdog organizations, and an independent judiciary as a
requirement for accountability. Other measures of accountability include the existence of fair
grievance procedures, such as dispute resolution, that are available to the public. Data likely come
from interviews of stakeholders.
• Disparity in coverage between lowest- and highest-income groups/regions/rural/urban areas.
Equitable coverage demonstrates rational and transparent resource allocation. However,
comparability is an issue, as the size of the different groups or regions affects result and may limit
the usefulness of country comparisons of data. Data can be found in household surveys.
• CPIA Index. The World Bank’s CPIA Index is based on a set of criteria captured in 16
subcomponents grouped in four clusters: (a) economic management; (b) structural policies; (c)
policies for social inclusion and equity; and (d) public sector management and institutions (World
Bank, 2007 and 2008).
• Proportion of informal payments within the public health care system. Informal payments
can reduce the utilization of services by patients who cannot pay and reduce the quality of care
from loss of revenue at facilities. The frequency of informal payments is often used in studies
and interventions. Data can be gathered from household surveys, corruption perception surveys,
and key informant interviews, although the latter are subjective, subject to measurement error,
and can only provide a rough indication of trends.
• Regulation. The most frequently cited indicator is pharmaceutical regulation, which itself has
many measures, including “the existence of adequate regulation to ensure the safety, efficacy, and
quality of medicines” and “the proportion of pharmaceutical sales that consist of counterfeit.”
Regulation of medicines and procurement overlap with the medicines sector indicators. Indicators
also can include absenteeism, accreditation, and licensure of health professionals, although these
may overlap with the workforce sector. Data on regulatory and enforcement policies may be
available at the MOH; other sources include key informant interviews and facility surveys.
• Social responsiveness. One goal of many donor-funded health reforms is to increase the extent to
which citizens of a country are able to participate in health policy decisions. These indicators
measure the government’s ability to facilitate collaboration among government, civil society, and

64
other stakeholders to participate in the planning, budgeting, and monitoring of activities in the
health sector. They include voice and accountability and whether CSOs empower individuals to
express their views to government bodies. Indicators frequently relate to the number of civil
society members or civil groups trained. Indicators for this objective vary depending on the
country context and may not be relevant in all situations. Data likely come from interviews of
stakeholders.
One measure that overlaps with health service delivery is evaluation of staff performance. Evaluations
can improve the accountability of the government in providing appropriate services and monitor the
responsiveness of the public health sector. They also can measure staff absenteeism or adherence to
regulations and protocols. Indicators on patient expenditure are not included here and would overlap with
financing. Both the DHS and WHS can provide data on inequity, and WHS can provide data on
expenditure. Neither the WHS or DHS questions distinguish between formal and informal payments.
Issues and Concerns
Data availability is a critical concern when selecting indicators. Several indicators have data available for
most countries in the public domain, using sources such as household surveys, World Bank indexes, and
Transparency International’s CPI and Global Corruption Barometer survey. Currently, there is no health-
specific policy and institutional assessment tool, but WHO is proposing one based on measures in The
World Bank’s CPIA (WHO, 2008). The CPIA itself is not yet suitable to be deconstructed into a health-
specific index (Bos, 2006). There are five health- and education-related indicators in the CPIA under the
“building human resources” dimension. These five indicators are similar to the WHO Policy Index and
account for 1.67 percent of the total score, with an overall correlation with the CPIA of .76. Recent
discussions and recommendations for revising the CPIA included recommendations for evaluating
national poverty reduction policies and social policies but not for increasing the significance of health
sector in the score. There are other issues in creating a health-specific CPIA, including the inclusion of
the overall country policy framework in any assessment of health governance.
Another general concern is the simplistic nature of some indicators. For instance, the WHO indicator on
the existence of a country health strategy is a yes/no indicator that does not capture the breadth of how
governance may affect health outcomes. Governance indicators also are not always sector specific. As a
result, it is not always possible to disaggregate them to be health sector specific. They often capture a
composite picture of a country’s governance but not down to the sector level. A good example is the set
of governance-related indicators that are part of the MCC category of Ruling Justly.

65
The comparability of measures of governance is also a challenge, as the definitions and contexts of
governance vary across countries and depend on the degree of development. Moreover, in any external
assessment, data for many indicators must still be gathered in collaboration with the MOH. Reliable
reporting of data is also a concern. There have been no systematic reviews on governance or
accountability in health systems, and research analytic frameworks and measurements vary widely
(Alliance for Health Policy and Systems Research, 2008). Research responses are often qualitative, which
affects their quality. For example, the assessment of data reliability and the quality, timeliness, and extent
of data use are graded according to qualitative answers on seven questions. The methodology and issues
concerning corruption perception surveys also are currently being debated. The perceptions of the public
regarding services will vary according to local circumstances, and perceptions can be different from
actual behaviors. The level of decentralization and the development of information systems also affect
many aspects of financing and policy measures.

66
References
Alliance for Health Policy and Systems Research. (2008). Neglected Health Systems Research:
Governance and Accountability. Geneva: World Health Organization.
Alliance for Health Policy and Systems Research. WHO Web site. http://www.who.int/alliance-hpsr/en/
Becker, L., Pickett, J., & Levine, R. (2006). Measuring Commitment to Health: The Global Health
Indicators Working Group Report. Washington, DC: Center for Global Development.
Bennett, S. & Fairbank, A. (2003). The System-Wide Effects of The Global Fund to Fight AIDS,
Tuberculosis and Malaria: A Conceptual Framework. Technical Report No. 031. Bethesda, MD: The
Partners for Health Reform Plus Project, Abt Associates Inc.
Bertrand, J. & Escudero, G. (2002). Compendium of Indicators for Evaluating Reproductive Health
Programs. Vol. I. USAID/MEASURE Evaluation Manual Series No. 6. Chapel Hill, NC: Carolina
Population Center.
Bos, E. (2006). Health Systems Metrics: The Country Policy and Institutional Assessment Indicator.
Slide presentation at Health Systems Metrics Technical Meeting, Glion, Switzerland, 28–29 September
2006: The World Bank. http://www.who.int/healthmetrics/documents/boscpia.ppt#257
Brinkerhoff, D. (2007). Health Governance: Why, What, and How. Presentation at Global Health Mini-
University, Washington, DC, October 2007. Bethesda, MD: Abt Associates, Health Systems 20/20.
Cameron, A., Ewen, M., Ross-Degnan, D., Ball, D., & Laing, R. (2008). Medicine Prices, Availability,
and Affordability in 36 Developing and Middle-income Countries: A Secondary Analysis. The Lancet,
373(9659):240–9.
Capacity Project Web site. http://www.capacityproject.org/
Carandang, E.& Pierre-Jacques, M. (2005). Using Indicators to Measure Country Pharmaceutical
Situations. Drug Action Program (DAP) and WHO Action Program on Essential Drugs and WHO
Boston Collaborating Center on Pharmaceutical Policy. Slide presentation.
http://archives.who.int/tbs/tbs2005/ indicators.ppt
Chen, L., Evans, T., Anand, S., Boufford, J., Brown, H., & Chowdhury, M. (2004). Human Resources for
Health: Overcoming the Crisis. The Lancet, 364:1984–1990.
Child Survival and Technical Support Plus Web site. http://www.childsurvival.com/rhfa_1.cfm
Christianson, J., Leatherman, S., & Sutherland, K. (2007). Financial Incentives, Healthcare Providers
and Quality Improvements. London: The Health Foundation.
Daniels, N., Bryant, J., Castano, R., Dantes, O., Khan, K., & Pannarunothai, S. (2000). Benchmarks of
Fairness for Health Care Reform: A Policy Tool for Developing Countries. Bulletin of the World Health
Organization, 78 (6).

67
DELIVER. (2006). Contraceptive Security Index 2006: A Tool for Priority Setting and Planning. Task
Order 1 of the USAID | Health Policy Initiative. Arlington, VA: DELIVER, for the United States Agency
for International Development.
DELIVER. (2009). Logistics System Assessment Tool. (LSAT). Task Order. Arlington, VA: DELIVER,
for the United States Agency for International Development.
Department of State. (2006). U.S. Foreign Assistance Performance Indicators for Use in Developing FY
2007 Operational Plans. Washington, DC: Department of State. http://www.state.gov/documents/
organization/78557.pdf
de Vries, W. (1998). How Are We Doing? Performance Indicators for National Statistical Systems.
Netherlands Official Statistics,13: 5–13.
Department for International Development. (2008). Country Governance Analysis: A How-to Note. A
DfID Practice Paper. Politics and the State Team. London: DfID.
DfID Health Resource Center Web site. http://www.dfidhealthrc.org/
Doherty, J. & Gilson, L. (2006). Proposed Areas of Investigation for the KN: An Initial Scoping of the
Literature. Geneva: WHO Commission on Social Determinants of Health.
ECHIM. (2008). ECHI shortlist indicators.
http://www.healthindicators.org/healthindicators/root/o13.html
ECHIM. (2008a). ECHIM Comprehensive Lists. http://www.healthindicators.org/healthindicators/
object_document/o5415n28314.html
ECHIM. (2008b). Links to Indicator Information. http://www.healthindicators.org/
healthindicators/object_document/o4963n28314.html
Evans, D., Tandon, A., Murray, C., & Lauer, J. (2001). Comparative Efficiency of National Health
Systems: Cross National Econometric Analysis. British Medical Journal, 323:307–310.
Fresle, D. & Wolfheim, C. (1997). Public Education in Rational Drug Use: A Global Survey. Geneva:
World Health Organization.
Future Health Systems Consortium Web site. http://www.futurehealthsystems.org/
GAVI Alliance. (2007). GAVI Alliance Health System Strengthening (HSS) Applications, Geneva:
GAVI Alliance.
Global HIV/AIDS Initiatives Network Web site http://www.ghinet.org/index.htm
Gottret, P. & Schieber, G. (2006). Health Financing Revisited: A Practitioner’s Guide. Washington, DC:
The World Bank.
Hall, T.