ArticlePDF Available

Data quality of reported child immunization coverage in 194 countries between 2000 and 2019

Authors:

Abstract and Figures

Analyzing immunization coverage data is crucial to guide decision-making in national immunization programs and monitor global initiatives such as the Immunization Agenda 2030. We aimed to assess the quality of reported child immunization coverage data for 194 countries over 20 years. We analyzed child immunization coverage as reported to the World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) between 2000-2019 by all WHO Member States for Bacillus Calmette-Gué rin (BCG) vaccine birth dose, first and third doses of diphtheria-tetanus-pertussis-containing vaccine (DTP1, DTP3), and first dose of measles-containing vaccine (MCV1). We assessed completeness, consistency, integrity, and congruence and assigned data quality flags in case anomalies were detected. Generalized linear mixed-effects models were used to estimate the probability of flags worldwide and for different country groups over time. The probability of data quality flags was 18.2% globally (95% confidence interval [CI] 14.8-22.3). The lowest probability was seen in SouthEast Asia (6.3%, 3.3-11.8, p = 0.002), the highest in the Americas (29.7%, 22.7-37.9, p < 0.001). The probability of data quality flags declined by 5.1% per year globally (3.2-7.0, p < 0.001). The steepest decline was seen in Africa (-9.6%,-13.0 to-5.8, p < 0.001), followed by Europe (-5.4%,-9.2 to-1.6, p = 0.0055), and the Americas (-4.9%,-9.2 to-0.6, p = 0.026). Most country groups showed a statistically significant decline, and none had a statistically significant increase. Over the past two decades, the quality of global immunization coverage data appears to have improved. However, progress has not been universal. The results highlight the need for joint efforts so that all countries collect, report, and use high-quality data for action in immunization.
Content may be subject to copyright.
RESEARCH ARTICLE
Data quality of reported child immunization
coverage in 194 countries between 2000 and
2019
Cornelius RauID
1,2
*, Daniel Lu¨deckeID
3
, Laure B. Dumolard
1
, Jan Grevendonk
1
, Brenton
M. WiernikID
4
, Robin KobbeID
5
, Marta Gacic-Dobo
1
, M. Carolina Danovaro-Holliday
1
1Immunization Analysis & Insights (IAI), Department of Immunization, Vaccines and Biologicals (IVB), World
Health Organization, Geneva, Switzerland, 2Division of Neonatology and Pediatric Intensive Care,
Department of Pediatrics, University Medical Center Hamburg-Eppendorf, Hamburg, Germany, 3Institute of
Medical Sociology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany, 4Department of
Psychology, University of South Florida, Tampa, FL, United States of America, 5Division of Infectious
Diseases, First Department of Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
*co.rau@uke.de
Abstract
Analyzing immunization coverage data is crucial to guide decision-making in national immu-
nization programs and monitor global initiatives such as the Immunization Agenda 2030.
We aimed to assess the quality of reported child immunization coverage data for 194 coun-
tries over 20 years. We analyzed child immunization coverage as reported to the World
Health Organization (WHO) and the United Nations Children’s Fund (UNICEF) between
2000–2019 by all WHO Member States for Bacillus Calmette-Gue
´rin (BCG) vaccine birth
dose, first and third doses of diphtheria-tetanus-pertussis-containing vaccine (DTP1,
DTP3), and first dose of measles-containing vaccine (MCV1). We assessed completeness,
consistency, integrity, and congruence and assigned data quality flags in case anomalies
were detected. Generalized linear mixed-effects models were used to estimate the probabil-
ity of flags worldwide and for different country groups over time. The probability of data qual-
ity flags was 18.2% globally (95% confidence interval [CI] 14.8–22.3). The lowest probability
was seen in South-East Asia (6.3%, 3.3–11.8, p = 0.002), the highest in the Americas
(29.7%, 22.7–37.9, p <0.001). The probability of data quality flags declined by 5.1% per
year globally (3.2–7.0, p <0.001). The steepest decline was seen in Africa (-9.6%, -13.0 to
-5.8, p <0.001), followed by Europe (-5.4%, -9.2 to -1.6, p = 0.0055), and the Americas
(-4.9%, -9.2 to -0.6, p = 0.026). Most country groups showed a statistically significant
decline, and none had a statistically significant increase. Over the past two decades, the
quality of global immunization coverage data appears to have improved. However, progress
has not been universal. The results highlight the need for joint efforts so that all countries
collect, report, and use high-quality data for action in immunization.
PLOS GLOBAL PUBLIC HEALTH
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 1 / 18
a1111111111
a1111111111
a1111111111
a1111111111
a1111111111
OPEN ACCESS
Citation: Rau C, Lu¨decke D, Dumolard LB,
Grevendonk J, Wiernik BM, Kobbe R, et al. (2022)
Data quality of reported child immunization
coverage in 194 countries between 2000 and 2019.
PLOS Glob Public Health 2(2): e0000140. https://
doi.org/10.1371/journal.pgph.0000140
Editor: Tharani Loganathan, University of Malaya
Faculty of Medicine, MALAYSIA
Received: August 22, 2021
Accepted: December 3, 2021
Published: February 3, 2022
Copyright: ©2022 Rau et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: All data are available
from public sources without limitations.
Administrative and official immunization coverage
levels, immunization schedules, and WHO/UNICEF
Estimates on National Immunization Coverage
(WUENIC) can be retrieved from the World Health
Organization, Department of Immunization,
Vaccines, and Biologicals (https://
immunizationdata.who.int/). Population data used
in this study can be obtained from the United
Nations, Department of Economic and Social
Affairs (https://population.un.org/wpp/Download/
1. Introduction
High-quality data are key in public health and development programs [13]. The global com-
munity has repeatedly highlighted the need for better data to track progress and ensure
accountability of global initiatives such as the Global Strategy for Women’s, Children’s and
Adolescents’ Health, Universal Health Coverage, and the Sustainable Development Goals [4
9]. The COVID-19 pandemic has illustrated that well-functioning health information systems
and high-quality data are crucial to prevention, provision of care, and the successful rollout of
new vaccines [10].
In immunization programs, data are needed to guide decision-making and monitor perfor-
mance at the local, regional, and global level. Performance is often measured in terms of
immunization coverage, that is the proportion of vaccinated individuals in the target popula-
tion for a specific vaccine dose. Since the inception of the World Health Organization’s
(WHO) Expanded Programme on Immunization (EPI) in 1974 and the adoption of the Global
Vaccine Action Plan (GVAP) in 2012, estimated coverage rates have increased worldwide but
plateaued around 85% between 2010 and 2019 [1114]. Early estimates suggest major back-
slides during the COVID-19 pandemic [15,16]. Technical immunization advisory groups
have repeatedly expressed concern about immunization data quality since the turn of the mil-
lennium [11,1720]. Gavi, the Vaccine Alliance, cited data quality problems in 2020 as a “very
high risk” to their investments [21].
In 2020, WHO Member States endorsed the Immunization Agenda 2030 (IA2030):a Global
Strategy to Leave No One Behind [22,23]. Data-guidance is a core principle to achieve the
vision of a “world where everyone, everywhere, at every age fully benefits from vaccines for
good health and well-being” [24,25]. The IA2030 highlights the need for improved data, and a
strong base of analysis is essential to implement an effective Monitoring and Evaluation
Framework for the Agenda [26]. However, the current extent of quality problems in reported
immunization data remains unclear. A WHO Strategic Advisory Group of Experts (SAGE)
Working Group on the Quality and Use of Global Immunization and Surveillance Data and
two literature reviews have found a paucity of evidence on immunization data quality [2730].
This study aimed to assess the quality of reported child immunization coverage data for 194
countries between 2000–2019.
2. Materials and methods
2.1 Data sources
All WHO Member States yearly report their childhood immunization coverage to WHO and
the United Nations Children’s Fund (UNICEF) via the Joint Reporting Form on Immuniza-
tion (JRF) [31]. Countries can report two types of data on every vaccine dose in the national
immunization schedule: administrative coverage and official coverage [32]. Administrative
coverage (or admin coverage) is calculated by dividing the number of children who received a
specific vaccine dose during a reported year (numerator) by the number of children who
should have received the vaccine (denominator). Countries usually obtain the numerator from
aggregated doses administered in health facilities and use census projections or similar infor-
mation to estimate the denominators. In contrast, official coverage comprises a percentage
only and is a country’s best estimate of the coverage reached [28]. Thus, reporting official cov-
erage is an opportunity for countries to adjust administrative data or report data from nation-
ally representative surveys or another well-explained source. All countries are expected to
report at least one type of data. Countries that do not have a centralized immunization report-
ing system often only report official estimates [28].
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 2 / 18
Standard/Population/). Mortality estimates are
available from the UN Inter-agency Group for Child
Mortality Estimation (IGME) (https://childmortality.
org/). Data on historical income classifications
(https://datahelpdesk.worldbank.org/
knowledgebase/articles/906519-world-bank-
country-and-lending-groups), and Fragile and
Conflict-Affected Situations (FCS) (https://www.
worldbank.org/en/topic/fragilityconflictviolence/
brief/harmonized-list-of-fragile-situations) can be
downloaded from the World Bank. Estimates of
birth registration levels are provided by the United
Nations Children’s Fund (UNICEF) (https://data.
unicef.org/topic/child-protection/birth-registration/
). Information on country eligibility for support by
Gavi, the Vaccine Alliance, can be found on https://
www.gavi.org/programmes-impact/our-impact/
disbursements-and-commitments. In addition,
immunization coverage input data for this study
and flagging results by country are stored in the
following repository: https://doi.org/10.17605/OSF.
IO/R69EX.
Funding: The authors received no specific funding
for this work.
Competing interests: I have read the journal’s
policy and the authors of this manuscript have the
following competing interests: MCD-H, LBD, MGD,
and JG work for the World Health Organization.
The authors alone are responsible for the views
expressed in this publication and they do not
necessarily represent the decisions, policy, or
views of the World Health Organization. CR is a
board member of the German Society for Tropical
Pediatrics and International Child Health (GTP). RK
is a board member of the German Society for
Paediatric Infectious Diseases (DGPI). RK reports
honoraria for lectures in non-pharmaceutical
educational events in pediatrics, infectious
diseases, and tropical medicine. RK attended an
advisory board meeting on shortage of
immunoglobulins by Shire Germany in 2020. CR
and RK have been involved in clinical trials for
which their employer has received funding from
Pfizer/BioNTech and Sedana Medical. All other
authors declare no competing interests.
We gathered official and administrative coverage (percentage), number of doses given
(numerator), and number of children in the target group (denominator) for Bacillus Calmette-
Gue
´rin (BCG) birth dose, first and third dose of diphtheria-tetanus-pertussis-containing vac-
cine (DTP1, DTP3), and first dose of measles-containing vaccine (MCV1) as reported to
WHO and UNICEF by all WHO Member States for 20 years (2000–2019), as of 15 July 2020.
For year-to-year comparisons, data for the year 1999 were also considered. The four vaccines
were selected because they are almost universally used at birth (BCG), or in infancy as part of a
basic vaccination series (MCV1, DTP1, DTP3).
2.2 Country classifications
Several grouping classifications were used to examine differences in the data quality between
different groups of countries, based on demographic, economic, and political indicators (see
S1 Appendix). Estimates of the countries’ total population, live births cohort, and surviving
infants cohort were obtained from the United Nations Population Division (UNPD) [33].
Information on WHO World Regions, immunization schedules and WHO/UNICEF Esti-
mates on National Immunization Coverage (WUENIC) were provided by WHO [34,35]. Esti-
mates of infant mortality rate (IMR), that is the probability of dying between birth and one
year of age per 1000 live births, were obtained from the UN Inter-agency Group for Child
Mortality Estimation (IGME), as of September 2020 [36]. Historical classification of countries
by income based on gross national income per capita in 2000–2019 [37], and fragile and con-
flict-affected situations (FCS) in 2004–2019 [38] were derived from the World Bank, as of July
2020. Latest estimates of birth registration levels, that is the percentage of children under age
five whose births are registered, were obtained from UNICEF, as of July 2020 [39]. In addition,
we classified countries by whether they had received financial support from Gavi, the Vaccine
Alliance, between 2001–2019 [40].
2.3 Definition of data quality
There is no universally accepted definition of data quality in immunization programs [28,29].
For this analysis, we adopted an approach by Bloland and MacNeil [41] and the SAGE Work-
ing Group on Immunization Data Quality [28] that defined high-quality data as “accurate, pre-
cise, relevant, complete and timely enough for the intended purpose” (Table 1). We used this
Table 1. Dimensions of data quality in immunization programs.
Dimension Definition
Accuracy Closeness of a measurement or estimate to the true value
• Congruence
(proxy)
Degree to which data obtained from different sources agree with each other
• Integrity (proxy) Degree to which data are unchanged from the original
Precision Degree of spread of a series of measurements that is independent of accuracy
• Consistency
(proxy)
Degree to which data attributes are free from contradiction, large fluctuations over time,
and coherent with other data in a specific context of use
Relevancy Degree to which data reflect what is most important for decision-making
Completeness Whether or not all relevant data needed for decision-making are available for use
Timeliness Degree to which data are current and available when needed to inform decisions
Notes: Adapted from Bloland and MacNeil and the SAGE Working Group on the Quality and Use of Immunization
and Surveillance Data [28,41].
https://doi.org/10.1371/journal.pgph.0000140.t001
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 3 / 18
working definition, acknowledging that the dimensions of timeliness and relevancy were not
verifiable directly using the available data.
2.4 Data quality checks
We assessed the quality of reported immunization coverage data in terms of completeness,
consistency, integrity, and congruence, based on existing recommendations by WHO and the
SAGE Working Group combined with our methodology as defined below [28,4246]. We
used a stepwise approach that included 1) checking country data for any given anomaly, 2)
flagging summaries of data that contained anomalies, and 3) modeling the probability of flags
for groups of countries.
First, a series of data quality checks were applied to the reported data, with each check
related to one of the four dimensions of data quality assessed. Forty-two countries without
BCG birth dose in the national routine immunization schedule were excluded from analyses
involving BCG vaccine. Eleven countries with a total population of fewer than 90,000 people
in 2019 were excluded from checks comparing denominators with external sources because
UNPD does not provide estimates of births and surviving infants for these countries. Fourteen
countries that do not use a centralized immunization reporting system were excluded from
analyses involving admin coverage, numerators, or denominators. Finally, we only considered
data for years with membership for countries that became WHO Member States during the
study period. Thus, data from Timor-Leste were assessed beginning in 2002, from Montenegro
beginning in 2006, and from South Sudan beginning in 2011.
2.4.1 Completeness. Completeness was assessed by calculating proportions of missing
numerators, denominators, admin and official percent coverage.
2.4.2 Congruence. Congruence was assessed by comparing BCG denominators to UNPD
estimates of live births with deviations of 10% considered abnormal. The implied infant
mortality rate (IIMR) was calculated as
Implied infant mortality rate ¼number of live births number of surviving infants
number of live births
using BCG denominators as a proxy for the number of live births and DTP1, DTP3, and
MCV1 denominators, respectively, as proxies for the number of surviving infants [42]. Implied
infant mortality rates that were zero or negative, or outside the 90% uncertainty intervals (UI)
of UN-IGME estimates, and denominators for DTP1, DTP3, or MCV1 differing by 10%
from UNPD estimates of surviving infants were considered abnormal.
2.4.3 Consistency. Consistency was assessed by identifying same numerators or denomi-
nators as reported in the preceding year to detect potential copying and pasting of previous
year data, coverage levels equal or over 100%, and data differing from year to year by 10%.
Dropout rates for numerators and coverage levels between DTP1 and DTP3 were calculated as
DTP dropout rate %ð Þ ¼ DTP1DTP3
DTP1100
and considered abnormal if zero or negative.
2.4.4 Integrity. Integrity was assessed by recalculating admin coverage as
Recalculated administrative coverage %ð Þ ¼ numerator
denominator 100:
If the recalculated admin coverage was 100% or did not match the admin coverage, the
numerator and denominator were considered abnormal.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 4 / 18
2.5 Flagging coverage reports
To account for the heterogeneous landscape of immunization systems among countries, we
summarized country-provided information on each vaccine dose for every year as one coun-
try-year-vaccine coverage report (hereinafter referred to as “coverage report”). A coverage
report was flagged if at least one data quality check in the underlying data revealed anomalies.
To prevent countries that reported both admin and official data on the same vaccine dose and
year from being disadvantaged in case of anomalies found in admin but not in official data,
official data that passed the data quality checks could outweigh anomalies in admin data (Fig 1
and S1-S4 Figs in S1 Appendix).
2.6 Statistical analysis
We modeled the probability of data quality flags worldwide, for different vaccine doses, and
separate country groups over time using generalized linear mixed-effects models. The model-
ing strategy is described in detail in the online supplement (see S1 Appendix).
The country variable was used as level-two random effect in all models to account for varia-
tion between countries. Year and vaccine dose were used as random slopes to allow for varying
effects between countries.
First, we estimated the global probability of data quality flags pooled for all years using an
intercept-only model including random effects only. Then, using a main model, we calculated
the probability of data quality flags by vaccine dose, WHO World Region, World Bank income
group, quintile of total population size, immunization coverage level, and support by Gavi, the
Vaccine Alliance. Due to limited data availability, two additional models were calculated that
included the same variables as previously mentioned plus the variable birth registration level
and fragile- and conflict-affected situations (FCS) status, respectively.
Second, we estimated the global time trend using a model without interaction terms. Again,
we used a main model to calculate the probability of data quality flags for each vaccine dose
and country group plus additional models for birth registration level and FCS status.
Third, the same models were fitted; however, the predictor for year used in the interaction
was modeled as a spline with five degrees of freedom to allow plotting over time.
All analyses were conducted using the R language for statistical computing, version 4.0.2
[47]. The glmmTMB package version 1.0.2.1 was used to fit mixed-effects models [48]. Pre-
dicted time trends for data quality flags were calculated using the emmeans package [49] ver-
sion 1.5.2.1 and ggeffects package [50] version 0.16.0. The a priori significance level was set at
p = 0.05.
3. Results
3.1 Data quality checks
All 194 Member States reported immunization coverage data to WHO and UNICEF between
2000–2019. However, our data quality checks revealed anomalies in 47% (26390/55836) of
expected data points involving all reporting countries but one. See Table 2 for descriptive
results of data quality checks.
Completeness rates were similar among different types of data. Where data were available,
checks for congruence showed the highest frequencies of anomalies, followed by consistency
and integrity.
Denominators were most affected with a 91% (12568/13744) rate of abnormal data quality
check results, followed by admin coverage with 40% (5439/13744), official coverage with 33%
(4814/14604), and numerators with 26% (3569/13744).
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 5 / 18
Over the period analyzed, DTP1 was the vaccine dose most affected by data quality issues
with a 59% (8646/14604) rate of abnormal data quality check results, followed by DTP3 with
48% (6977/14604), BCG with 41% (4965/12024), and MCV1 with 40% (5802/14604). See S1
Table in S1 Appendix.
3.2 Modeled probability of data quality flags
The overall probability of data quality flags in immunization coverage reports was 18.2%
globally (95% confidence interval [CI] 14.8–22.3) between 2000–2019 (Fig 2 and S2 Table in
S1 Appendix).
Three out of six WHO Regions showed a statistically significant difference from the mean
of this country classification. South-East Asia had a significantly lower probability (6.3%, 3.3–
11.8, p = 0.002). The highest probabilities were seen in the Americas (29.7%, 22.7–37.9,
p<0.001) and in Africa (23.2%, 17.3–3.5, p = 0.013). Upper-middle income countries had a
statistically significant lower probability of data quality flags than the classification mean
(12.8%, 9.8–16.4, p = 0.033). Countries with an immunization coverage level of 90% to under
95% had a statistically significant lower probability of data quality flags than the classification
mean (11.2%, 8.1–15.2, p = 0.024).
There was no statistically significant difference in the probability of data quality flags by
population size quintile, birth registration level, FCS status, or support by Gavi, the Vaccine
Alliance, and the corresponding classification mean.
The probability of data quality flags across vaccine doses ranged from 8.2% for BCG birth
dose (95% CI 6.1–10.8, p <0.001) to 36.0% for DTP1 (95% CI 29.8–42.6, p <0.001). Coverage
Countries without centralized immunization
reporting system that report official coverage
data only (n = 14)
Countries with centralized immunization
reporting system that report admin and
official coverage data (n = 180)
One or more potential data quality problems in
official coverage for coverage report found
One or more potential data quality problems in
administrative coverage or numerator or
denominator for coverage report found
Coverage report flagged Coverage report not flagged
yes no
yes
no
Fig 1. Method for flagging the quality of reported immunization data by type of country immunization reporting system for
194 WHO Member States. Notes: Countries without a centralized immunization reporting system were: Andorra, Austria,
Belgium, Canada, Finland, France, Germany, Greece, Luxemburg, Monaco, Norway, Sweden, Switzerland, and the United States of
America. For detailed flowcharts listing all data quality checks for each vaccine dose by type of country immunization reporting
system see S1-S4 Figs in S1 Appendix.
https://doi.org/10.1371/journal.pgph.0000140.g001
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 6 / 18
Table 2. Immunization coverage data points and reporting countries affected by potential data quality issues, by
dimension of data quality, 194 WHO Member States, 2000–2019.
Data points affected/
data points reported
(%)
Countries affected/
countries reporting
(%)
Completeness
Type of data point missing
Admin coverage missing 1123/13744 (8%) 124/180 (69%)
Denominator missing 1445/13744 (11%) 134/180 (74%)
Official coverage missing 1609/14604 (11%) 150/194 (77%)
Numerator missing 1532/13744 (11%) 137/180 (76%)
Total data points missing 5709/55836 (10%) 172/194 (89%)
Congruence
DTP3 denominator versus UNPD surviving infants 10% 1042/3104 (34%) 139/171 (81%)
MCV1 denominator versus UNPD surviving infants 10% 1046/3049 (34%) 145/171 (85%)
DTP1 denominator versus UNPD surviving infants 10% 973/2826 (34%) 132/166 (80%)
BCG denominator versus UNPD live births 10% 1061/2846 (37%) 133/161 (83%)
Implied IMR using MCV1 zero or negative 1176/2892 (41%) 159/168 (95%)
Implied IMR using DTP1 zero or negative 1304/2761 (47%) 159/166 (96%)
Implied IMR using DTP3 zero or negative 1416/2940 (48%) 162/169 (96%)
Implied IMR using DTP1 outside 90% UI of UN-IGME IMR 2315/2634 (88%) 158/158 (100%)
Implied IMR using DTP3 outside 90% UI of UN-IGME IMR 2483/2813 (88%) 161/161 (100%)
Implied IMR using MCV1 outside 90% UI of UN-IGME IMR 2446/2769 (88%) 160/160 (100%)
Consistency
Same numerator as in preceding year 58/11389 (1%) 22/179 (12%)
Same denominator as in preceding year 269/11468 (2%) 51/180 (28%)
Official coverage 100% 588/12995 (5%) 88/194 (45%)
Admin coverage 100% 998/12621 (8%) 104/180 (58%)
Recalculated admin coverage 100% 1125/12159 (9%) 118/180 (66%)
Denominator year-to-year difference 10% 1404/11360 (12%) 150/180 (83%)
Official coverage year-to-year difference 10% 1579/12191 (13%) 133/194 (69%)
Admin coverage year-to-year difference 10% 1651/12018 (14%) 139/178 (78%)
DTP1 to DTP3 doses dropout rate zero or negative 486/2945 (17%) 113/173 (65%)
DTP1 to DTP3 admin coverage dropout rate zero or negative 623/3027 (21%) 125/174 (72%)
DTP1 to DTP3 official coverage dropout rate zero or negative 662/3085 (21%) 136/187 (73%)
Integrity
Admin coverage different from recalculated admin coverage 1214/12116 (10%) 169/180 (94%)
All checks combined
Type of data point
Numerator 3569/13744 (26%) 176/180 (98%)
Official coverage 4814/14604 (33%) 192/194 (99%)
Administrative coverage 5439/13744 (40%) 180/180 (100%)
Denominator 12568/13744 (91%) 180/180 (100%)
Total 26390/55836 (47%) 193/194 (99%)
Notes: Country data as reported by 15 July 2020. The number of combined data quality checks applied to each type of
data point was not equal, thus data should be interpreted with caution. BCG = Bacillus Calmette-Gue
´rin vaccine
birth dose. DTP1 = first dose of diphtheria-tetanus-pertussis-containing vaccine. DTP3 = third dose of diphtheria-
tetanus-pertussis-containing vaccine. IMR = infant mortality rate. MCV1 = first dose of measles-containing vaccine.
UI = uncertainty interval. UN-IGME = United Nations Inter-agency Group for Child Mortality Estimation.
UNPD = United Nations Population Division.
https://doi.org/10.1371/journal.pgph.0000140.t002
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 7 / 18
WHO World Region
Global
0% 10% 20% 30% 40% 50%
Overall
South-East Asia Region
European Region
Eastern Mediterranean Region
Western Pacific Region
African Region
Region of the Americas
Upper-middle income
Lower-middle income
High income
Low income
Highest quintile
Third quintile
Second quintile
Fourth quintile
Lowest quintile
90% to under 95% immunized
95% and more immunized
Below 80% immunized
80% to under 90% immuniz ed
90% to under 95% re
g
istered
Below 80% registered
95% and more registered
80% to under 90% registered
FCS status: no
FCS status: yes
Gavi support: yes
Gavi su
pp
ort: no
Probability of data quality flags
World Bank income group
Population size
Immunization coverage level
Birth registration level
FCS
st
s: yes
Fragile and conflict-affected situati ons (FCS)
Support by Gavi, the Vaccine Alliance
Country classifications:
Global
WHO World Region
World Bank income group
Population size
Immunization coverage level
Birth registration level
Fragile and conflict-affected
situations (FCS)
Support by Gavi, the Vaccine
Alliance
Fig 2. Modeled probability of data quality flags for immunizationcoverage reports for DTP1, DTP3, MCV1, and BCG
worldwide and by different country classifications, 194 WHO Member States, 2000–2019. Notes: Error bars represent 95%
confidence intervals (CI). Country data as reported by 15 July 2020. BCG = Bacillus Calmette-Gue
´rin vaccine birth dose.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 8 / 18
reports for BCG birth dose and for MCV1 (9.0%, 95% CI 7.1–11.5, p <0.001) had a statistically
significant lower probability of data quality flags, while reports for DTP1 and DTP3 (19.3%,
95% CI 15.9–23.3, p <0.001) had a statistically significant higher probability of data quality
flags (S3 Table and S5 Fig in S1 Appendix).
3.3 Trends of the probability of data quality flags
The global probability of data quality flags declined by 5.1% per year (95% CI 3.2–7.0,
p<0.001) between 2000–2019 (Figs 3and 4, and S4 Table in S1 Appendix).
Three out of six WHO World Regions showed a statistically significant decrease in the
probability of data quality flags. The steepest decline was seen in Africa (-9.6%, -13.0 to -5.8,
p<0.001), followed by Europe (-5.4%, -9.2 to -1.6, p = 0.0055), and the Americas (-4.9%, -9.2
to -0.6, p = 0.026).
All other country groups showed statistically significant declines except lower-middle
income countries, countries within the highest quintile of population size, countries with an
immunization coverage level below 80%, countries with a birth registration rate between 80%
to under 95%, and fragile and conflict-affect situations (FCS). No country group showed a sta-
tistically significant overall increase. However, some groups experienced a rise in the year 2019
(Fig 4).
The yearly trend of the probability of data quality flags across vaccine doses ranged from
-6.7% for DTP1 (95% CI -9.1 to -4.4, p <0.001) to -4.3% for MCV1 (95% CI -6.9 to -1.7,
p<0.001). All vaccine doses saw a statistically significant decline (S5 Table and S6 Fig in
S1 Appendix).
4. Discussion
This study provides a comprehensive quality assessment of child immunization coverage data
for all 194 WHO Member States using publicly available reported data.
About one in five vaccine coverage reports sent to WHO/UNICEF between 2000–2019 con-
tained data that warrant further quality investigation. Across WHO World Regions, South-
East Asia had the lowest rate of potential data quality issues, while the share of quality flags in
reports from the Americas and Africa was higher than the global mean. In addition, we found
fewer potential data quality issues in data from countries with an immunization coverage level
of 90% to under 95%, and upper-middle income countries. Reports of BCG and MCV1 had
lower rates of potential data quality problems, whereas DTP1 and DTP3 had higher rates.
The overall findings were consistent with previous evidence [18,5160]. However, most
existing studies were limited to specific groups of countries [5255,59], or to comparisons
with external data such as household surveys [52,54,55], or with population estimates [56,
5860]. Compared to prior work, this analysis found lower rates of consistency problems. For
example, previous assessments by WHO and Stashko et al. found year-to-year differences of at
least ten percent in up to 20% of coverage data, compared to 14% for admin coverage and 13%
for official coverage in this study [51,60]. A 2009 assessment in the Americas found negative
DTP1 = first dose of diphtheria-tetanus-pertussis-containing vaccine. DTP3 = third dose of diphtheria-tetanus-pertussis-
containing vaccine. FCS = fragile and conflict-affected situations. MCV1 = first dose of measles-containing vaccine. Countries
were grouped separately for each year by World Bank income groups, population size, and fragile and conflict-affected situations
(FCS) classification. All other groupings were done for all years together. FCS status was availablefor 2004–2019 only.
Immunization coverage level was based on average DTP1 and DTP3 coverage estimated by WHO and UNICEF for 2017–2019, as
of July 2020. Birth registration levels refer to children under age five who have been registered based on the latest available
UNICEF estimate. Support by Gavi, the Vaccine Alliance, refers to funding in any year between 2000–2019. See S1 Appendix for
detailed lists of countries.
https://doi.org/10.1371/journal.pgph.0000140.g002
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 9 / 18
-15% -10% -5% 0% 5% 10% 15%
Overall
African Region
Af i R i
European Region
Western Pacific Region
W P ifi R i
Region of the Americas
f
Eastern Mediterranean Region
Mdi R i
South-East Asia Region
High income
Low income
Upper-middle income
Lower-middle income
Second quintile
Lowest quintile
Fourth quintile
Third quintile
Highest quintile
80% to under 90% immuniz ed
90% to under 95% immuniz ed
95% and more immunized
Below 80% immunized
Below 80% registered
80% to under 90% registered
95% and more registered
90% to under 95% registered
FCS status: yes
FCS status: no
Gavi support: yes
Gavi support: no
Trend of probability of data quality flags
World Bank income group
Population size
Immunization coverage level
Birth registration level
FCS status:
no
Fragile and conflict-affected situati ons (FCS)
Support by Gavi, the Vaccine Alliance
South
East Asia Region
WHO World Region
Global
Country classifications:
Global
WHO World Region
World Bank income group
Population size
Immunization coverage level
Birth registration level
Fragile and conflict-affected
situations (FCS)
Support by Gavi, the Vaccine
Alliance
Fig 3. Modeled trends of the probability of data quality flags for immunization coverage reportsfor DTP1, DTP3, MCV1,
and BCG, by different country classifications, 194 WHO Member States, 2000–2019. Notes: Colored lines represent 95%
confidence intervals (CI). Country data as reported by 15 July 2020. BCG = Bacillus Calmette-Gue
´rin vaccine birth dose.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 10 / 18
dropout rates between DTP1 and DTP3 doses in 34% of cases [18], compared to 17% in this
study. In contrast, the frequency of congruence and completeness issues was similar to previ-
ous analyses. For example, the proportion of deviations of BCG denominators from UNPD
live birth estimates in the literature ranged from 27% in the Americas [18], to 37% globally
[60], and 49% in Sub-Saharan Africa [61], with the finding of 37% in this study in between.
Stashko et al. saw admin coverage levels of 100% or above in 6%–11% of reporting events,
compared to 8% in this assessment [60].
While existing evidence repeatedly found data quality issues in immunization coverage
data, the underlying causes have rarely been investigated [27,29]. For example, population
groups may be systematically excluded from accessing care, or data from people vaccinated in
the private sector may never be reported and thus not included in the calculation of adminis-
trative coverage [29]. Some studies have used household surveys to assess reported coverage.
Although surveys are helpful to complement program data, they cannot guide program man-
agement alone as they are infrequent, expensive, often limited in geographic representation,
and themselves not free from data quality concerns [62]. Official coverage levels reported to
WHO and UNICEF may not contain any accompanying explanation, and what approach of
estimation is used or how the latter changes over time may be unknown [28]. Coverage may
drop or increase by more than 10% from one year to the next if a vaccine stock-out or a decline
in vaccine acceptance occurs [30]. The same can occur to target population estimates due to
mass migration, conflict, or humanitarian crises [28,42,60]. Data can also fluctuate when
dealing with small population sizes [42]. Thus, interpreting quality in countries with small
populations needs to be done with caution.
This study found potential data quality issues in all country groups. However, further stud-
ies are needed to explore the underlying causes. Further exploration is required to investigate
likely differences in the data quality across vaccines, especially whether the wide use of DTP
coverage as a marker of immunization performance plays a role [28,63]. In-depth analyses of
the relationship between data quality and potential inequality factors, such as subnational state
or district, ethnicity, and gender, are also needed. Some country groups showed an apparent
upward trend in potential data quality issues in 2019. WHO and UNICEF collected the data
for 2019 in the second quarter of 2020. Research is needed to determine whether increased
rates of potential data quality problems in 2019 may be due to limited capacity for reporting
during the COVID-19 pandemic in 2020. It should also be explored whether 2020 data col-
lected in 2021 was more affected by delays and inconsistencies.
While our analysis examined data quality from a global perspective, identifying problematic
data is a challenge once it has entered the reporting chain. Consequently, data quality assess-
ments should be conducted at all levels, beginning at the point of vaccination. Here, data are
often not available, not analyzed, or not suitable for decision-making [1,64]. Time-consuming
or duplicative documentation can overburden health workers and inhibit efficient workflows
[2]. In addition, data quality might suffer if people are not empowered to use and benefit from
the data they collect. Innovative tools, better guidance, and continuous on-the-job training are
DTP1 = first dose of diphtheria-tetanus-pertussis-containing vaccine. DTP3 = third dose of diphtheria-tetanus-pertussis-
containing vaccine. FCS = fragile and conflict-affected situations. MCV1 = first dose of measles-containing vaccine. Countries
were grouped separately for each year by World Bank income groups, population size, and fragile and conflict-affected situations
(FCS) classification. All other groupings were done for all years together. FCS status was availablefor 2004–2019 only.
Immunization coverage level was based on average DTP1 and DTP3 coverage estimated by WHO and UNICEF for 2017–2019,
as of July 2020. Birth registration levels refer to children under age five who have been registered based on the latest available
UNICEF estimate. Support by Gavi, the Vaccine Alliance, refers to funding in any year between 2000–2019. See S1 Appendix for
detailed lists of countries.
https://doi.org/10.1371/journal.pgph.0000140.g003
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 11 / 18
FCS status: yes FCS status: no Gavi support: no Gavi support: yes
80% to under 90% registered 95% and more registered Below 80% registered 90% to under 95% registered
80% to under 90% immunized Below 80% immunized 95% and more immunized 90% to under 95% immunized Global
Lowest quintile Fourth quintile Second quintile Third quintile Highest quint ile
Eastern Mediterranean Region European Region South-East Asia Region Lower-middle income Upper-middle income
Region of the Americas African Region Western Pacific Region Low income High income
2000
0
2005
5
2010
0
2015 2000
0
2005
5
2010
0
2015 2000
0
2005
5
2010
0
2015 2000
0
2005
5
2010
0
2015
2000
0
2005
5
2010
0
2015
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Year
Probabilit
y
of data qualit
y
fla
g
s
Country classifications:
Global
WHO World Region
World Bank income
group
Population size
Immunization coverage
level
Birth registration level
Fragile and conflict-
affected situations
(FCS)
Support by Gavi, the
Vaccine Alliance
Fig 4. Modeled trends of the probability of data quality flags for immunization coverage reportsfor DTP1, DTP3, MCV1, and BCG, by
different country classifications, 194 WHO Member States, 2000–2019. Notes: Shading represents 95% confidence intervals (CI). Country
data as reported by 15 July 2020. BCG = Bacillus Calmette-Gue
´rin vaccine birth dose. DTP1 = first dose of diphtheria-tetanus-pertussis-
containing vaccine. DTP3 = third dose of diphtheria-tetanus-pertussis-containing vaccine. FCS = fragile and conflict-affected situations.
MCV1 = first dose of measles-containing vaccine. Countries were grouped separately for each year by World Bank income groups, population
size, and fragile and conflict-affected situations (FCS) classification. All other groupings were done for all years together. FCS status was
available for 2004–2019 only. Immunization coverage level was based on average DTP1 and DTP3 coverage estimated by WHO and UNICEF
for 2017–2019, as of July 2020. Birth registration levels refer to children under age five who have been registered based on the latest available
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 12 / 18
needed to facilitate change and create a “data use culture” and promote "data literacy" [44].
Further, operational research should be promoted to identify barriers to change and create evi-
dence-based interventions to increase data quality [65,66]. This should include learning from
other health sectors and programs. For example, research on target population estimates for
antenatal care or mass drug administration such as vitamin A supplementation or deworming
campaigns can guide quality improvements of denominators in immunization campaigns
[44].
While the quality of global immunization coverage data has improved, countries have not
made equal progress. More could be learned from WHO World Regions and countries that
have reduced quality flags. Moving forward, the global community should prioritize lower-
middle income countries, larger countries, fragile and conflict-affected situations, and coun-
tries with low immunization performance to improve immunization data quality.
Tackling data quality requires increased commitment from stakeholders at all levels, includ-
ing national governments, the non-profit sector, and organizations within the UN system [3].
Building on previous guidance [6771], WHO has published a technical package of five essen-
tial interventions in 2021 to strengthen country health information systems, including advice
on tools and standards to improve data quality [2,72]. Also, WHO and UNICEF have
launched a cloud-based platform for countries to check and report their immunization data
online (eJRF) which replaced the Excel-based Joint Reporting Form on Immunization [73] fea-
turing pre-populated historical data, real-time validation checks, automatic calculations, and a
module to collect data on COVID-19 immunization rates monthly [73].
4.1 Strengths and limitations
This study has several strengths. First, to our knowledge, it is the first comprehensive global
assessment of reported child immunization data quality since 1998 for all 194 WHO Member
States. Second, it uses a systematic approach based on a multi-dimensional definition of quality
and a set of checks derived from public recommendations. Third, we present global estimates
on immunization coverage data quality, allowing for trend analysis and comparisons between
country groups.
This study also has limitations. First, the lack of a gold standard for measuring immuniza-
tion coverage makes it impossible to directly ascertain the data quality dimension of accuracy.
Therefore, like other studies, we used congruence and integrity as proxies for measuring accu-
racy, and consistency as a proxy for measuring precision. Second, as the underlying cause for
anomalies in the data was unknown, “false positive” flags may have been assigned, and some
coverage reports that were not flagged may be problematic. Third, our analysis focused only
on reported national-level coverage and the modeling strategy was designed to derive esti-
mates for groups of countries, not for individual countries. Fourth, as the number of external
data sources and quality checks available was not equal for all vaccine doses and types of data,
comparisons between different vaccine doses and data types should be interpreted with cau-
tion. Finally, this analysis refers to a pre-COVID-19 immunization era. Data on the year 2020
may reveal further quality disturbances not yet captured by this study. Increasing trends in
data quality problems in 2019 should be discussed cautiously, as these estimates are less well
supported by surrounding values.
UNICEF estimate. Support by Gavi, the Vaccine Alliance, refers to funding in any year between 2000–2019. See S1 Appendix for detailed lists
of countries.
https://doi.org/10.1371/journal.pgph.0000140.g004
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 13 / 18
5. Conclusions
Over the past two decades, the quality of global immunization coverage data has improved.
However, progress has not been universal. While Africa showed the fastest improvements, fol-
lowed by Europe and the Americas, other country groups had stagnating proportions of poten-
tially problematic data. These findings can inform national and international policy and action
to strengthen national immunization information systems. The adoption of the Immunization
Agenda 2030 provides momentum to enter the next era of global vaccination programs driven
by high-quality data.
Supporting information
S1 Appendix. Online supplement including modeling strategy, supplementary tables, sup-
plementary figures, and lists of countries.
(DOCX)
Acknowledgments
The authors would like to thank Dr Pola Hahlweg and Franziska Badenschier for their com-
ments on a draft version of this manuscript.
Author Contributions
Conceptualization: Cornelius Rau, Robin Kobbe, M. Carolina Danovaro-Holliday.
Data curation: Cornelius Rau, Laure B. Dumolard.
Formal analysis: Cornelius Rau, Daniel Lu¨decke.
Investigation: Cornelius Rau.
Methodology: Cornelius Rau, Daniel Lu¨decke, Laure B. Dumolard, Jan Grevendonk, Robin
Kobbe, Marta Gacic-Dobo, M. Carolina Danovaro-Holliday.
Project administration: Cornelius Rau.
Resources: Cornelius Rau.
Software: Cornelius Rau, Daniel Lu¨decke.
Supervision: Robin Kobbe, Marta Gacic-Dobo, M. Carolina Danovaro-Holliday.
Validation: Cornelius Rau, Daniel Lu¨decke, Brenton M. Wiernik, M. Carolina Danovaro-
Holliday.
Visualization: Cornelius Rau, Daniel Lu¨decke.
Writing – original draft: Cornelius Rau, Daniel Lu¨decke, M. Carolina Danovaro-Holliday.
Writing – review & editing: Cornelius Rau, Daniel Lu¨decke, Laure B. Dumolard, Jan Greven-
donk, Brenton M. Wiernik, Robin Kobbe, Marta Gacic-Dobo, M. Carolina Danovaro-
Holliday.
References
1. Phumaphi J, Mason E, Alipui NK, Cisnero JR, Kidu C, Killen B, et al. A crisis of accountability for wom-
en’s, children’s, and adolescents’ health. The Lancet. Lancet Publishing Group; 2020. pp. 222–224.
https://doi.org/10.1016/S0140-6736(20)31520-8 PMID: 32673598
2. World Health Organization. SCORE for Health Data Technical Package: Global report on health data
systems and capacity, 2020. Geneva; 2021.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 14 / 18
3. United Nations. Data Strategy of the Secretary General for Action by Everyone, Everywhere with
Insight, Impact and Integrity, 2020–22. New York; 2020.
4. Every Woman Every Child. Global Strategy for Women’s, Children’s and Adolescents’ Health 2016–
2030. New York; 2015.
5. The United Nations General Assembly. Transforming our world: the 2030 Agenda for Sustainable
Development. Resolution A/RES/70/1. New York; 2015.
6. The World Bank. Tracking Universal Health Coverage: 2017 Global Monitoring Report. Washington D.
C.; 2017.
7. United Nations Children’s Fund. Is every child counted? Status of data for children in the SDGs. New
York; 2017.
8. Kim JY. Data for better health—and to help end poverty. Lancet. 2012; 380: 2055. https://doi.org/10.
1016/S0140-6736(12)62162-X PMID: 23245597
9. Chan M. From new estimates to better data. Lancet. 2012; 380: 2054. https://doi.org/10.1016/S0140-
6736(12)62135-7 PMID: 23245596
10. Vander Stichele RH, Hay C, Fladvad M, Sturkenboom MCJM, Chen RT. How to ensure we can track
and trace global use of COVID-19 vaccines? Vaccine. Elsevier Ltd; 2021. pp. 176–179. https://doi.org/
10.1016/j.vaccine.2020.11.055 PMID: 33293161
11. Strategic Advisory Group of Experts on Immunization. The Global Vaccine Action Plan 2011–2020.
Review and lessons learned. Geneva; 2019.
12. Chard AN, Gacic-Dobo M, Diallo MS, Sodha S V., Wallace AS. Routine Vaccination Coverage—World-
wide, 2019. MMWR Morb Mortal Wkly Rep. 2020; 69: 1706–1710. https://doi.org/10.15585/mmwr.
mm6945a7 PMID: 33187395
13. World Health Assembly. WHO Expanded Programme on Immunization. Resolution WHA27.57.
Geneva; 1974.
14. World Health Organization. Global Vaccine Action Plan. Geneva; 2013.
15. Causey K, Fullman N, Sorensen RJD, Galles NC, Zheng P, Aravkin A, et al. Estimating global and
regional disruptions to routine childhood vaccine coverage during the COVID-19 pandemic in 2020: a
modelling study. Lancet. 2021;0. https://doi.org/10.1016/S0140-6736(21)01337-4 PMID: 34273292
16. World Health Organization. COVID-19 pandemic leads to major backsliding on childhood vaccinations,
new WHO, UNICEF data shows. 2021 [cited 15 Jul 2021]. Available: https://www.who.int/news/item/
15-07-2021-covid-19-pandemic-leads-to-major-backsliding-on-childhood-vaccinations-new-who-
unicef-data-shows
17. Pan American Health Organization. XV. Meeting of the Technical Advisory Group on Vaccine-prevent-
able Diseases. Municipalities: Improving Immunization Services [Final Report]. Washington D.C.; 2002.
18. Danovaro-Holliday MC, Pacis Tirso CL, Kurtis H. XVIII Meeting of the Technical Advisory Group on Vac-
cine-preventable Diseases. Immunization in the Americas: Prioritizing Vulnerable Populations [Abstract
Book]. Washington D.C.: Pan American Health Organization; 2009.
19. Strategic Advisory Group of Experts on Immunization. Global Vaccine Action Plan. Assessment Report
2013. Geneva; 2013.
20. Strategic Advisory Group of Experts on Immunization. Global Vaccine Action Plan. Assessment Report
2015. Geneva; 2015.
21. Alliance Gavi. Risk & Assurance Report 2020. Geneva; 2020.
22. Lancet The. 2021: the beginning of a new era of immunisations? Lancet. 2021; 397: 1519. https://doi.
org/10.1016/S0140-6736(21)00900-4 PMID: 33894817
23. Nations United. UN News: New UN-led global immunization push aims to save more than 50 million
lives. 2021 [cited 26 Apr 2021]. Available: https://news.un.org/en/story/2021/04/1090592
24. World Health Organization. Immunization Agenda 2030: A Global Strategy to Leave No One Behind.
Geneva; 2020.
25. World Health Assembly. Immunization Agenda 2030. Decision WHA73(9). Geneva; 2020.
26. World Health Organization. Implementing the Immunization Agenda 2030: A Framework for Action
through Coordinated Planning, Monitoring & Evaluation, Ownership & Accountability, and Communica-
tions & Advocacy. Geneva; 2021 [cited 31 Mar 2021]. Available: http://www.immunizationagenda2030.
org/framework-for-action
27. Wetherill O, Lee C, Dietz V. Root Causes of Poor Immunisation Data Quality and Proven Interventions:
A Systematic Literature Review. Ann Infect Dis Epidemiol. 2017; 2: 1012.
28. Strategic Advisory Group of Experts on Immunization (SAGE). Report of the SAGE Working Group on
Quality and Use of Immunization and Surveillance Data. Geneva; 2019.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 15 / 18
29. Harrison K, Rahimi N, Danovaro-Holliday MC. Factors limiting data quality in the expanded programme
on immunization in low and middle-income countries: A scoping review. Vaccine. Elsevier Ltd;
2020. pp. 4652–4663. https://doi.org/10.1016/j.vaccine.2020.02.091 PMID: 32446834
30. Scobie HM, Edelstein M, Nicol E, Morice A, Rahimi N, MacDonald NE, et al. Improving the quality and
use of immunization and surveillance data: Summary report of the Working Group of the Strategic Advi-
sory Group of Experts on Immunization. Vaccine. Elsevier Ltd; 2020. pp. 7183–7197. https://doi.org/10.
1016/j.vaccine.2020.09.017 PMID: 32950304
31. Burton A, Monasch R, Lautenbach B, Gacic-Dobo M, Neill M, Karimov R, et al. WHO and UNICEF esti-
mates of national infant immunization coverage: methods and processes. Bull World Health Organ.
2009; 87: 535–41. https://doi.org/10.2471/blt.08.053819 PMID: 19649368
32. World Health Organization. WHO/UNICEF Joint Reporting Form on Immunization (JRF)—Online
example. 2020 [cited 15 Jul 2020]. Available: https://www.who.int/immunization/monitoring_
surveillance/routine/reporting/WHO_UNICEF_JRF_EN.xls
33. United Nations: Department of Economic and Social Affairs—Population Division. World Population
Prospects 2019, Online Edition. New York; 2019 [cited 23 Feb 2021]. Available: https://population.un.
org/wpp/Download/Standard/Population/
34. World Health Organization. List of WHO Member States. 2020 [cited 15 Jul 2020]. Available: https://
www.who.int/countries
35. World Health Organization. Immunization database. Expanded Programme on Immunization (EPI).
Department of Immunization, Vaccines and Biologicals (IVB). 2020 [cited 15 Jul 2020]. Available:
https://immunizationdata.who.int
36. United Nations Children’s Fund. Levels & Trends in Child Mortality. Estimates developed by the UN
Inter-agency Group for Child Mortality Estimation. Report 2019. United Nations Children’s Fund, World
Health Organization, World Bank Broup and United Nations Population Division. New York; 2020.
37. The World Bank. Classifying countries by income. 2021 fiscal year. In: World Development Indicators
[Internet]. 2020 [cited 17 Aug 2020]. Available: https://datahelpdesk.worldbank.org/knowledgebase/
articles/906519-world-bank-country-and-lending-groups
38. The World Bank. Classification of Fragile and Conflict-Affected Situations. 2020 [cited 15 Jul 2020].
Available: https://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-
situations
39. United Nations Children’s Fund. Birth Registration for Every Child by 2030: Are we on track? New York;
2019.
40. Alliance Gavi. Disbursements and commitments. 2020 [cited 30 Sep 2020]. Available: https://www.gavi.
org/programmes-impact/our-impact/disbursements-and-commitments
41. Bloland P, MacNeil A. Defining & assessing the quality, usability, and utilization of immunization data.
BMC Public Health. 2019; 19: 380. https://doi.org/10.1186/s12889-019-6709-1 PMID: 30947703
42. World Health Organization. Assessing and Improving the Accuracy of Target Population Estimates for
Immunization Coverage. Working Paper. Geneva; 2015.
43. WHO/UNICEF. Immunization Coverage Data. An online training to enhance staff understanding of
immunization data systems and how to use data to make decisions that lead to higher immunization
performance. 2016 [cited 15 Jul 2020]. Available: https://agora.unicef.org/course/info.php?id=2064
44. World Health Organization. Global Framework to Strengthen Immunization and Surveillance Data for
Decision-making. A companion document to the Global Vaccine Action Plan (GVAP). Final draft.
Geneva; 2018.
45. World Health Organization. Public Health Data Triangulation for Immunization and Vaccine-preventable
Disease Surveillance Programs: Draft Framework Document. Geneva; 2019.
46. World Health Organization. Handbook on the Use, Collection, and Improvement of Immunization Data:
Draft for comments. Geneva; 2018.
47. R Core Team. R: A Language and Environment for Statistical Computing. In: R Foundation for Statisti-
cal Computing [Internet]. 2020. Available: https://www.r-project.org
48. Brooks ME, Kristensen K, van Benthem KJ, Magnusson A, Berg CW, Nielsen A, et al. {glmmTMB} Bal-
ances Speed and Flexibility Among Packages for Zero-inflated Generalized Linear Mixed Modeling. R
J. 2017; 9: 378–400.
49. Lenth R. emmeans: Estimated Marginal Means, aka Least-Squares Means. 2020 [cited 24 Apr 2021].
Available: https://cran.r-project.org/package=emmeans
50. Lu¨decke D. ggeffects: Tidy Data Frames of Marginal Effects from Regression Models. J Open Source
Softw. 2018; 3: 772. https://doi.org/10.21105/joss.00772
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 16 / 18
51. World Health Organization. Children’s Vaccine Initiative. Report of the Meeting of the Scientific Advisory
Group of Experts (SAGE), Geneva, 9–11 June 1998. Geneva; 1998.
52. Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination
coverage in 45 countries. Lancet. 2003; 362: 1022–7. https://doi.org/10.1016/S0140-6736(03)14411-X
PMID: 14522532
53. Ronveaux O, Rickert D, Hadler S, Groom H, Lloyd J, Bchir A, et al. The immunization data quality audit:
verifying the quality and consistency of immunization monitoring systems. Bull World Health Organ.
2005; 83: 503–10. PMID: 16175824
54. Lu C, Michaud CM, Gakidou E, Khan K, Murray CJ. Effect of the Global Alliance forVaccines and Immu-
nisation on diphtheria, tetanus, and pertussis vaccine coverage: an independent assessment. Lancet.
2006; 368: 1088–1095. https://doi.org/10.1016/S0140-6736(06)69337-9 PMID: 16997663
55. Lim SS, Stein DB, Charrow A, Murray CJL. Tracking progress towards universal childhood immunisa-
tion and the impact of global initiatives: a systematic analysis of three-dose diphtheria, tetanus, and per-
tussis immunisation coverage. Lancet. 2008; 372: 2031–46. https://doi.org/10.1016/S0140-6736(08)
61869-3 PMID: 19070738
56. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy andquality of immunization
information systems in forty-one low income countries. Trop Med Int Health. 2009; 14: 2–10. https://doi.
org/10.1111/j.1365-3156.2008.02181.x PMID: 19152556
57. Brown DW, Burton AH, Gacic-Dobo M, Karimov R. A Review of Target Population Estimates and
Implied Infant Mortality Rates from National Immunization Programmes During 2000–2010. Open Pub-
lic Health J. 2013;6.
58. Brown DW, Burton AH, Gacic-Dobo M, Karimov RI. A Comparison of National Immunization Pro-
gramme Target Population Estimates with Data from an Independent Source and Differences in Com-
puted Coverage Levels for the Third Dose of DTP Containing Vaccine. World J Vaccines. 2014; 04: 18–
23. https://doi.org/10.4236/wjv.2014.41004
59. Kaiser R, Chakauya JM, Shibeshi ME. Trends in differences between births and surviving infants
reported for immunization program planning and external data sources in Eastern and Southern Africa
2000–2013. Vaccine. 2016; 34: 1148–51. https://doi.org/10.1016/j.vaccine.2015.05.074 PMID:
26057134
60. Stashko LA, Gacic-Dobo M, Dumolard LB, Danovaro-Holliday MC. Assessing the quality and accuracy
of national immunization program reported target population estimates from 2000 to 2016. Chico RM,
editor. PLoS One. 2019; 14: e0216933. https://doi.org/10.1371/journal.pone.0216933 PMID: 31287824
61. Brown DW, Burton AH, Gacic-Dobo M, Mihigo R. Proportionate Target Population Estimates Used by
National Immunization Programmes in Sub-Saharan Africa and Comparison with Values from an Exter-
nal Source. World J Vaccines. 2014; 04: 147–156. https://doi.org/10.4236/wjv.2014.43017
62. Cutts FT, Claquin P, Danovaro-Holliday MC, Rhoda DA. Monitoring vaccination coverage: Defining the
role of surveys. Vaccine. Elsevier Ltd; 2016. pp. 4103–4109. https://doi.org/10.1016/j.vaccine.2016.06.
053 PMID: 27349841
63. World Health Organization. Meeting of the Strategic Advisory Group of Experts on Immunization,
November 2011 –conclusions and recommendations. Relev Epidemiol Hebd. 2012; 87: 1–16. PMID:
22242233
64. MA4Health. Measurement and Accountability for Results in Health: The Roadmap for Health Measure-
ment and Accountability. Washington D.C.; 2015.
65. Pan American Health Organization and PATH. Immunization Data: Evidence for Action. A Realist
Review of What Works to Improve Data Use for Immunization, Evidence from Low- and Middle- Income
Countries. Washington D.C. and Seattle; 2019.
66. Osterman AL, Shearer JC, Salisbury NA. A realist systematic review of evidence from low- and middle-
income countries of interventions to improve immunization data use. BMC Heal Serv Res 2021 211.
2021; 21: 1–16. https://doi.org/10.1186/s12913-021-06633-8 PMID: 34238291
67. World Health Organization. The immunization data quality self-assessment (DQS) tool. Geneva; 2005.
68. World Health Organization. The immunization data quality audit (DQA) procedure. Geneva; 2003.
69. World Health Organization. Data quality review: a toolkit for facility data quality assessment. Module 3:
Data verification and system assessment. Geneva; 2017.
70. World Health Organization. Data quality review: a toolkit for facility data quality assessment. Module 2:
Desk review of data quality. Geneva; 2017.
71. World Health Organization. Data quality review: a toolkit for facility data quality assessment. Module 1:
Framework and metrics. Geneva; 2017.
72. World Health Organization. SCORE for Health Data Technical Package: Tools and Standards for
SCORE Essential Interventions. Geneva; 2020.
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 17 / 18
73. World Health Organization. WHO/UNICEF Joint Reporting Process. 2021 [cited 5 Apr 2021]. Available:
https://www.who.int/teams/immunization-vaccines-and-biologicals/immunization-analysis-and-
insights/global-monitoring/who-unicef-joint-reporting-process
PLOS GLOBAL PUBLIC HEALTH
Immunization coverage and data quality
PLOS Global Public Health | https://doi.org/10.1371/journal.pgph.0000140 February 3, 2022 18 / 18
... Using administrative data for immunization coverage rates can be problematic [17]. The numerator data may suffer from incompleteness due to non-reporting or delayed reporting [18]. ...
Article
Full-text available
While there is a coordinated effort around reaching zero dose children and closing existing equity gaps in immunization delivery, it is important that there is agreement and clarity around how ‘zero dose status’ is defined and what is gained and lost by using different indicators for zero dose status. There are two popular approaches used in research, program design, and advocacy to define zero dose status: one uses a single vaccine to serve as a proxy for zero dose status, while another uses a subset of vaccines to identify children who have missed all routine vaccines. We provide a global analysis utilizing the most recent publicly available DHS and MICS data from 2010 to 2020 to compare the number, proportion, and profile of children aged 12 to 23 months who are ‘penta-zero dose’ (have not received the pentavalent vaccine), ‘truly’ zero dose (have not received any dose of BCG, polio, pentavalent, or measles vaccines), and ‘misclassified’ zero dose children (those who are penta-zero dose but have received at least one other vaccine). Our analysis includes 194,829 observations from 82 low- and middle-income countries. Globally, 14.2% of children are penta-zero dose and 7.5% are truly zero dose, suggesting that 46.5% of penta-zero dose children have had at least one contact with the immunization system. While there are similarities in the profile of children that are penta-zero dose and truly zero dose, there are key differences between the proportion of key characteristics among truly zero dose and misclassified zero dose children, including access to maternal and child health services. By understanding the extent of the connection zero dose children may have with the health and immunization system and contrasting it with how much the use of a more feasible definition of zero dose may underestimate the level of vulnerability in the zero dose population, we provide insights that can help immunization programs design strategies that better target the most disadvantaged populations. If the vulnerability profiles of the truly zero dose children are qualitatively different from that of the penta-zero dose children, then failing to distinguish the truly zero dose populations, and how to optimally reach them, may lead to the development of misguided or inefficient strategies for vaccinating the most disadvantaged population of children.
... 18 It has been reported that the Americas and Africa are more likely to have potential data quality issues. 32 Although the quality of data has improved over the past two decades, 32 data obtained during the COVID-19 pandemic might suffer from further issues. Finally, the study only examined cross-country disparities in routine childhood vaccine coverage, as is typical of prior studies. ...
Article
Full-text available
Background: Global routine childhood vaccine coverage has plateaued in recent years, and the COVID-19 pandemic further disrupted immunisation services. We estimated global and regional inequality of routine childhood vaccine coverage from 2019 to 2021, particularly assessing the impacts of the COVID-19 pandemic. Methods: We used longitudinal data for 11 routine childhood vaccines from the WHO-UNICEF Estimates of National Immunization Coverage (WUENIC), including 195 countries and territories in 2019-2021. The slope index of inequality (SII) and relative index of inequality (RII) of each vaccine were calculated through linear regression to express the difference in coverage between the top and bottom 20% of countries at the global and regional levels. We also explored inequalities of routine childhood vaccine coverage by WHO regions and unvaccinated children by income groups. Findings: Globally between January 1, 2019 and December 31, 2021, most childhood vaccines showed a declining trend in coverage, and therefore an increasing number of unvaccinated children, especially in low-income and lower-middle-income countries. Between-country inequalities existed for all 11 routine childhood vaccine coverage indicators. The SII for the third dose of diphtheria-tetanus-pertussis-containing vaccine (DTP3) coverage was 20.1 percentage points (95% confidence interval: 13.7, 26.5) in 2019, and rose to 23.6 (17.5, 30.0) in 2020 and 26.9 (20.0, 33.8) in 2021. Similar patterns were found for RII results and in other routine vaccines. In 2021, the second dose of measles-containing vaccine (MCV2) coverage had the highest global absolute inequality (31.2, [21.5-40.8]), and completed rotavirus vaccine (RotaC) coverage had the lowest (7.8, [-3.9, 19.5]). Among six WHO regions, the European Region consistently had the lowest inequalities, and the Western Pacific Region had the largest inequalities for many indicators, although both increased from 2019 to 2021. Interpretation: Global and regional inequalities of routine childhood vaccine coverage persisted and substantially increased from 2019 to 2021. These findings reveal economic-related inequalities by vaccines, regions, and countries, and underscore the importance of reducing such inequalities. These inequalities were widened during the COVID-19 pandemic, resulting in even lower coverage and more unvaccinated children in low-income countries. Funding: Bill & Melinda Gates Foundation.
... 52 Despite improved data quality over the last 2 decades, gains were not universal, with resource-constrained countries and those with lower immunization performance continuing to have limited to poorer quality data. 53 Although data on numerous indicators are often collected and reported by countries, we identified few indicators in the resources included in our review that measured data-driven decisionmaking and program planning at the national level. Availability of data does not necessarily translate into action; mechanisms and accountability frameworks to incorporate data into decision-making are needed. ...
Article
Full-text available
Background: Vaccination coverage is widely used to assess immunization performance but, on its own, provides insufficient information to drive improvements. Assessing the performance of underlying components of immunization systems is less clear, with several monitoring and evaluation (M&E) resources available for use in different operational settings and for different purposes. We studied these resources to understand how immunization system performance is measured. Methods: We reviewed peer-reviewed and gray literature published since 2000 to identify M&E resources that include national-level indicators measuring the performance of immunization systems or their components (governance, financing, regulation, information systems, vaccine logistics, workforce, service delivery, and demand generation). We summarize indicators by the system components or outcomes measured and describe findings narratively. Results: We identified 20 resources to monitor immunization program objectives and guide national strategic decision-making, encompassing 631 distinct indicators. Indicators for immunization program outcomes comprised the majority (124/631 [19.7%]), largely vaccination coverage (110/124 [88.7%]). Almost all resources (19/20 [95%]) included indicators for vaccine logistics (83/631 [13.2%]), and those for regulation (19/631 [3.0%]) and demand generation (28/631 [4.4%]) were least common. There was heterogeneity in how information systems (92/563 [14.6%]) and workforce (47/631 [7.4%]) were assessed across resources. Indicators for vaccination coverage in adults, data use in decision-making, equity and diversity, effectiveness of safety surveillance, and availability of a public health workforce were notably lacking. Conclusions: Between the resources identified in this review, we identified considerable variability and gaps in indicators assessing the performance of some immunization system components. Given the multitude of indicators, policymakers may be better served by tailoring evaluation resources to their specific context to gain useful insight into health system performance and improve data use in decision-making for immunization programs.
... Secondly, the different input data sets have their inherent biases (e.g., admin estimates being greater than 100) which are likely the result of inaccurate denominator and/or numerator estimates, large differences between consecutive coverage estimates (in time) and recall bias associated with survey data for multi-dose vaccines (Cutts et al., 2016). A complete overview and analysis of data quality issues associated with these data sources are provided in Rau et al. (2022); Stashko et al. (2019). Although we implemented some ad hoc measures to correct these biases wherever possible, e.g., recall-bias adjustment for survey estimates of DTP3 and PCV3 and rounding down of administrative estimates greater than 100 whilst persevering the differential between multi-dose vaccines, they are better addressed at the point of data collection and summarization. ...
Preprint
Full-text available
Estimates of national immunization coverage are crucial for guiding policy and decision-making in national immunization programs and setting the global immunization agenda. WHO and UNICEF estimates of national immunization coverage (WUENIC) are produced annually for various vaccine-dose combinations and all WHO Member States using information from multiple data sources and a deterministic computational logic approach. This approach, however, is incapable of characterizing the uncertainties inherent in coverage measurement and estimation. It also provides no statistically principled way of exploiting and accounting for the interdependence in immunization coverage data collected for multiple vaccines, countries and time points. Here, we develop Bayesian hierarchical modeling approaches for producing accurate estimates of national immunization coverage and their associated uncertainties. We propose and explore two candidate models: a balanced data single likelihood (BDSL) model and an irregular data multiple likelihood (IDML) model, both of which differ in their handling of missing data and characterization of the uncertainties associated with the multiple input data sources. We provide a simulation study that demonstrates a high degree of accuracy of the estimates produced by the proposed models, and which also shows that the IDML model is the better model. We apply the methodology to produce coverage estimates for select vaccine-dose combinations for the period 2000-2019. A contributed R package {\tt imcover} implementing the No-U-Turn Sampler (NUTS) in the Stan programming language enhances the utility and reproducibility of the methodology.
Article
Full-text available
Background Several studies have indicated that universal health coverage (UHC) improves health service utilization and outcomes in countries. These studies, however, have primarily assessed UHC's peacetime impact, limiting our understanding of UHC's potential protective effects during public health crises such as the Coronavirus Disease 2019 (COVID-19) pandemic. We empirically explored whether countries' progress toward UHC is associated with differential COVID-19 impacts on childhood immunization coverage.Methods and findingsUsing a quasi-experimental difference-in-difference (DiD) methodology, we quantified the relationship between UHC and childhood immunization coverage before and during the COVID-19 pandemic. The analysis considered 195 World Health Organization (WHO) member states and their ability to provision 12 out of 14 childhood vaccines between 2010 and 2020 as an outcome. We used the 2019 UHC Service Coverage Index (UHC SCI) to divide countries into a "high UHC index" group (UHC SCI ≥80) and the rest. All analyses included potential confounders including the calendar year, countries' income group per the World Bank classification, countries' geographical region as defined by WHO, and countries' preparedness for an epidemic/pandemic as represented by the Global Health Security Index 2019. For robustness, we replicated the analysis using a lower cutoff value of 50 for the UHC index. A total of 20,230 country-year observations were included in the study. The DiD estimators indicated that countries with a high UHC index (UHC SCI ≥80, n = 35) had a 2.70% smaller reduction in childhood immunization coverage during the pandemic year of 2020 as compared to the countries with UHC index less than 80 (DiD coefficient 2.70; 95% CI: 0.75, 4.65; p-value = 0.007). This relationship, however, became statistically nonsignificant at the lower cutoff value of UHC SCI
Article
Full-text available
Background: The COVID-19 pandemic and efforts to reduce SARS-CoV-2 transmission substantially affected health services worldwide. To better understand the impact of the pandemic on childhood routine immunisation, we estimated disruptions in vaccine coverage associated with the pandemic in 2020, globally and by Global Burden of Disease (GBD) super-region. Methods: For this analysis we used a two-step hierarchical random spline modelling approach to estimate global and regional disruptions to routine immunisation using administrative data and reports from electronic immunisation systems, with mobility data as a model input. Paired with estimates of vaccine coverage expected in the absence of COVID-19, which were derived from vaccine coverage models from GBD 2020, Release 1 (GBD 2020 R1), we estimated the number of children who missed routinely delivered doses of the third-dose diphtheria-tetanus-pertussis (DTP3) vaccine and first-dose measles-containing vaccine (MCV1) in 2020. Findings: Globally, in 2020, estimated vaccine coverage was 76·7% (95% uncertainty interval 74·3-78·6) for DTP3 and 78·9% (74·8-81·9) for MCV1, representing relative reductions of 7·7% (6·0-10·1) for DTP3 and 7·9% (5·2-11·7) for MCV1, compared to expected doses delivered in the absence of the COVID-19 pandemic. From January to December, 2020, we estimated that 30·0 million (27·6-33·1) children missed doses of DTP3 and 27·2 million (23·4-32·5) children missed MCV1 doses. Compared to expected gaps in coverage for eligible children in 2020, these estimates represented an additional 8·5 million (6·5-11·6) children not routinely vaccinated with DTP3 and an additional 8·9 million (5·7-13·7) children not routinely vaccinated with MCV1 attributable to the COVID-19 pandemic. Globally, monthly disruptions were highest in April, 2020, across all GBD super-regions, with 4·6 million (4·0-5·4) children missing doses of DTP3 and 4·4 million (3·7-5·2) children missing doses of MCV1. Every GBD super-region saw reductions in vaccine coverage in March and April, with the most severe annual impacts in north Africa and the Middle East, south Asia, and Latin America and the Caribbean. We estimated the lowest annual reductions in vaccine delivery in sub-Saharan Africa, where disruptions remained minimal throughout the year. For some super-regions, including southeast Asia, east Asia, and Oceania for both DTP3 and MCV1, the high-income super-region for DTP3, and south Asia for MCV1, estimates suggest that monthly doses were delivered at or above expected levels during the second half of 2020. Interpretation: Routine immunisation services faced stark challenges in 2020, with the COVID-19 pandemic causing the most widespread and largest global disruption in recent history. Although the latest coverage trajectories point towards recovery in some regions, a combination of lagging catch-up immunisation services, continued SARS-CoV-2 transmission, and persistent gaps in vaccine coverage before the pandemic still left millions of children under-vaccinated or unvaccinated against preventable diseases at the end of 2020, and these gaps are likely to extend throughout 2021. Strengthening routine immunisation data systems and efforts to target resources and outreach will be essential to minimise the risk of vaccine-preventable disease outbreaks, reach children who missed routine vaccine doses during the pandemic, and accelerate progress towards higher and more equitable vaccination coverage over the next decade. Funding: Bill & Melinda Gates Foundation.
Article
Full-text available
Background The use of routine immunization data by health care professionals in low- and middle-income countries remains an underutilized resource in decision-making. Despite the significant resources invested in developing national health information systems, systematic reviews of the effectiveness of data use interventions are lacking. Applying a realist review methodology, this study synthesized evidence of effective interventions for improving data use in decision-making. Methods We searched PubMed, POPLINE, Centre for Agriculture and Biosciences International Global Health, and African Journals Online for published literature. Grey literature was obtained from conference, implementer, and technical agency websites and requested from implementing organizations. Articles were included if they reported on an intervention designed to improve routine data use or reported outcomes related to data use, and targeted health care professionals as the principal data users. We developed a theory of change a priori for how we expect data use interventions to influence data use. Evidence was then synthesized according to data use intervention type and level of the health system targeted by the intervention. Results The searches yielded 549 articles, of which 102 met our inclusion criteria, including 49 from peer-reviewed journals and 53 from grey literature. A total of 66 articles reported on immunization data use interventions and 36 articles reported on data use interventions for other health sectors. We categorized 68 articles as research evidence and 34 articles as promising strategies . We identified ten primary intervention categories, including electronic immunization registries, which were the most reported intervention type ( n = 14). Among the research evidence from the immunization sector, 32 articles reported intermediate outcomes related to data quality and availability, data analysis, synthesis, interpretation, and review. Seventeen articles reported data-informed decision-making as an intervention outcome, which could be explained by the lack of consensus around how to define and measure data use. Conclusions Few immunization data use interventions have been rigorously studied or evaluated. The review highlights gaps in the evidence base, which future research and better measures for assessing data use should attempt to address.
Article
Full-text available
Endorsed by the World Health Assembly in 2020, the Immunization Agenda 2030 strives to reduce morbidity and mortality from vaccine-preventable diseases across the life course (1). This report, which updates previous reports (2), presents global, regional,* and national vaccination coverage estimates and trends as of 2019 and describes the number of surviving infants who did not receive the first dose of diphtheria and tetanus toxoids and pertussis-containing vaccine (DTP1) during the first year of life (i.e., zero-dose children), which serves as a proxy for children with poor access to immunization and other health services. Global estimates of coverage with the third dose of DTP (DTP3), the first dose of measles-containing vaccine (MCV1), and the third dose of polio vaccine (Pol3) ranged from 84% to 86% during 2010-2019. Worldwide, 19.7 million children (15%) were not vaccinated with DTP3 in 2019, 13.8 million (70%) of whom were zero-dose children. During 2010-2019, the number of zero-dose children increased in the African, Americas, and Western Pacific regions. Global coverage with the second MCV dose (MCV2) increased from 42% in 2010 to 71% in 2019. During 2010-2019, global coverage with underused vaccines increased for the completed series of rotavirus vaccine (rota), pneumococcal conjugate vaccine (PCV), rubella-containing vaccine (RCV), Haemophilus influenzae type b vaccine (Hib), hepatitis B vaccine (HepB), and human papillomavirus vaccine (HPV). Achieving universal coverage with all recommended vaccines will require tailored, context-specific strategies to reach communities with substantial proportions of zero-dose and incompletely vaccinated children, particularly those in remote rural, urban poor, and conflict-affected communities (3).
Article
Full-text available
Concerns about the quality and use of immunization and vaccine-preventable disease (VPD) surveillance data have been highlighted on the global agenda for over two decades. In August 2017, the Strategic Advisory Group of Experts (SAGE) established a Working Group (WG) on the Quality and Use of Global Immunization and Surveillance Data to review the current status and evidence to make recommendations, which were presented to SAGE in October 2019. The WG synthesized evidence from landscape analyses, literature reviews, country case-studies, a data triangulation analysis, as well as surveys of experts. Data quality (DQ) was defined as data that are accurate, precise, relevant, complete, and timely enough for the intended purpose (fit-for-purpose), and data use as the degree to which data are actually used for defined purposes, e.g., immunization programme management, performance monitoring, decision-making. The WG outlined roles and responsibilities for immunization and surveillance DQ and use by programme level. The WG found that while DQ is dependent on quality data collection at health facilities, many interventions have targeted national and subnational levels, or have focused on new technologies, rather than the people and enabling environments required for functional information systems. The WG concluded that sustainable improvements in immunization and surveillance DQ and use will require efforts across the health system — governance, people, tools, and processes, including use of data for continuous quality improvement (CQI) — and that the approaches need to be context-specific, country-owned and driven from the frontline up. At the country level, major efforts are needed to: (1) embed monitoring DQ and use alongside monitoring of immunization and surveillance performance, (2) increase workforce capacity and capability for DQ and use, starting at the facility level, (3) improve the accuracy of immunization programme targets (denominators), (4) enhance use of existing data for tailored programme action (e.g., immunization programme planning, management and policy-change), (5) adopt a data-driven CQI approach as part of health system strengthening, (6) strengthen governance around piloting and implementation of new information and communication technology tools, and (7) improve data sharing and knowledge management across areas and organizations for improved transparency and efficiency. Global and regional partners are requested to support countries in adopting relevant recommendations for their setting and to continue strengthening the reporting and monitoring of immunization and VPD surveillance data through processes periodic needs assessment and revision processes. This summary of the WG’s findings and recommendations can support “data-guided” implementation of the new Immunization Agenda 2030.
Article
Full-text available
Few public health interventions can match the immense achievements of immunization in terms of mortality and morbidity reduction. However, progress in reaching global coverage goals and achieving universal immunization coverage have stalled; with key stakeholders concerned about the accuracy of reported coverage figures. Incomplete and incorrect data has made it challenging to obtain an accurate overview of immunization coverage, particularly in low- and middle-income countries (LMIC). To date, only one literature review concerning immunization data quality exists. However, it only included articles from Gavi-eligible countries, did not go deep into the characteristics of the data quality problems, and used a narrow ‘data quality’ definition. This scoping review builds upon that work; exploring the “state of data quality” in LMIC, factors affecting data quality in these settings and potential means to improve it. Only a small volume of literature addressing immunization data quality in LMIC was found and definitions of ‘data quality’ varied widely. Data quality was, on the whole, considered poor in the articles included. Coverage numerators were seen to be inflated for official reports and denominators were inaccurate and infrequently adjusted. Numerous factors related to these deficiencies were reported, including health information system fragmentation, overreliance on targets and poor data management processes. Factors associated with health workers were noted most frequently. Authors suggested that data quality could be improved by ensuring proper data collection tools, increasing workers’ capacities and motivation through training and supervision, whilst also ensuring adequate and timely feedback on the data collected. The findings of this scoping review can serve as the basis to identify and address barriers to good quality immunization data in LMICs. Overcoming said barriers is essential if immunization’s historic successes are to continue.
Article
Full-text available
Background: A common means of vaccination coverage measurement is the administrative method, done by dividing the aggregated number of doses administered over a set period (numerator) by the target population (denominator). To assess the quality of national target populations, we defined nine potential denominator data inconsistencies or flags that would warrant further exploration and examination of data reported by Member States to the World Health Organization (WHO) and UNICEF between 2000 and 2016. Methods and findings: We used the denominator reported to calculate national coverage for BCG, a tuberculosis vaccine, and for the third dose of diphtheria-tetanus-pertussis-containing (DTP3) vaccines, usually live births (LB) and surviving infants (SI), respectively. Out of 2,565 possible reporting events (data points for countries using administrative coverage with the vaccine in the schedule and year) for BCG and 2,939 possible reporting events for DTP3, 194 and 274 reporting events were missing, respectively. Reported coverage exceeding 100% was seen in 11% of all reporting events for BCG and in 6% for DTP3. Of all year-to-year percent differences in reported denominators, 12% and 11% exceeded 10% for reported LB and SI, respectively. The implied infant mortality rate, based on the country's reported LB and SI, would be negative in 9% of all reporting events i.e., the country reported more SI than LB for the same year. Overall, reported LB and SI tended to be lower than the UN Population Division 2017 estimates, which would lead to overestimation of coverage, but this difference seems to be decreasing over time. Other inconsistencies were identified using the nine proposed criteria. Conclusions: Applying a set of criteria to assess reported target populations used to estimate administrative vaccination coverage can flag potential quality issues related to the national denominators and may be useful to help monitor ongoing efforts to improve the quality of vaccination coverage estimates.