Effects of an mHealth intervention for community health workers on maternal and child nutrition and health service delivery in India: Protocol for a quasi-experimental mixed-methods evaluation

Abstract

Introduction
Millions of children in India still suffer from poor health and under-nutrition, despite substantial improvement over decades of public health programmes. The Anganwadi centres under the Integrated Child Development Scheme (ICDS) provide a range of health and nutrition services to pregnant women, children <6 years and their mothers. However, major gaps exist in ICDS service delivery. The government is currently strengthening ICDS through an mHealth intervention called Common Application Software (ICDS-CAS) installed on smart phones, with accompanying multilevel data dashboards. This system is intended to be a job aid for frontline workers, supervisors and managers, aims to ensure better service delivery and supervision, and enable real-time monitoring and data-based decision-making. However, there is little to no evidence on the effectiveness of such large-scale mHealth interventions integrated with public health programmes in resource-constrained settings on the service delivery and subsequent health and nutrition outcomes.

Methods and analysis
This study uses a village-matched controlled design with repeated cross-sectional surveys to evaluate whether ICDS-CAS can enable more timely and appropriate services to pregnant women, children <12 months and their mothers, compared with the standard ICDS programme. The study will recruit approximately 1500 Anganwadi workers and 6000+ mother-child dyads from 400+ matched-pair villages in Bihar and Madhya Pradesh. The primary outcomes are the proportion of beneficiaries receiving (a) adequate number of home visits and (b) appropriate level of counselling by the Anganwadi workers. Secondary outcomes are related to improvements in other ICDS services, and knowledge and practices of the Anganwadi workers and beneficiaries.

Ethics and dissemination
Ethical oversight is provided by the Committee for the Protection of Human Subjects at the University of California at Berkeley, and the Suraksha Independent Ethics Committee in India. The results will be published in peer-reviewed journals and analysis data will be made public.

Trial registration number
ISRCTN83902145

Full text

1
NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
Effects of an mHealth intervention for
community health workers on maternal
and child nutrition and health service
delivery in India: protocol for a quasi-
experimental mixed-methods evaluation
Sneha Nimmagadda,1 Lakshmi Gopalakrishnan,1 Rasmi Avula,2 Diva Dhar,3
Nadia Diamond-Smith,4 Lia Fernald,5 Anoop Jain,5 Sneha Mani,2 Purnima Menon,2
Phuong Hong Nguyen,6 Hannah Park,4 Sumeet R Patil, 1 Prakarsh Singh,7
Dilys Walker4
To cite: NimmagaddaS,
GopalakrishnanL, AvulaR,
etal. Effects of an mHealth
intervention for community
health workers on maternal and
child nutrition and health service
delivery in India: protocol for
a quasi-experimental mixed-
methods evaluation. BMJ Open
2019;9:e025774. doi:10.1136/
bmjopen-2018-025774
►Prepublication history and
additional material for this
paper are available online. To
view these les, please visit
the journal online (http:// dx. doi.
org/ 10. 1136/ bmjopen- 2018-
025774).
Received 8 August 2018
Revised 12 December 2018
Accepted 6 February 2019
For numbered afliations see
end of article.
Correspondence to
DrSumeet RPatil;
srpatil@ neerman. org
Protocol
© Author(s) (or their
employer(s)) 2019. Re-use
permitted under CC BY.
Published by BMJ.
ABSTRACT
Introduction Millions of children in India still suffer
from poor health and under-nutrition, despite substantial
improvement over decades of public health programmes.
The Anganwadi centres under the Integrated Child
Development Scheme (ICDS) provide a range of health and
nutrition services to pregnant women, children <6 years
and their mothers. However, major gaps exist in ICDS
service delivery. The government is currently strengthening
ICDS through an mHealth intervention called Common
Application Software (ICDS-CAS) installed on smart
phones, with accompanying multilevel data dashboards.
This system is intended to be a job aid for frontline
workers, supervisors and managers, aims to ensure better
service delivery and supervision, and enable real-time
monitoring and data-based decision-making. However,
there is little to no evidence on the effectiveness of such
large-scale mHealth interventions integrated with public
health programmes in resource-constrained settings on
the service delivery and subsequent health and nutrition
outcomes.
Methods and analysis This study uses a village-matched
controlled design with repeated cross-sectional surveys to
evaluate whether ICDS-CAS can enable more timely and
appropriate services to pregnant women, children <12
months and their mothers, compared with the standard
ICDS programme. The study will recruit approximately
1500 Anganwadi workers and 6000+ mother-child dyads
from 400+ matched-pair villages in Bihar and Madhya
Pradesh. The primary outcomes are the proportion of
beneciaries receiving (a) adequate number of home visits
and (b) appropriate level of counselling by the Anganwadi
workers. Secondary outcomes are related to improvements
in other ICDS services, and knowledge and practices of the
Anganwadi workers and beneciaries.
Ethics and dissemination Ethical oversight is provided
by the Committee for the Protection of Human Subjects at
the University of California at Berkeley, and the Suraksha
Independent Ethics Committee in India. The results will be
published in peer-reviewed journals and analysis data will
be made public.
Trial registration number ISRCTN83902145
INTRODUCTION
Background and problem context
Millions of children in India continue to
suffer from poor health and under-nutrition,
despite decades of government programmes
aimed at reducing this burden and some
impressive gains through these years. In
2015–2016, 36% of children under 5 years of
age were underweight, 38% were stunted and
21% were wasted as per the National Family
Health Survey (NFHS-4), and these numbers
represent only modest improvements over
the past decade. Micronutrient deficiencies
Strengths and limitations of this study
►The study can provide important evidence on wheth-
er and the extent to which a large-scale mHealth
intervention can improve maternal and child health
and nutrition service delivery by the world’s largest
child development programme in India.
►Application of the gold-standard cluster-randomised
controlled trial design was not possible because of
pre-determined programme assignment and rapid
roll-out of the programme. Therefore, to nd attrib-
utable impacts, this evaluation settles for a scienti-
cally less robust but practicable quasi-experimental
design consisting of matched control-villages and
repeated cross-sectional measurements.
►Measurement biases may exist because blinding
is not possible and primary outcomes are mea-
sured subjectively via interview-based recall or
observations.
►Higher order impacts may be underestimated as the
follow-up period of <12 months may be too short for
the intervention to stabilise.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

2NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
are widespread, with more than 58% of preschool chil-
dren suffering from iron deficiency anaemia. Infant and
neonatal mortality rates also remain high at 41 and 30 per
1000 live births respectively, despite substantive reduc-
tions over past decades1
The Integrated Child Development Services Scheme
(ICDS), launched in 1975, is one of India’s national flag-
ship programmes to support the health, nutrition, and
development needs of children below 6 years of age
and pregnant and lactating women, through a network
of Anganwadi Centres (AWCs), each typically serving a
population of 800–1000.2 3 Early observational studies
found that ICDS is associated with better coverage and
delivery of services related to nutrition, healthcare, and
pre-school education and improved maternal and child
nutrition.4–6 Using the NFHS data from 2005 to 2006,
Kandpal7 and Jain8 found that ICDS is associated with
small to modest improvements in child health and nutri-
tion, especially among the most vulnerable populations.
However, several reviews and evaluations of ICDS over
the past 18 years have also found persistent gaps, including
inadequate infrastructure at the AWC, Anganwadi worker
(AWW) service delivery issues (eg, poor quality supple-
mentary food, few home visits and no counselling etc),
human resource issues (eg, vacancies, increasing range
of duties expected of the AWWs, inadequate training of
AWWs, limited supervision etc) and poor data manage-
ment (eg, irregularities in record keeping at AWCs, inef-
fective monitoring of service delivery etc).3 4 9–11 The most
recent NFHS (2015–206) also highlights the gaps in ICDS
service delivery. Only about 59% of children under 6
years received any service from an AWC, 53% received
supplementary food services and 47% were weighed.
Similarly, only 60% of mothers received any AWC services
during pregnancy, and 54% received any service during
the breastfeeding phase.1
With a goal to improve the functioning of ICDS,
the Government of India launched the ICDS Systems
Strengthening and Nutrition Improvement Programme
(ISSNIP) in 2012 which focused on infrastructure upgra-
dation and training of AWWs to build their knowledge
on health and nutrition topics under the Incremental
Learning Approach. At the same time, a pilot-scale
mHealth intervention to improve ICDS service delivery
was implemented in Bihar between 2012 and 2013. A
randomised controlled trial of this intervention found
a significant increase in the proportion of beneficiaries
receiving visits from frontline workers at different life-
stages - last trimester of pregnancy (42% vs 52%), first
week after delivery (60% vs 73%) and complementary
feeding stage >5 months after delivery (36% vs 45%).12
The intervention also significantly increased the propor-
tion of beneficiaries receiving at least three antenatal
care visits (29% vs 50%), the proportion of beneficia-
ries consuming at least 90 iron folic acid tablets during
pregnancy (11% vs 17%), the proportion of mothers
breastfeeding immediately after birth (62% vs 76%)
and the proportion of mothers starting complementary
feeding at the right time (32% vs 41%). Subsequently, the
ISSNIP was restructured in 2015 by integrating ICDS in
seven states with an at-scale mHealth intervention called
Common Application Software (ICDS-CAS) installed on
smart phones and with accompanying multilevel data
dashboards. This system is intended to be a job aid for
frontline workers, supervisors and managers, and aims to
ensure better service delivery and supervision by enabling
real-time monitoring and data-based decision-making.
While there is a growing body of evidence on the effec-
tiveness of mHealth interventions, it almost entirely
consists of small-scale studies or pilot interventions under
well controlled settings, and often of poor research
quality. For example, a systematic review examining 17
studies set in low and middle-income countries found
that small scale mHealth interventions, particularly those
delivered using SMS, were associated with increased util-
isation of healthcare, including uptake of recommended
prenatal and postnatal care consultation, skilled birth
attendance and vaccination, but only two of these studies
were graded as being at low risk of bias.13 Barnett and
Gallegos14 reviewed nine studies that assessed the impact
of using of mobile phones for health and nutrition
surveillance, and found that while the available evidence
suggests that mobile phones may play an important role
in nutrition surveillance by reducing the time required to
collect data and by enhancing data quality, the available
evidence is of poor methodological quality and is gener-
ally based on small pilot studies and mainly focuses on
feasibility issues. Another recent systematic review of 25
studies found evidence that mobile tools helped commu-
nity health workers improve the quality of care provided,
the efficiency of services and the capacity for programme
monitoring.15 However, most of these studies were pilots
and provided little or no information about the effec-
tiveness of mHealth interventions when integrated with
large-scale public health programmes.
This study seeks to address this critical gap in the
evidence base in the context of the largest public health
and nutrition programme in the world, ICDS, with
1.4 million AWCs serving at the grassroots level across
India. The impact evaluation is conducted in two large
states in India — Madhya Pradesh (MP) and Bihar —
using a quasi-experimental, matched-controlled pre-mea-
surement and post-measurement design. The overall
evaluation framework consists of additional components
such a process evaluation, a technology evaluation and an
economic evaluation.
This evaluation is also timely as India launched the
National Nutrition Mission on 8 March 2018 with the
goal of reducing malnutrition in a phased manner across
entire of India and subsumed ISSNIP and ICDS-CAS
under it.16 Therefore, ISSNIP and ICDS-CAS are poised
to be scaled-up rapidly to reach almost the entire popu-
lation of India through 1.4 million AWCs by 2020. This
scale-up effort will be informed by robust evidence on the
effectiveness of the mHealth intervention, as well as on
how its implementation can be improved.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

3
NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
The ICDS-CAS intervention
Currently, the ICDS-CAS intervention is being imple-
mented at scale in seven states, covering over 107 000
AWCs, and through them, a population of 9.8 million
registered beneficiaries. The intervention consists of two
components as follows.17
(1) An android CAS application and smartphones for
AWWs and the female supervisor: The CAS app was devel-
oped on an open source mobile platform (CommCare).
The app digitises and automates ten of the eleven ICDS
paper registers maintained by AWWs, enables name-based
tracking of beneficiaries, prioritises home visits at crit-
ical life-stages through a home visit-scheduler, improves
record keeping and retrieval of growth and nutrition
status of children, helps track immunisation, monitors
the timeliness and quality of different services delivered
by AWWs, and includes checklists and videos as job aids.
A female supervisor typically manages a cluster of 10–20
AWCs and the CAS app is expected to help her monitor
AWWs remotely, assess quality of service delivery, and
serves as a job aid to train AWWs. The app is installed
on new smartphones that are provided to the AWWs
and supervisors. Both AWWs and supervisors are trained
on the use of the app and how the features help them
improve service delivery. Helpdesks at block and district
levels for technical support are also established.
The CAS app is especially expected to improve home
visit service delivery by AWWs through improved chan-
nels of information (easy access to past records of the
beneficiary for customised messaging, educational anima-
tion videos as a job aid, life-stage-appropriate checklists
for counselling messages) and timely nudges (automatic
creation of visit-due lists, alerts for approaching or missed
visits and timely intimation of delays to the female super-
visor). Thus, improved home visits in terms of timeli-
ness, frequency, and perhaps, a more effective message
delivery mechanism are expected to result in increased
knowledge and better recall of correct health and nutri-
tion practices by the beneficiaries and higher demand
for related government services. However, for the actual
behaviours to change and sustain, supply side constraints
must be addressed to meet the demand for services (eg,
adequate supply of supplementary food, adequate provi-
sions of Iron Folic Acid (IFA) tablets, regular immunisa-
tion camps, etc). Such improvements can be expected
only in the mid-to-long-term because they are beyond the
sphere of influence of ICDS-CAS and need more ICDS-
wide improvements.
(2) A web-enabled dashboard for real-time monitoring
by ICDS officials: Data generated at the AWC-level are
aggregated and analysed via web-enabled dashboards for
Child Development Project Officers at the project-level
(typically an administrative block with 80–100 AWCs),
District Programme Officers, the state ICDS Directorate
and the Ministry of Women and Child Development
(MWCD) at the national level. For example, the monthly
progress reports are prepared manually at the AWC-level
and then aggregated to the project-level which require
weeks to be finalised and reviewed, but the CAS app and
dashboards will automate and produce these reports
in almost real-time. The dashboard infographics are
expected to help identify bottlenecks at various levels
more efficiently, help prioritise local issues, and allow
managers to take data-driven decisions.
Figure 1 presents the ICDS-CAS information flow
from the AWC through to the MWCD. The logic model
in figure 2 summarises how ICDS-CAS is expected to
improve service delivery, and ultimately improve health
and nutritional outcomes in mothers and their children.
The listed short-term outcomes are those expected to
be achieved in the planned evaluation follow-up period
of <12 months. The longer-term outcomes related to
improved health and nutrition will be measured (except
improvements in cognitive abilities) and analysed from a
learning perspective, but these remain aspirational in the
context of this evaluation study.
Evaluation framework and research objectives
The ICDS-CAS evaluation framework consists of four
components – an impact evaluation, a process evaluation,
an economic evaluation and a technology evaluation.
This paper describes the protocol for the impact evalu-
ation in detail to guide the final analysis plan. The other
three components are summarised online in supplemen-
tary figure 1 without binding details because their objec-
tives, scope and methods can change as per the evolving
learning needs of policy makers. We intend to analyse
and publish impact evaluation and process evaluation as
standalone research articles in peer-reviewed journals.
The analysis and publication plan for the other compo-
nents is yet to be determined.
Corresponding to the short-term strategic objective of
ICDS-CAS to improve quality and quantity of AWW and
beneficiary interactions, the main research questions
that impact evaluation seeks to answer are:
1. Does ICDS-CAS improve the timeliness or frequency of
home visits by AWWs for pregnant women, infants and
their mothers?
2. Does ICDS-CAS improve the extent or level of coun-
selling by AWWs to pregnant women and mothers of
infants?
METHODS AND ANALYSIS
Study setting
The ICDS-CAS programme is being implemented in
57 districts from seven ISSNIP states in India where the
burden of under-nutrition is highest. This evaluation
is restricted to two states, MP and Bihar, which were
selected because of the possibility of selecting an ISSNIP
district as a control, willingness of the states to support
the evaluation, and the suggestions by the MWCD and the
funding agency. Online supplementary table 1 presents
key health and nutrition related indicators for villages in
MP and Bihar. Both states have a high burden of under-
five mortality (69 per 1000 live births in MP and 60 in
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

4NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
Bihar), stunting (43.6% of children aged 0–5 years in
MP, and 49.3% in Bihar) and anaemia (>55% of chil-
dren and pregnant women in both states). Antenatal and
delivery-related indicators are, in general, worse in Bihar,
whereas MP has relatively poor demographic, water-sani-
tation, education and mortality related indicators– 8.3%
of mothers in MP and 3% in Bihar had full antenatal care;
79.5% of households in MP and 98.2% in Bihar had an
improved drinking-water source; and 50.2% of children
aged 12–23 months in MP and 61.9% in Bihar were fully
immunised.
Overview of the identication strategy
The attributable effects of ICDS-CAS will be identified
using a quasi-experimental matched, controlled design
with repeated cross-sectional pre-intervention and post-in-
tervention measurements. This identification strategy
is grounded in the Neyman–Rubin potential outcomes
model where a matched cohort design can yield unbiased
estimates of the causal effects under a strong assumption
that all confounders are measured and balanced between
the intervention and comparison groups.18–20 We use a
1:1 nearest neighbour propensity score matching (PSM)
method to identify pairs of intervention and comparison
villages.21–25 We plan to identify the effects of ICDS-CAS
by comparing post-intervention outcome indicators
between the matched groups, while controlling for any
pre-intervention or baseline differences in the outcomes
averaged at the village level and adjusting for the matched
paired design.
Sample design and power calculations
The initial sample size was determined as 400 villages in
each arm with one AWW and, on average, three moth-
er-child dyads in each village to measure a relative
effect of 15% in a standardised counterfactual outcome
[Normal(0,1)], with a significance level of 0.05, power
of 80% and intra-cluster correlation (ICC) of 0.15. The
actual baseline survey sample consisted of 852 villages to
account for refusals and loss to follow-up.
After the baseline survey was conducted in June-August
2017, completely separate from our efforts to design the
evaluation, the Indian government decided to include
ICDS-CAS in the National Nutrition Mission. Conse-
quently, the original evaluation objectives were revised to
estimate the effects of ICDS-CAS separately for MP and
Bihar to draw deeper insights into the heterogeneity of
impacts. Therefore, the evaluation sample needed to be
powered to detect smaller magnitudes of effects on a few
process-related secondary outcomes. We plan to increase
the sample power by following the same panel of villages
but recruiting almost twice as many participants in each
village – up to two AWWs and up to eight mother-child
dyads. Assuming 200 pairs of villages and 1200 mothers
per arm in each state, the sample will have 80% power at a
significance level of 0.05 to detect an absolute difference
of 5%–9%-points from the counterfactual levels between
10%–50% with ICC between 0.15–0.30. With 400 AWWs
per arm in each state, the sample will have 80% power to
detect effects of 8–12%-points for AWW level outcomes
Figure 1 ICDS-CAS information ow from the Anganwadi Centre to the Ministry of Women and Child Development. Solid lines
correspond to interactions. Dotted lines correspond to data ow. AWW, Anganwadi worker; CDPO, Child Development Project
Ofcer; ICDS-CAS, Integrated Child Development Services-Common Application Software; MWCD, Ministry of Women and
Child Development.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

5
NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
from the counterfactual levels of 10%–40% with ICC
between 0.25–0.45.
Sampling and recruitment of study participants
Figure 3 summarises the sampling and recruitment
of study participants for the baseline survey. First, we
sampled intervention districts and selected geographically
and administratively matched comparison districts. Then,
we randomly sampled intervention villages in two steps,
first sampling the blocks and then the villages. Finally, we
pair-matched intervention villages with villages from the
comparison districts and selected the best matched 426
pairs as discussed next.
Sampling of Intervention and Comparison Districts: We
randomly sampled three pairs of geographically and
administratively matched intervention and compar-
ison districts which shared a boundary and belonged
to the same ICDS division to control for division-level
confounders related to ICDS administration and manage-
ment as well as cultural, social, environmental and
economic factors at the micro-region scale. Within each
state, we first sample three intervention districts. The
corresponding control districts were selected by default
if only one eligible district was available within the divi-
sion, or randomly, if multiple comparison districts were
available. In MP, the sample purposively included one
pair of tribal-only districts from an equity-focused evalua-
tion perspective. Figure 4 depicts the states and districts
included in the evaluation.
Selecting Matched Pairs of Treatment and Control Villages:
From each selected district, we randomly sampled two
blocks in MP and three blocks in Bihar. Next, we randomly
sampled 345 villages from six intervention blocks in MP
and 315 villages from nine ICDS-CAS blocks in Bihar. In
both states, the sample frame was restricted to villages
with a population of 500 or more to increase the possi-
bility that the villages are sufficiently large to be served
by dedicated AWCs. Next, we matched the sampled treat-
ment villages with villages from the paired comparison
district using a 1:1 nearest neighbour PSM method using
the following variables from Census 2011 for matching:
Figure 2 Logic model of ICDS-CAS and measurement of outcomes. AWW, Anganwadi worker; CDPO, Child Development
Project Ofcer; DPO, District Programme Ofcer; ICDS-CAS, IntegratedChild Development Services -Common Application
Software; LS, lady supervisor; MWCD, Ministry of Women andChild Development.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

6NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
Figure 3 Sampling of study participants from Madhya Pradesh and Bihar. AWC, Anganwadi Centre; AWW, Anganwadi worker;
CAS, Common Application Software; ICDS, Integrated Child Development Services; ISSNIP, ICDS Systems Strengthening and
Nutrition Improvement Programme. Sources: # Women and Child Development Department, Government of Madhya Pradesh
(MIS) http://mpwcdmis.gov.in/ (See1m5bxnuzmixun00bsctvfj))/DataEntryAwc.aspx (Accessed 8 June 2018); & Integrated Child
Development Services, Government of Bihar http://www.icdsbih.gov.in/AnganwadiCenters.aspx?GL=16 (Accessed 8 June
2018); *Programme documentation from implementing agencies.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

7
NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
distance between village and block headquarters (kilo-
metres); population; % of schedule caste or schedule
tribe households; if a village is served by public transport;
if a village is connected to a major road; if a village has
a public ration shop; if a village has a post office; if a
village has a bank; % of households in a village with a
bank account; if a village has an agricultural society; if a
villages has a self-help group; % of households in a village
serviced by closed drainage system; % of households in
a village with improved source of drinking water; % of
households in a village with improved sanitation facility;
% of households in a village using electricity as the main
source of light; % of households in a village with a pucca
house; household asset index for the village).20–23
Figure 4 Sampled intervention and comparison districts. Green areas indicate intervention districts. Blue areas indicate
comparison districts.
Table 1 Comparison of matching performance in reducing bias
Madhya Pradesh Bihar
Ujjain-
Dewas
Barwani-
Alirajpur
Katni-
Jabalpur
Samastipur-
Darbhanga
Lakhisarai-
Muzaffarpur
Sitamarhi-
Jamui
Standardised mean bias - before matching 16.6 27.5 40.3 27 35.4 40.8
Standardised mean bias - after matching 6.5 8 9.9 8.3 6.6 9.7
% reduction in mean bias 61% 71% 75% 69% 81% 76%
P-value of LR test - before matching 0.000 0.000 0.000 0.000 0.000 0.000
P-value of LR test - after matching 0.929 0.929 0.905 0.788 0.997 0.490
Mean difference in propensity score 0.002 0.072 0.131 0.004 0.046 0.181
LR, log reduction.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

8NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
Table 1 summarises the matching performance in terms
of reduction in the sum of standardised mean bias after
matching. The standardised mean bias was reduced by
61%–81% after matching across the six district pairs. The
Log Reduction test statistic before matching was statisti-
cally significant (p<0.001), but became insignificant after
matching (p>0.75), suggesting that both groups are statis-
tically similar on average after the matching procedure.
We then selected the best matched village pairs– in terms
of the difference in the propensity scores of the matched
pairs –35 pairs of villages for each intervention block in
MP and 24 pairs of villages for each intervention block
in Bihar. Overall, we selected 216 matched pairs (432
villages) in Bihar and 210 matched pairs (420 villages)
in MP.
Recruitment of Study Participants: In the already
conducted baseline survey, we randomly sampled one
AWC if more than one AWC existed in a selected village.
Using the list of beneficiaries available at the selected
AWC as a sample frame, we randomly sampled four moth-
er-child dyads after stratification by life-stages: (1) mother
of child aged <3 months, (2) mother of child aged three
to <6 months, (3) mother of child aged six to <12 months
and (4) mother of child aged 12 to <24 months. Addition-
ally, we randomly sampled up to five pregnant women or
mothers with a child <3 months for more in-depth anal-
ysis of ICDS services during pregnancy and child birth. To
measure AWC-level outcomes, we conduct a survey of the
selected AWWs as well.
The endline will be a repeated cross-sectional sample
where pregnant women and mothers with children <12
months will be recruited using the list of beneficiaries
available at the AWC at that time. We do not plan to recruit
children older than 12 months at the endline because the
ICDS-CAS intervention would be active for <12 months
and we can realistically measure the changes only among
children who were born after the ICDS-CAS implementa-
tion started.
All survey participants are/will be recruited for the
study after being administered informed consent as per
the Institutional Review Board approved protocol. Addi-
tional assent is taken just before taking anthropometric
measurements (height and weight) of the children.
Patient and public involvement
This research was done without patient or public involve-
ment in conceptualisation, design, implementation, anal-
ysis, manuscript development or dissemination.
Primary outcomes
The primary outcomes to assess the effectiveness of
ICDS-CAS compared with the standard ISSNIP and ICDS
are:
1. The proportion of pregnant women and mothers of
children <12 months who received adequate number
of home visits by AWWs in the past 3 months (ade-
quate number will be the minimum number of visits
a respondent must receive as per ICDS guidelines for
the current life-stage/age. Additionally, we will use, as
a supporting indicator, the number of visits as a contin-
uous outcome indicator.26); and
2. The proportion of pregnant women and mothers of
children <12 months who received appropriate extent
or level of counselling from AWWs during their inter-
actions (at home, at AWCs or in other settings) in the
past 3 months (appropriate level of counselling will be
a recall of at least half of the correct messages/coun-
selling that a respondent should receive as per ICDS
guidelines for the current life-stage/age. Additionally,
we will use, as a supporting indicator, number of cor-
rect messages or services recalled by the respondent as
a continuous outcome.26).
Secondary outcomes
Several outputs and outcomes according to the logic
model presented in figure 2 are secondary outcomes in
this evaluation study as discussed before. These include
outcomes related to supervisory and capacity building
support to AWWs, infrastructure and supplies at AWC,
AWW level outcomes (motivation, satisfaction, knowl-
edge, time allocation for services and record keeping,
time allocation for service delivery, number of benefi-
ciaries served) and additional ICDS services that can be
improved by ICDS-CAS but also critically dependent on
other external factors (growth monitoring of children,
provision of IFA and supplemental nutrition, immunisa-
tion tracking, referrals, etc). We will also measure higher
order but distal or aspirational outcomes related to
knowledge, practices, health and nutrition at the bene-
ficiary level.
Outcome measurements
All beneficiary-level and AWW-level outcomes will be
measured through structured interviews, with verifica-
tion of registers and documents wherever possible. The
household instruments were developed using standard
questions from the WHO and Demographic and Health
Survey frameworks, and capture information about the
demographic and socio-economic characteristics of the
household, birth history and family planning, pregnancy
care, awareness and utilisation of AWC services at different
life-stages, IYCF practices, immunisation, knowledge of
health and nutrition and child health and nutritional
outcomes (child growth). The AWW instrument captures
information about coverage of beneficiaries, service
delivery, supervisory support, work incentives and moti-
vation, training received, time allocation, knowledge, and
infrastructure and supplies at the AWC. The interviews
are administered in Hindi on an android tablet-based
SurveyCTOTM platform.
Analysis plan
The effects of the ICDS-CAS intervention will be esti-
mated as,
Yij,t=1
=β
0
+β
1
.T
j
+Pair ID
K
+
−
Y
j,t=0
+ε
,
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

9
NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
where
Yij,t=1
= an outcome of interest for beneficiary i in village
j at endline (t=1);
Tj
= indicator variable which is 1 for intervention and 0
for comparison villages;
Pair IDK
= fixed effects to account for k pairs of matched
villages;
−
Yj,t=0
= average pre-intervention or baseline (t = 0) level of
the outcome Y in the village;
ε = the error term of the model; and
β1 = the effect of the intervention on outcome Y.
We will adjust for the pre-intervention average level of
outcome in a village (
−
Yj,t=0
) to control for, at least, the
measured or unmeasured time invariant village-level and
higher-level confounders. To the extent that the endline
questionnaires or sampling are different from those in
the baseline, the construction of the outcome indica-
tors in the baseline and endline may differ slightly, but
such differences (if any) will not affect the interpretation
of impact parameters estimated using the above model
specification. Additionally, as a consistency check, we
will estimate adjusted treatment effects that control for
any observed imbalance in important baseline covari-
ates. We may additionally control for covariates that can
help increase the precision of impact estimates. We do
not plan to correct the p-values for multiple comparisons
because we only test a limited number of indicators for
primary outcomes. However, for secondary outcomes,
when multiple indicators related to a topic or theme are
compared, we will adjust the p-values for multiple compar-
isons. We also plan to estimate the effects by adjusting for
pairing only at the district level and not at the village-level
if accounting for village matching results in substantial
sample loss. All analyses will be done in STATA and repli-
cated by two different analysts.
Baseline balance
Online supplementary tables 2 and 3 present the base-
line balance at AWC, household and individual benefi-
ciary levels. The balance is assessed in terms of group
mean difference after adjusting for the pairing of
villages. As a robustness check, the group mean differ-
ences obtained by adjusting only for the district pairing
(and not the village pairs) are also presented. Overall,
practically perfect balance in terms of exogenous vari-
ables such as household or individual characteristics is
achieved, but there are a few differences in the AWC and
AWW level characteristics and service delivery. Almost all
indicators related to home visits and growth monitoring
were balanced as the magnitude of the group differences
are of little practical significance, except for the growth
monitoring related outcomes in Bihar. A few secondary
outcomes were meaningfully different as well. However,
such differences are expected when more than 100
covariates and outcome indicators are tested for balance.
We also do not see a discernible pattern where control or
intervention groups are consistently better or worse off
than the other group. Considering the preponderance of
a highly similar distribution of variables, we infer that the
matching resulted in exchangeable or balanced groups,
and any residual confounding at the community level can
be removed by controlling for the baseline outcomes.
DISCUSSION
The evaluation will provide evidence on whether and
to what extent ICDS-CAS mHealth can improve health
and nutrition service delivery beyond what is feasible
with traditional non-technology-based approaches under
ISSNIP. Additionally, the analysis of a range of lower
order outputs and outcomes can help us identify the
pathways through which ICDS-CAS has worked, or the
critical failure points.
The study faces a few limitations in identifying unbiased
estimates of the programme effects due to the nature of
the intervention and constraints on the study design. First,
confounding or selection bias cannot be theoretically
ruled out in an observational study such as ours. While the
matching procedure appears to be successful, it may not
have removed all residual and unobserved confounding.
Measurement biases including the Hawthorne effect27
are possible because the outcomes are measured through
interview recalls and observations. The external validity
of the findings can be questioned because the purposive
sampling of states, pairing of districts and the PSM-based
sampling of villages do not result in a statistically repre-
sentative sample of entire ICDS-CAS programme area.
Finally, as is the case with most large-scale programmes,
ICDS-CAS implementation may be delayed and the
planned follow-up period may not be adequate for the
impacts to materialise.
While these limitations are common in observational
studies, the study team has tried to minimise the risk to
validity of the findings by reducing the observed pre-in-
tervention imbalance using a large set of variables from
Census 2011 for matching, controlling for at least the
cluster level time-invariant confounders by using a
repeated cross sectional design, measuring the primary
outcomes at the beneficiary level (beneficiaries will
be blinded to their intervention status in the study),
measuring a large set of indicators as per the logic model
to test whether ICDS-CAS is working through hypothe-
sised pathways and delaying the endline survey as much
as possible before the intervention is implemented in the
control districts. The evaluation framework also includes
other components which can assess the intervention
using mixed-methods approaches, and can help build
confidence in the study findings.
Overall, this study will contribute to the evidence base
on whether mHealth interventions can improve commu-
nity health worker efficiency and effectiveness. This is
also a highly policy-relevant evaluation which can inform
scale-up of the intervention to potentially cover the entire
country by 2020.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from

10 NimmagaddaS, etal. BMJ Open 2019;9:e025774. doi:10.1136/bmjopen-2018-025774
Open access
ETHICS AND DISSEMINATION
The results will be published in peer-reviewed jour-
nals and presented in conferences and dissemination
meetings.
Author afliations
1NEERMAN, Center for Causal Research and Impact Evaluation, Mumbai, India
2International Food Policy Research Institute, New Delhi, India
3Bill and Melinda Gates Foundation India, New Delhi, Delhi, India
4University of California San Fransisco, San Fransisco, USA
5School of Public Health, University of California, Berkeley, Berkeley, California, USA
6International Food Policy Research Institute, Washington, District of Columbia, USA
7Institute of Labour Economics (IZA), Seattle, Washington, USA
Contributors DW and LF are the principal investigators on the grant from Bill and
Melinda Gates Foundation. ND-S, LF, PM, HP, SRP, PS, DW contributed to the study
design and lead different sub-components of the study, DD commissioned the study
and reviewed the study design, all authors were involved in protocol development
and nalising instruments. SN and SP led the sampling. Data collection was mainly
managed by SN, LG, RA, SM, AJ. SN, PHN, ND-S and SP conducted analyses and
SN, LG, SP developed the rst draft of the manuscript. All authors reviewed, revised
and approved the nal manuscript.
Funding This study is funded by Grant No. OPP1158231 from Bill and Melinda
Gates Foundation to the University of California, San Francisco and University of
California, Berkeley.
Competing interests DD is a Program Ofcer with the Measurement, Learning
& Evaluation (MLE) team at the Bill & Melinda Gates Foundation (BMGF) India
Country Ofce. BMGF has funded this study as well as the support to the scale-up
of the ICDS-CAS programme. However, as part of the MLE team, DD has had no
role in the ICDS-CAS program design or implementation and was responsible for
conceptualizing and commissioning the evaluation. She continuous to advise on
study design, analysis and communication of ndings to stakeholders.
Patient consent for publication Not required.
Ethics approval Protocols have been reviewed and approved by institutional
review boards at the University of California, Berkeley (Ref. No. 2016-08-9092),
and the India-based Suraksha Independent Ethics Committee (Protocol No.
2016-08-9092).
Provenance and peer review Not commissioned; externally peer reviewed.
Open access This is an open access article distributed in accordance with the
Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits
others to copy, redistribute, remix, transform and build upon this work for any
purpose, provided the original work is properly cited, a link to the licence is given,
and indication of whether changes were made. See: https:// creativecommons. org/
licenses/ by/ 4. 0/.
REFERENCES
1. International Institute for Population Sciences (IIPS) and ICF. National
Family Health Survey (NFHS-4), 2015-16: India. Mumbai:: IIPS. 2017
http:// rchiips. org/ NFHS/ NFHS- 4Reports/ India. pdf (accessed 7 Mar
2018).
2. Rao N, Kaul V. India's integrated child development services scheme:
challenges for scaling up. Child Care Health Dev 2018;44:31–40.
3. Gragnolati M, Bredenkamp C, Gupta MD, et al. ICDS and Persistent
Undernutrition: Strategies to Enhance the Impact. Econ Polit Wkly
2006;41:1193–201.
4. Kapil U. Integrated Child Development Services (ICDS) scheme: a
program for holistic development of children in India. Indian J Pediatr
2002;69:597–601.
5. Agarwal KN, Agarwal DK, Agarwal A, et al. Impact of the integrated
child development services (ICDS) on maternal nutrition and birth
weight in rural Varanasi. Indian Pediatr 2000;37:1321–7.
6. Tandon BN. Nutritional interventions through primary health care:
impact of the ICDS projects in India. Bull World Health Organ
1989;67:77–80.
7. Kandpal E. Beyond average treatment effects: distribution of child
nutrition outcomes and program placement in India’s ICDS. World
Dev 2011;39:1410–21.
8. India IF. Has the ICDS helped reduce stunting in India?Ideas India.
http://www. ideasforindia. in/ topics/ human- development/ has- the-
icds- helped- reduce- stunting- in- india. html (accessed 6 Apr 2018).
9. Adhikari S, Bredenkamp C. Moving Towards An Outcomes-Oriented
Approach to Nutrition Program Monitoring: The India ICDS Program,
2009.
10. Planning Commission Programme Evaluation Organisation.
Evaluation report on integrated child development services Vol.I.
2011 http:// planningcommission. nic. in/ reports/ peoreport/ peoevalu/
peo_ icds_ v1. pdf (Accessed 7 Mar 2018).
11. NITI Aayog Programme Evaluation Organisation. A quick
evaluation study of anganwadis under ICDS. 2015 http:// niti. gov. in/
writereaddata/ les/ document_ publication/ report- awc. pdf (accessed
23 Apr 2018).
12. Math Policy Res. Evaluation of the Information and Communication
Technology (ICT) Continuum of Care Services (CCS) Intervention
in Bihar. https://www. mathematica- mpr. com/ our- publications-
and- ndings/ publications/ evaluation- of- the- information- and-
communication- technology- ict- continuum- of- care- services- ccs
(accessed 23 Mar 2018).
13. Lee SH, Nurmatov UB, Nwaru BI, et al. Effectiveness of mHealth
interventions for maternal, newborn and child health in low- and
middle-income countries: Systematic review and meta-analysis. J
Glob Health 2016;6:010401.
14. Barnett I, Gallegos JV. Using mobile phones for nutrition surveillance:
a review of evidence. 2013 https:// opendocs. ids. ac. uk/ opendocs/
handle/ 123456789/ 2602 (accessed 11 Jun 2018).
15. Braun R, Catalani C, Wimbush J, et al. Community health workers
and mobile technology: a systematic review of the literature. PLoS
One 2013;8:e65772.
16. Press Information Bureau Government of India Ministry of Women
and Child Development. PM launches National Nutrition Mission, and
pan India expansion of Beti Bachao Beti Padhao, at Jhunjhunu in
Rajasthan. http:// pib. nic. in/ newsite/ PrintRelease. aspx? relid= 177166
(Accessed 23 Mar 2018).
17. CAS Project Management Unit. Rollout of ICT-RTM Core Deck, 2018.
18. Rubin DB. Causal Inference Using Potential Outcomes. J Am Stat
Assoc 2005;100:322–31.
19. Rubin DB. Estimating causal effects of treatments in randomized and
nonrandomized studies. J Educ Psychol 1974;66:688–701.
20. Arnold BF, Khush RS, Ramaswamy P, et al. Causal inference
methods to study nonrandomized, preexisting development
interventions. Proc Natl Acad Sci U S A 2010;107:22605–10.
21. Rubin D R, Rosenbaum P. Constructing a control group using
multivariate matched sampling methods that incorporate the
propensity score. Am Stat 1985;39.
22. Stuart EA. Matching methods for causal inference: a review and a
look forward. Statistical Science 2010;25:1–21.
23. Marco C, Sabine K. Some practical guidance for the implementation
of propensity score matching. J Econ Surv 2008;22:31–72.
24. Rosenbaum PR, Rubin DB. The central role of the propensity score in
observational studies for causal effects. Biometrika 1983;70:41–55.
25. Brady H, Collier D S, Sekhon J. The neyman-rubin model of causal
inference and estimation via matching methods. Oxf Handb Polit
Methodol (Epub ahead of print 1 Jan 2008).
26. Integrated Child Development Services. Home visit planner (Register
8). https://www. icds- wcd. nic. in/ RevisedMIS/ State% 20Specic%
20Formats% 20in% 20Local% 20Languages/ State% 20Specic%
20Revised% 20MIS% 20Formats/ Goa/ Reg- 8- Home% 20Visits%
20Planner% 20( English- Goa). pdf (accessed 8 Nov 2018).
27. Fernald DH, Coombs L, DeAlleaume L, et al. An assessment of the
Hawthorne Effect in practice-based research. J Am Board Fam Med
2012;25:83–6.
on 28 March 2019 by guest. Protected by copyright.http://bmjopen.bmj.com/BMJ Open: first published as 10.1136/bmjopen-2018-025774 on 27 March 2019. Downloaded from