Content uploaded by Rainer Tan
Author content
All content in this area was uploaded by Rainer Tan on Feb 14, 2024
Content may be subject to copyright.
TECHNICAL NOTE
Digitalizing Clinical Guidelines: Experiences in the
Development of Clinical Decision Support Algorithms for
Management of Childhood Illness in Resource-Constrained
Settings
Fenella Beynon,
a,b
Frédérique Guérin,
c
Riccardo Lampariello,
d,e
Torsten Schmitz,
a,b
Rainer Tan,
a,b,f,g
Natschja Ratanaprayul,
h
Tigest Tamrat,
i
Karell G. Pellé,
j
Gaud Catho,
k,l
Kristina Keitel,
a,b,m
Irene Masanja,
g
Clotilde Rambaud-Althaus
n
Key Messages
nClinical decision support systems (CDSSs)—digital
tools that support health care providers to improve
quality of care—have been developed by various
groups to support the management of childhood
illness in low- and middle-income countries.
nCDSSs can differ substantially because of the
need for interpretation when translating narrative
guidelines into decision logic.
nRelative to paper-based integrated management
of childhood illness (IMCI) guidance, 4 CDSS
developers all made adaptations to scope, con-
tent, and structure to cover a broader range of
conditions, enhance precision, support rationale
resource use, expedite care for severely ill
children, and improve usability and acceptability.
nThe extent of adaptations highlights the need for
guideline developers to provide greater precision
in their recommendations to reduce the potential
for divergence from evidence-based practice
during digitalization.
nMultistakeholder efforts are needed to build and
adhere to standards for CDSS development to
ensure transparency and accountability and to
maximize impact on health and quality-of-care
outcomes.
ABSTRACT
Clinical decision support systems (CDSSs) can strengthen the
quality of integrated management of childhood illness (IMCI) in
resource-constrained settings. Several IMCI-related CDSSs have
been developed and implemented in recent years. Yet, despite
having a shared starting point, the IMCI-related CDSSs are mark-
edly varied due to the need for interpretation when translating
narrative guidelines into decision logic combined with considera-
tions of context and design choices. Between October 2019 and
April 2021, we conducted a comparative analysis of 4 IMCI-related
CDSSs. The extent of adaptations to IMCI varied, but common
themes emerged. Scope was extended to cover a broader range of
conditions. Content was added or modified to enhance precision,
align with new evidence, and support rational resource use.
Structure was modified to increase efficiency, improve usability,
and prioritize care for severely ill children. The multistakeholder
development processes involved syntheses of recommendations
from existing guidelines and literature; creation and validation
of clinical algorithms; and iterative development, implementation,
and evaluation. The common themes surrounding adaptations of
IMCI guidance highlight the complexities of digitalizing evidence-
based recommendations and reinforce the rationale for leverag-
ing standards for CDSS development, such as the World Health
Organization’s SMART Guidelines. Implementation through mul-
tistakeholder dialogue is critical to ensure CDSSs can effectively
and equitably improve quality of care for children in resource-
constrained settings.
INTRODUCTION
Poor quality of care results in an estimated 5 million
excess deaths annually in low- and middle-income
countries.
1
Emphasizing the need to improve quality of
care for children to reduce mortality in children aged
younger than 5 years,
2
the World Health Organization
(WHO) Integrated Management of Childhood Illness
(IMCI) strategy includes simple, structured guidance
to support health care providers in implementing
evidence-based recommendations.
3–6
Focused on com-
mon conditions contributing to the highest burden of
a
Swiss Tropical and Public Health Institute, Basel, Switzerland.
b
University of Basel, Basel, Switzerland.
c
Geneva Science-Policy Interface, University of Geneva, Switzerland.
d
Terre des Hommes, Lausanne, Switzerland.
e
D-tree, USA.
f
Digital and Global Health Unit, Unisanté, Center for Primary Care and Public
Health, Lausanne, Switzerland.
g
Ifakara Health Institute, Dar es Salaam, Tanzania.
Global Health:Science and Practice 2023 | Volume 11 | Number 4 1
morbidity and mortality, the IMCI strategy may
reduce child mortality by 15% when fully imple-
mented.
3,6
Since its launch in the 1990s, the IMCI
strategy has been rolled out to more than 100
countries.
3,4
But adherence by health care provi-
ders, faced with the difficult task of providing care
with limited training and resources, remains low
in many countries.
7,8
Clinical decision support systems (CDSSs)—
digital tools that provide tailored guidance based
on individual patient information—have been
recommended by WHO to support health care
providers’adherence to guidelines.
9–11
However,
substantial heterogeneity in acceptability, uptake,
and impact has been found, attributable to a com-
plex interplay of differences in context, design,
and implementation.
12–14
CDSS developers, working with implementers
and users, are confronted by a multitude of design
choices, including which clinical recommenda-
tions to incorporate, how to translate narrative
guidance into decision logic, and what standards
and technology to use to implement them.
15–17
Differences in approach to and documentation of
these decisions have resulted in wide variation in
CDSS quality and transparency.
18,19
Multisectoral
experts have highlighted the particular impor-
tance of transparency of the clinical algorithms
used in CDSSs to ensure stakeholders are able to
understand if (and how) the decision logic reflects
or differs from evidence-based recommendations
in guidelines and peer-reviewed literature.
20
In recognition of these challenges, WHO
recently launched the SMART (Standards-based,
Machine-readable, Adaptive, Requirements-based,
and Testable) Guidelines initiative. The SMART
Guidelines provide a standardized approach to the
digitalization of WHO recommendations, with the
aim of facilitating more rapid and effective uptake
of evidence-based practice.
19
Elaborating on earlier
work by Boxwala,
16
the WHO SMART Guideline
framework is organized in different “knowledge
layers”(Box 1).
19,21
Since 2021, digital adaptation
kits (Layer 2) have been published for antenatal
care, family planning, and HIV, with forthcoming
kits planned for immunization and child health in
humanitarian settings.
22
Before this move toward standardization by
WHO, several CDSSs aimed at supporting health
care providers in the management of childhood ill-
ness in primary care in low- and middle-income
countries had emerged, with demonstrated improve-
ments in quality of care, health outcomes, and anti-
microbial stewardship.
23–29
Having been developed
by various groups and for a wide range of contexts,
the resulting clinical algorithms (Layer 2) differ de-
spite a shared starting point.
In this technical note, we share lessons learned
by the eIMCI working group—a collaboration of
IMCI-related CDSS practitioners, scientists, and
policy actors—and highlight considerations when
developing clinical algorithms (i.e., from Layer
1 to Layer 2). Though not representative of all
IMCI-related CDSSs, the group—formed through
Geneva Health Forum CDSS sessions—includes
representatives from organizations involved in
the development of clinical algorithms for 4 major
child health CDSSs implemented in collaboration
with ministries of health, nongovernmental orga-
nizations, academic institutes, and other stake-
holders in 15 countries.
Through a series of 10 formal working group
meetings, informal discussions, and written ex-
changes via semistructured questionnaires and
work on shared documents between October
2019 and April 2021, we compared the clinical algo-
rithms of these 4 CDSSs with each other and with
paper-based IMCI guidance. The comparison was
structured around the following axes, defined itera-
tively over the course of group discussions: (1) context
and objectives of the interventions CDSS support, (2)
adaptations relative to the WHO IMCI chart booklet,
5
and (3) methods and processes of CDSS development.
CDSS OBJECTIVES AND CONTEXTS
OF DEVELOPMENT AND
IMPLEMENTATION
The CDSSs we compared are all knowledge-based
systems (i.e., rule-based, rather than “non-
knowledge-based,”which can extract rules using
machine learning), given our focus on translation
from clinical recommendations to decision logic.
They all provide step-by-step guidance for consul-
tations for sick children in facility-based primary
care in resource-constrained settings to enable
h
Department of Digital Health and Innovations, World Health
Organization, Geneva, Switzerland.
i
UNDP/UNFPA/UNICEF/World Bank Special Program of Research,
Development and Research Training in Human Reproduction (HRP),
Department of Sexual and Reproductive Health and Research, World
Health Organization, Geneva, Switzerland.
j
FIND, Geneva, Switzerland.
k
Division of Infectious Diseases, Geneva University Hospital and Faculty
of Medicine, University of Geneva, Geneva, Switzerland.
l
Global Health Institute, University of Geneva, Geneva, Switzerland.
m
Department of Pediatric Emergency Medicine, Department of
Pediatrics, Inselspital, University Hospital Bern, University of Bern, Bern,
Switzerland.
n
Médecins Sans Frontières Switzerland, Operational Center Geneva,
Geneva, Switzerland.
Correspondence to Fenella Beynon (fenella.beynon@swisstph.ch).
CDSSs provide
tailored guidance
to support health
care providers’
adherence to
guidelines;
however,
substantial
heterogeneity in
acceptability,
uptake, and
impact has been
found.
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 2
comparability (rather than those developed for
only a portion of the consultation such as medica-
tion dosing, for well children, or for use at the
community level based on integrated community
case management).
30
The 4 CDSSs, detailed in Table 1, all aim to sup-
port health care providers to manage sick children
in primary care to contribute to reducing mor-
bidity and mortality in children aged younger
than 5 years and improving the rational use of
resources.
23–25,28,31–42
The developing and
implementing organizations also acknowledged
that they aimed to leverage CDSS potential to:
(1) be updated more readily than paper-based
guidelines; (2) contribute to health worker de-
velopment of knowledge and skills through on-
the-job training; and (3) enhance the quality
and accessibility of data for decision-making
andfeedbacktohealthcareproviders.
Although these tools were developed with
shared aims, they were developed and adapted
for different contexts. The organizations that initi-
ated their development have taken different inter-
vention approaches and reached different scales in
field implementation (Table 1).
IDENTIFIED ADAPTATIONS FROM THE
IMCI CHART BOOKLET
Differences between the clinical algorithms are to
be expected, given their differing implementation
contexts, in line with the IMCI strategy of adapt-
ing generic global guidelines to local epidemiology
and available resources. Yet, above and beyond
contextual differences, 3 major categories of adap-
tation—scope, content, and structure—were iden-
tified as common to all the CDSSs when compared
to paper-based IMCI guidance. The extent of these
adaptations differed according to the individual
CDSS.
Extended Scope
The WHO IMCI chart booklet proposes a simple syn-
dromic approach.
5
For children aged 2–59 months,
this is based on the clinical assessment of a few basic
danger signs: assessment of 4 main symptom groups;
(1) cough/difficulty breathing, (2) diarrhea, (3) fever
(predominantly for malaria and measles), (4) ear
problems; and screening for malnutrition, anemia,
and HIV. This narrow scope reflected a desire to strike
a balance between ensuring low-skilled health care
providers were equipped with the guidance to appro-
priately identify and manage the main causes of mor-
bidity and mortality in primary care while not feeling
overburdened with overly complex guidelines.
Relative to IMCI, the scope of most of the clin-
ical algorithms compared was extended to cover a
broader range of clinical conditions, such as der-
matological or throat problems (Table 2). One
tool extended to include over 50 additional diag-
noses and a wider age range coverage.
The rationale for extending the scope was to sup-
port health care providers to implement evidence-
based practice for a wider range of problems, thus
enhancing quality of care for children presenting
with non-IMCI conditions. Additionally, some orga-
nizations noted that broadening scope encouraged
adoption by health care providers who found the
tools more relevant to their practice.
Content
All tools also included additional or modified con-
tent, such as new diagnostic tests or changes to di-
agnostic criteria, to enhance sensitivity and/or
specificity of the algorithm or to align with new
evidence (Table 2).
This was particularly notable for children pre-
senting with fever. The IMCI chart booklet focuses
fever assessment on malaria and measles. Beyond
these 2 conditions, it only advises health care pro-
viders to “Look for any bacterial cause of fever
BOX 1. The 5 Knowledge Layers of the World Health Organization SMART Guideline Framework
19,21
1. Narrative: guideline and data recommendations
2. Operational: digital adaptation kits, comprised of semistructured documentation of operational and functional
requirements
3. Machine readable: structured, software-neutral specifications, code, terminology, and interoperability
standards
4. Executable: reference software, able to execute static algorithms and interoperable digital components and de-
liver operational and functional requirements
5. Dynamic: executable dynamic algorithms that are trained and optimized with advanced analytics to achieve pri-
oritized outcomes
All tools included
additional or
modified content
to enhance
sensitivity and/or
specificity of the
algorithm or to
alignwithnew
evidence.
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 3
TABLE 1. Clinical Decision Support Systems for Pediatric Primary Care in Resource-Constrained Settings Selected for Comparative
Assessment
CDSS and Software;
Platform Developing Agency and
Collaborators Intervention Type Countries, Dates Implemented, and
Implementation Scale
IeDA;
Commcare/Dimagi Terre des hommes
MOHs, LSHTM, Centre Muraz,
UNIGE, FIND, WHO, Tableau,
Cloudera, ITU, IPE Global
QI/HSS/Research Since 2010, >20 million consultations;
Burkina Faso, 2010, 1844 HF
Mali, 2017, 50 HF
Niger, 2019, 2 HF
India, 2020, 296 HF
Guinea, 2022, 15 HF
ALMANACH;
Commcare/Dimagi SwissTPH and IHI
MOHs, ICRC, Adamawa
SPHCDA, Somali Red Crescent
Society
QI in humanitarian settings/
HSS;Tanzania: Research
pilot
Since 2015, >535,000 consultations;
Tanzania, 2011–2012, 6 HF (research)
Afghanistan, 2015–2017, 3 HF
Nigeria, 2016–ongoing, 412 HF
Somalia, 2020–ongoing, 23–28 HF
Libya, 2022–ongoing, 6–28 HF
MSFeCARE-Ped;
Logiak/Things Prime MSF Switzerland
MOHs QI in humanitarian settings MSF projects 2017–ongoing in 16 projects,
39 HF, >410,000 consultations:
Central African Republic
Democratic Republic of Congo
Kenya
Mali
Mozambique
Niger
South Sudan
(historically Chad, Nigeria, and
Tanzania)
ePOCT;
Mangologic SwissTPH
IHI, MOH Research/Pilot Tanzania 2014–2016; 9 HF; 1,586
consultations
ePOCTþ;
medAL-creator and
medAL-reader/
Wavemind
SwissTPH and Unisanté
MOHs, PATH, IHI, KGMU,
UCAD, UON, IHI, NIMR-
MMRC, MOHs, EPFL, FIND
QI/HSS/Research 2019–ongoing; >310,000 consultations in
large-scale research studies:
TIMCI project in
India, 9 HF pilot
Kenya, 60 HF
Senegal, 60 HF
Tanzania, 24 HF
Dynamic project in
Tanzania, 60 HF
Rwanda, 39 HF
Abbreviations: CDSS, clinical decision support system; EPFL, École polytechnique fédérale de Lausanne; ePOCT, electronic point-of-care test; FIND, the global
alliance for diagnostics; HF, health facilities; HSS, health systems strengthening; ICRC, International Committee of the Red Cross; IHI, Ifakara Health Institute; ITU,
International Telecommunication Union; KGMU, King George’s Medical University, Lucknow; LSHTM, London School of Hygiene and Tropical Medicine; MOH,
ministry of health; MSF, Médecins Sans Frontières; NIMR-MMRC, National Institute of Medical Research Mbeya Medical Research Centre; QI, quality (of care)
improvement; RCT, randomized controlled trial; SDC, Swiss Agency for Development and Cooperation; SPHCDA, State Primary Health Care Development
Agency; SwissTPH, Swiss Tropical and Public Health Institute; TIMCI, Tools for Integrated Management of Childhood Illness; UCAD, Université Cheikh Anta Diop
de Dakar; UON, University of Nairobi; UNIGE, University of Geneva; WHO, World Health Organization.
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 4
TABLE 2. Comparison of Scope of Clinical Decision Support Systems to IMCI Guidelines
IMCI IeDA ALMANACH MSF-eCARE ePOCT1
Age 0–2 months X X Planned Planned X
2–59 months X X X X X
5–14 years Some countries
Main symptoms Danger signs X X X X X
Cough/difficulty breathing X X X X X
Diarrhea X X X X X
Fever (malaria, measles) X X X X X
Ear problem X X X X X
Skin problem HIV-related HIV-related X X X
Fever (non-malaria) Mentioned Mentioned X X X
Sore throat X X Some countries
Mouth problem X Some countries
Eye problem X Some countries
Abdominal pain X Some countries
Trauma Some countries
Systematic screening and
management Anemia X X X X X
Malnutrition X X X X X
HIV X X X X Some countries
TB Pilot X
Prevention Vitamin A X X X X
Deworming X X X X
Vaccination X X X X X
Feeding X X X X
Point-of-care and lab tests Malaria X X X X X
HIV X X X Some countries
Hemoglobin X Optional Some countries
Urine X X Some countries
Pulse oximetry Pilot Optional X
Glucose X Optional Some countries
Stool microscopy X Some countries
Strep-A X
Dengue Pilot Some countries
Typhoid X Some countries
C-reactive protein Some countries
Abbreviations: ALMANACH, Algorithm for the Management of Childhood Illnesses; ePOCTþ, Electronic Point-of-Care Test Plus; IeDA, Integrated eDiagnosis
Approach; IMCI, Integrated Management of Childhood Illness.
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 5
[and]...Give appropriate antibiotic treatment for an
identified bacterial cause.”
5
The footnotes list several
symptoms and signs, but no clear diagnostic or man-
agement criteria are provided. Participants noted,
from experience and literature, that these broad
recommendations often led health care providers to
overprescribe antibiotics. For this reason, most tools
provided decision support for the assessment and
management of common or serious bacterial
causes of fever, with most also including addition-
al diagnostic tests.
The common rationale for extending or modi-
fying content was to enhance quality of care and
support better use of resources, particularly anti-
microbials. The degree of modifications varied
across the CDSS according to the type of interven-
tion, stakeholder priorities, and developer and
stakeholder perceptions of the capacity to support
health care providers to appropriately implement
more complex algorithms.
Revised Structure
The IMCI chart booklet follows a linear process in
which, for each presenting symptom or syndrome,
the health worker is advised to “ask”certain ques-
tions, then “look, (listen), feel”for signs (and per-
form measurements/tests where relevant) to classify
and identify treatment before moving on to the next
presenting symptom/issue (Figure 1A). This
modular syndrome-based assessment approach
is not aligned with the usual flow of a clinical
consultation, where similar tasks are grouped
together (Figure 1B). Although the IMCI chart
booklet presents syndromes in a set order,
health care providers can flip back and forth be-
tween charts.
CDSSs tend to enforce a predefined navigation
through the clinical algorithm, enforcing manda-
tory responses to clinical prompts to ensure a sys-
tematic and complete (and thus safe) consultation
process.
23,25,29,33
This is defined by the clinical al-
gorithm and the constraints of the digital solution.
Therefore, the pathway structure varies between
different CDSSs, with the overall workflow follow-
ing a structure similar to IMCI guidance (Figure
1A), a classical primary care clinical encounter
(Figure 1B), or a specifically tailored workflow
(Figure 1C). Regardless of the exact workflow, all
organizations emphasized the importance of en-
suring that the CDSS should neither disrupt nor
delay the consultation. Relative to IMCI guidance,
the CDSSs implemented 2 types of modifications
aimed at enhancing the consultation structure
(Box 2).
COMPARISON OF PROCESSES FOR
CDSS DEVELOPMENT
The process of development and refinement of
clinical algorithms can take several years, from
the identification of needs and objectives of a
CDSS through development to implementation
and evaluation. Though various steps in the pro-
cess can be outlined in a linear fashion, there are
many feedback loops for iterations of the algo-
rithm over time (Figure 2). We focus here solely
on the clinical algorithm development process.
Although the details of the process differed be-
tween each CDSS, several common steps were
identified across organizations.
Sources of Additional Content
All organizations needed to draw on several
different sources in the process of algorithm de-
velopment (Table 3). Alongside national (and/or ge-
neric WHO) IMCI guidelines, primary care health
care providers are often expected to adhere to vari-
ous other national or international clinical guide-
lines, including disease-specific guidelines (such as
those for malaria, tuberculosis, or HIV), national for-
mularies, or standard treatment guidelines. Most of
the CDSSs integrated several different child health
guidelines within their clinical algorithms.
Due to lengthy update cycles, some guidelines
may not reflect the latest evidence (and some guide-
lines may conflict with each other). Furthermore,
many narrative clinical guidelines do not explicitly
provide the decision logic necessary to develop clin-
ical algorithms. Therefore, clinical algorithm devel-
opers also consulted peer-reviewed literature, other
guidelines, and national or international expert
opinion to ensure that algorithms reflected current
evidence and national policymaker perspectives.
The extent to which different sources were
drawn varied according to the CDSS purpose and
context (Table 3), but all organizations agreed
that early and ongoing engagement with stake-
holders is needed to determine the relevant
sources for the algorithm.
Finally, the algorithms themselves may be-
come a source (i.e., once they have been devel-
oped for 1 country, the algorithm may provide a
basis or framework for an algorithm in another
country [with adaptation to national guidelines]).
Human-Readable Format of the Clinical
Algorithm
To enable experts’review and validation of the
clinical algorithm and ensure transparency of the
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 6
FIGURE 1. Consultation Workflows: A. Representation of the WHO IMCI Chart Booklet workflow
a,b
; B. Generic
clinical consultation workflow
c
; C. Hybrid workflow combining elements of the symptom-driven workflow (A)
With natural consultation process (B)
d
Abbreviations: IMCI, Integrated Management of Childhood Illness; VSD, very severe disease; WHO, World Health Organization.
a
Integrated eDiagnosis Approach follows a similar workflow but also includes a registration/triage step in which temperature, anthro-
pometric measurements, and malaria rapid diagnostic test (if fever) are conducted, with this information being entered later in the
workflow.
b
Prevention: includes immunization, vitamin A, deworming, nutrition counseling, TB, HIV, etc.
c
Electronic Point-of-Care Test Plus follows a similar workflow.
d
Algorithm for the Management of Childhood Illnesses and MSFeCARE-Ped implemented workflows similar to this.
BOX 2. Clinical Decision Support System Modifications Designed to Enhance Consultation Structure
1. Modifications to expedite the identification and management of children with severe illness:
Reordered certain assessments to bring those most likely to be associated with severe illness first
Created predefined shortcuts (skip logic) when a severe illness is identified to avoid unnecessary tasks and ex-
pedite prereferral treatment and referral
2. Modifications to integrate components of the assessment, diagnosis, and management to improve efficiency and
user experience:
Reordered certain assessments to bring those to the beginning that may influence other diagnoses (e.g., ane-
mia, malnutrition) or to the end that rely on synthesizing information from other components of the consultation
(e.g., fever without identified source)
Grouped similar tasks according to stages of the consultation (i.e., grouping together medical history items, ex-
amination items, diagnostic tests, diagnoses, and management rather than performing each according to pre-
senting syndrome before moving to the next)
Integrated different diagnoses and treatments to ensure relevance of proposed final classifications and manage-
ment recommendations
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 7
TABLE 3. Sources or Reference Documents Used in the Adaptation and/or Development of New Content and Content-Validation
Committee
Source(s) Rationale CDSS
IMCI guidelines WHO generic and/or national IMCI guidelines to reflect
country specific adaptations.
May involve updates based on national/international
guidelines (e.g., national pneumonia guidelines may
have been updated more recently than IMCI).
All
Other national or international guide-
lines (e.g., standard treatment guide-
lines, formularies, disease-specific
guidelines for malaria, TB, HIV, etc.)
To extend scope (relative to IMCI) by incorporating infor-
mation from various guidelines relevant to the diagnosis
and management of children in primary care (that health
workers are expected to implement at a country level) or to
reflectmostup-to-daterelevantguidancenotyetincorpo-
rated into national guidelines due to long guideline update
cycles (e.g., WHO young infant guidelines).
All
Existing or newly conducted evidence
syntheses (systematic reviews/meta-
analyses)
To implement relevant guidance not yet incorporated into
national or international guidelines (due to long guide-
line update cycles), particularly if extending scope.
ALMANACH, MSFeCARE-Ped, ePOCTþ
Epidemiological data (surveillance/
research)
To target focus for additional scope/content (on condi-
tions with significant morbidity or mortality).
To incorporate flexibility within the algorithm (e.g., diag-
nostic and treatment advice differing based on location
of facilities in high/low prevalence areas for diseases
such as malaria).
MSFeCARE-Ped, ePOCTþ,IeDA
Existing clinical algorithm To use work already conducted on translating narrative
guidance into decision logic, including updates that may
have been made over time based on implementation ex-
perience or novel diagnostic/treatment approaches that
may have been evaluated elsewhere.
All
Expert opinion (national or internation-
al panel)
To advise which source documents (guidelines or other
evidence) should be used as a basis for the algorithm.
To advise/review/validate interpretation of narrative
guidelines/evidence translation to human-readable
algorithm.
All
Abbreviations: ALMANACH, Algorithm for the Management of Childhood Illnesses; ePOCTþ, Electronic Point-of-Care Test Plus; IeDA, Integrated eDiagnosis
Approach; IMCI, Integrated Management of Childhood Illness; WHO, World Health Organization.
FIGURE 2. High-Level Overview of Processes Involved in the Development and Implementation of Clinical
Decision Support Systems, With Feedback Loops Between Stages
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 8
content to relevant stakeholders, all participants
agreed on the importance of a clearly documented,
human-readable representation of the clinical
algorithm.
In the absence of universally agreed standards
for clinical algorithm representation at the time of
their development, each organization addressed
this issue slightly differently. All organizations de-
veloped diagrammatic representations, with some
demonstrating the entire algorithm and others
grouped according to either syndromes, diagnoses,
or stages of the consultation. Some organizations
used business process model and notation standards;
others developed decision tables. Despite differences
in format, the common rationale for these represen-
tations was to demonstrate the algorithms’clinical
workflow, content, and decision logic to support val-
idation by experts and/or end users.
Modifications and Updates
All organizations recognized that modifications
are required following digitalization of the clinical
algorithm during testing, piloting, and implementa-
tion. Given the complex nature of the algorithms,
interaction with the end product often uncovers
issues not apparent in the human-readable written
(or depicted) format. Issues may be identified during
verification (ensuring that the digital end product
represents the human-readable algorithm) before it
is in the hands of end users. Others are only identi-
fied from health worker feedback, observation, or
analysis of CDSS or research data during piloting or
implementation. Lastly, updates to narrative guide-
lines or new evidence may necessitate a need to up-
date the clinical algorithm.
Participants acknowledged the importance
of maintaining an up-to-date human-readable al-
gorithm to ensure transparency of substantive
updates to the digital algorithm (i.e., clinical con-
tent rather than user interface), which can be val-
idated by relevant experts.
DISCUSSION
This comparison reflects the collective experience
of several organizations in clinical algorithm de-
velopment for CDSSs targeting the diagnosis and
management of childhood illness in primary care
in resource-constrained settings. In identifying
the commonalities and differences in scope, con-
tent, structure, and development processes, we
aim to highlight important considerations in the
development of clinical algorithms for this popula-
tion and contribute to the global dialogue on
improving transparency, trust, and quality of
health worker decision support.
Rigorously and transparently developing CDSS
clinical algorithms is a complex and lengthy process
with many similarities to guideline adaptation.
43
Context was acknowledged as an important critical
driver of the content, structure, and development
process of clinical algorithms for the CDSSs included
in our analysis. Epidemiology, resources, clinical
workflow, and the programmatic context (from
controlled research settings to long-term health sys-
tems strengthening interventions) all influenced the
degree to which the clinical algorithm deviated from
IMCI guidance. Yet across all contexts, efficiency
and fit to the consultation were recognized as critical
in ensuring clinical safety and promoting uptake.
All algorithms drew on sources beyond IMCI
guidance, including other national and interna-
tional guidelines, published evidence, and expert
opinion. Context was critical in determining the
extent of additional sources used—from the avail-
ability of up-to-date guidelines to stakeholder pri-
orities—and the extent of the expectation of
health care providers to integrate many (some-
times conflicting) guidelines. Extending scope
and content provides opportunities to improve
quality of care, known to be worse for non-IMCI
problems,
44
and uptake by health care providers,
who report challenges when not supported by a
diagnosis,
39
though further understanding on the
usability and impact of more complex content is
needed. Incorporating wider evidence was found
to be limited by the dearth of literature on prog-
nostic and diagnostic predictors in pediatric pri-
mary care in low- and middle-income countries.
45
These issues highlight the critical importance of
addressing evidence gaps and of timely updates of
international guidelines and guidance on adapta-
tion for resource-constrained settings,
43
whether
guidance is in paper or digital form.
Though all organizations acknowledged the im-
portance of clearly documented human-readable
algorithms for validation and transparency, different
representation approaches were taken. Various
methods have been proposed to represent CDSS
clinical algorithms, but it is recognized that no single
representation can adequately capture the complex-
ity.
46
WHO’s SMART Guideline digital adaptation
kits—which “distill WHO guidelines and operational
resources into a standardized format that can be
more easily incorporated into digital tracking and
decision support systems”—along with the forth-
coming handbook for digitizing primary health
care, are an important step in enhancing the vali-
dity, transparency, and accessibility of CDSSs.
19,21,22
Context was
acknowledged as
an important
critical driver of
the content,
structure, and
development
process of clinical
algorithms for the
CDSSs included in
our analysis.
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 9
Their description of the process undertaken for the
development of a digital antenatal care module
reflects many issues applicable to CDSSs for the diag-
nosis and management of childhood illness.
47,48
Authors agreed that when digital adaptation kits be-
come available for child health in primary care,
stakeholders should collaborate to support adapta-
tion, implementation, and evaluation while con-
tinuing to foster innovation to support future
improvements in CDSS quality and impact.
Limitations
This article has several limitations. First, it reflects
the work of a predominantly Swiss-based working
group, and although the authors have collabor-
ated with ministries of health, nongovernmental
organizations, academic institutions, civil society
organizations, health care providers, and care-
givers, this article does not directly reflect their
views. We have since worked to address this by
working with the Geneva Digital Health Hub on
the formation of a broader CDSS Community of
Practice. Although in its infancy, this group al-
ready includes a wider membership from many
countries with whom we are collaborating to de-
velop a common working approach and objec-
tives. Further, this work does not represent an
exhaustive list of IMCI-related CDSSs; however,
from our network and literature search, we are
only aware of 3 other IMCI-related CDSSs in
South Africa, Bangladesh, and Tanzania (1 of
which is no longer in use), and no others have
been highlighted in a recent systematic review.
12
Lastly, we have not addressed other important
considerations in CDSS development and imple-
mentation. These include, among others: CDSS
evaluation—from performance to usability and
acceptability to clinical and cost-effectiveness in
controlled settings and at scale
49
; the algorithm
adaptation requirements for different levels of
care or differing epidemiology within a country;
and wider implementation considerations such as
training and mentorship, operational support, IT
systems interoperability, and regulation. These
issues all influence the content, structure, and de-
velopment process of clinical algorithms—and
their uptake and impact—but were beyond the
scope of this article.
CONCLUSION
The results of this comparison reflect the first step
by this group of practitioners, scientists, and policy
actors in embracing the Principles for Digital
Development
50
to collectively share learning and
expertise on CDSSs for IMCI in primary care.
Further evaluation of the relative effectiveness
and cost-effectiveness of different approaches is
needed to guide evidence-based practice in this com-
plex field. Building and adhering to standards for
CDSS development and implementation through
multistakeholder dialogue is critical to ensure digital
tools can effectively and equitably contribute to
improve health and quality of care for children in
resource-constrained settings.
Acknowledgments: We would like to thank all those who contributed to
the development of the ideas outlined here—from health care providers
and caregivers to ministry of health staff, researchers, nongovernmental
organizations, software developers, and funders of the development,
implementation, and evaluation of clinical decision support systems. We
would also like to acknowledge the role of Irene Masanja (Ifakara Health
Institute) as cochair of the working group until November 2019 and who
contributed to the conceptualization and comparative analysis but
passed away before submission of the article.
Author contributions: All authors: conceptualization, comparative
analysis, and writing–review and editing. FB and CRA: writing–original
draft. All authors reviewed and approved the final article.
Competing interests: None declared.
REFERENCES
1. Kruk ME, Gage AD, Joseph NT, Danaei G, García-Saisó S, Salomon
JA. Mortality due to low-quality health systems in the universal health
coverage era: a systematic analysis of amenable deaths in 137
countries. Lancet. 2018;392(10160):2203–2212. CrossRef.
Medline
2. World Health Organization (WHO). Standards for Improving the
Quality of Care for Children and Young Adolescents in Health
Facilities. WHO; 2018.Accessed June 14, 2023. https://www.
who.int/publications-detail-redirect/9789241565554
3. Costello A, Dalglish S. Towards a Grand Convergence for Child
Survival and Health: A Strategic Review of Options for the Future
Building on Lessons LearntFrom IMNCI. World HealthOrganization;
2016. Accessed June 14, 2023. https://apps.who.int/iris/handle/
10665/251855
4. Gove S; The WHO Working Group on Guidelines for Integrated
Management of the Sick Child. Integrated management of child-
hood illness by outpatient health workers: technical basis and
overview. Bull World Health Organ. 1997;75(Suppl 1):7–24.
Medline
5. World Health Organization (WHO). Integrated Management of
Childhood Illness Chart Booklet. WHO; 2014. Accessed June 14,
2023. https://cdn.who.int/media/docs/default-source/mca-
documents/child/imci-integrated-management-of-childhood-
illness/imci-in-service-training/imci-chart-booklet.pdf
6. Gera T, Shah D, Garner P, Richardson M, Sachdev HS. Integrated
management of childhood illness (IMCI) strategy for children under
five. Cochrane Database Syst Rev. 2016;(6):CD010123. CrossRef.
Medline
7. Kruk ME, Gage AD, Mbaruku GM, Leslie HH. Content of care in
15,000 sick child consultations in nine lower-income countries.
Health Serv Res. 2018;53(4):2084–2098. CrossRef.Medline
8. Krüger C, Heinzel-GutenbrunnerM, Ali M. Adherence to the inte-
grated management of childhood illness guidelines in Namibia,
Kenya, Tanzania and Uganda: evidence from the national service
provision assessment surveys. BMC Health Serv Res. 2017;17(1):
822. CrossRef.Medline
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 10
9. World Health Organization (WHO). WHO Guideline:
Recommendations onDigital Interventions for Health System
Strengthening. WHO; 2019. Accessed June 14, 2023. https://
www.who.int/publications/i/item/9789241550505
10. Bright TJ, Wong A, DhurjatiR, et al. Effect of clinical decision-support
systems: a systematic review. Ann Intern Med. 2012;157(1):29–43.
CrossRef.Medline
11. Jaspers MWM, Smeulers M, Vermeulen H, Peute LW. Effects of clini-
cal decision-support systems on practitioner performance and pa-
tient outcomes: a synthesisof high-quality systematic review findings.
J Am Med Inform Assoc. 2011;18(3):327–334. CrossRef.Medline
12. Agarwal S, Glenton C, Tamrat T, et al. Decision-support tools via
mobile devices to improve quality of care in primary healthcare set-
tings. Cochrane Database Syst Rev. 2021;7(7):CD012944. Medline
13. Odendaal WA, Anstey Watkins J, Leon N,et al. Health workers’
perceptions and experiences of using mHealth technologies to deliver
primary healthcare services: a qualitative evidence synthesis.
Cochrane Database Syst Rev. 2020;3(3):CD011942. CrossRef.
Medline
14. Keitel, K. & D’Acremont, V. Electronic clinical decision algorithms for
the integrated primary care management of febrile children in low-
resource settings: review of existing tools. Clin Microbiol Infect.
2018;24(8):845–855. CrossRef.Medline
15. Musen MA, Middleton B, Greenes RA. Clinical decision-support sys-
tems. In: Shortliffe EH, Cimino JJ, eds. Biomedical Informatics:
Computer Applications in Health Care and Biomedicine. 5th ed.
Springer; 2021:795–840. Accessed June 14, 2023. CrossRef
16. Boxwala AA, Rocha BH, Maviglia S, et al. A multi-layeredframe-
work for disseminating knowledge for computer-based decision
support. J Am Med Inform Assoc. 2011;18(Suppl 1):i132–i139.
CrossRef.Medline
17. Papadopoulos P, Soflano M, Chaudy Y, Adejo W, Connolly TM. A
systematic review of technologies and standards used in the devel-
opment of rule-based clinical decision support systems. Health
Technol (Berl). 2022;12(4):713–727. CrossRef
18. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN,
Kroeker KI. An overview of clinical decision support systems: bene-
fits, risks, and strategies for success. NPJ Digit Med. 2020;3(1):17.
CrossRef.Medline
19. Mehl G, Tunçalp Ö, Ratanaprayul N, et al. WHO SMARTguidelines:
optimising country-level use of guideline recommendations in the
digital age. Lancet Digit Health. 2021;3(4):e213–e216. CrossRef.
Medline
20. Pellé KG, Rambaud-Althaus C, D’Acremont V, et al. Electronic clini-
cal decision support algorithms incorporating point-of-care diag-
nostic tests in low-resource settings: a target product profile. BMJ
Glob Health. 2020;5(2):e002067. CrossRef.Medline
21. Tamrat T, Ratanaprayul N, Barreix M, et al. Transitioning to digital
systems: the role of World Health Organization’s digital adaptation
kits in operationalizing recommendations and interoperability stan-
dards. Glob Health Sci Pract. 2022;10(1):e2100320. CrossRef.
Medline
22. SMART guidelines. World Health Organization. Accessed June 14,
2023. https://www.who.int/teams/digital-health-and-innovation/
smart-guidelines
23. Sarrassat S, Lewis JJ, Some AS, SomdaS, Cousens S, Blanchet K. An
Integrated eDiagnosis Approach (IeDA) versus standard IMCI for
assessing and managing childhood illness in Burkina Faso: a
stepped-wedge cluster randomised trial. BMC Health Serv Res.
2021;21(1):354–354. CrossRef.Medline
24. Keitel K, Kagoro F, Samaka J, et al. A novel electronic algorithm us-
ing host biomarker point-of-care tests for the management of febrile
illnesses in Tanzanian children (e-POCT): a randomized, controlled
non-inferiority trial. PLoS Med. 2017;14(10):e1002411. CrossRef.
Medline
25. Bernasconi A, Crabbé F, Adedeji AM, et al. Results from one-year
use of an electronic Clinical Decision Support System in a post-
conflict context: an implementation research. PLoS One. 2019;
14(12):e0225634. CrossRef.Medline
26. Schmitz T, Beynon F, Musard C, et al. Effectiveness of an electronic
clinical decision support system in improving the management of
childhood illness in primary care in rural Nigeria: an observational
study. BMJ Open. 2022;12(7):e055315. CrossRef.Medline
27. Rambaud-Althaus C, Shao A, Samaka J, et al. Performance of health
workers using an electronic algorithm for the management of child-
hood illness in Tanzania: a pilot implementation study. Am J Trop
Med Hyg. 2017;96(1):249–257. CrossRef.Medline
28. Shao AF, Rambaud-Althaus C, Samaka J, et al. New algorithm for
managing childhood illness using mobile technology (ALMANACH):
a controlled non-inferiority study on clinical outcome and antibiotic
use in Tanzania. PLoS One. 2015;10(7):e0132316. CrossRef.
Medline
29. Mitchell M, Hedt-Gauthier BL, Msellemu D, Nkaka M, Lesh N. Using
electronic technology to improveclinical care –results from a before-
after cluster trial to evaluate assessment and classification of sick
children according to Integrated Management of Childhood Illness
(IMCI) protocol in Tanzania. BMC Med Inform Decis Mak. 2013;
13(1):95. CrossRef.Medline
30. World Health Organization (WHO), UNICEF. WHO/UNICEF Joint
Statement: Integrated Community Case Management (iCCM).
WHO/UNICEF; 2012. Accessed June 14, 2023. https://cdn.who.
int/media/docs/default-source/mca-documents/child/who-unicef-
joint-statement-child-services-access.pdf
31. Rambaud-Althaus C, Shao AF, Kahama-Maro J, Genton B,
d’Acremont V. Managingthe sick child in the era of declining ma-
laria transmission: development of ALMANACH, an electronic algo-
rithm for appropriate use of antimicrobials. PLoS One. 2015;10(7):
e0127674. CrossRef.Medline
32. Shao AF, Rambaud-Althaus C, Swai N,et al. Can smartphones and
tablets improve the management of childhood illness in Tanzania?
A qualitative study from a primary health care worker’s perspective.
BMC Health Serv Res. 2015;15(1):135–135. CrossRef.Medline
33. Bernasconi A, Crabbé F, Raab M, Rossi R. Can the use of digital
algorithms improve quality care? An example from Afghanistan.
PLoS One. 2018;13(11):e0207233. CrossRef.Medline
34. Bernasconi A, Crabbé F, Rossi R, et al. The ALMANACH project:
preliminary results and potentiality from Afghanistan. Int J Med
Inform. 2018;114:130–135. CrossRef.Medline
35. Blanchet K, Sanon V, Sarrassat S, Somé A.S. Realistic evaluation of
the Integrated Electronic Diagnosis Approach (IeDA) for the man-
agement of childhood illnesses at primary health facilities in Burkina
Faso. Int J Health Policy Manag.12(1):1–11. CrossRef
36. London School of Hygiene & Tropical Medicine (LSHTM); Centre
Muraz. Economic Impact Study of the Integrated eDiagnosis
Approach (IeDA) for the Management of Illness in Under-Five
Children at the PrimaryHealth Care Level in Burkina Faso. LSHTM;
2018.
37. Jané E, Foutry G, Sanou S. The use of digital tools at large scale: les-
sons from a health programme in Burkina Faso. Humanitarian
Altern. 2018;8:40–52.
38. Pandya H, Slemming W, Saloojee H. Health system factors affecting
implementation of integrated management of childhood illness
(IMCI): qualitative insights from a South African province. Health
Policy Plan. 2018;33(2):171–182. CrossRef.Medline
39. Bessat C, Zonon NA, D’Acremont V. Large-scale implementation of
electronic Integrated Management of Childhood Illness(eIMCI) at the
primary care level in Burkina Faso: a qualitative study on health
worker perception of its medical content, usability and impact on
antibiotic prescription and resistance. BMC Public Health. 2019;
19(1):449. CrossRef.Medline
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 11
40. Valori A, Juma K, Elyane F, et al. MSFeCARE-Ped, an electronic al-
gorithm to improve quality of paediatric primary care: lessons
learned from 50’000consultations in Nduta Camp, Tanzania. Poster
presented at: MSF Paediatric Days; April 5–6, 2019; Stockholm,
Sweden. CrossRef
41. Tan R, Cobuccio L, Beynon F, et al. ePOCTþand the medAL-suite:
development of an electronic clinical decision support algorithm and
digital platform for pediatric outpatients in low- and middle-income
countries. PLOS Digital Health. 2023;2(1):e0000170. CrossRef.
Medline
42. Tools for Integrated Management of Childhood Illness: improving
access to tools that detect severe illness. PATH. Accessed June 14,
2023. https://www.path.org/programs/market-dynamics/timci
43. Wang Z, Norris SL, Bero L. The advantages and limitations of guide-
line adaptation frameworks. Implement Sci. 2018;13(1):72.
CrossRef.Medline
44. Venkataramani M, Edward A, Ickx P, Younusi M, Ali Shah Alawi S,
Peters DH. Are children presenting with non-IMCI complaints at
greater risk for suboptimal screening? An analysis of outpatient visits
in Afghanistan. Int J Qual Health Care. 2017;29(5):662–668.
CrossRef.Medline
45. Chandna A, Tan R, Carter M, et al. Predictors of disease severity in
children presenting from the community with febrile illnesses: a
systematic review of prognostic studies. BMJ Glob Health. 2021;6
(1):e003451. CrossRef.Medline
46. Bilici E, Despotou G, Arvanitis TN. The use of computer-interpretable
clinical guidelines to manage care complexities of patients with mul-
timorbid conditions: a review. Digit Health. 2018;4. CrossRef.
Medline
47. Haddad SM, Souza RT, CecattiJG, et al. Building a digital tool forthe
adoption of the World Health Organization’s antenatal care recom-
mendations: methodological intersection of evidence, clinical logic,
and digital technology. J Med Internet Res. 2020;22(10):e16355.
CrossRef.Medline
48. World Health Organization (WHO). Digital Adaptation Kit for
Antenatal Care: Operational Requirements for Implementing WHO
Recommendations in Digital Systems. WHO; 2021. Accessed June
14, 2023. https://www.who.int/publications/i/item/
9789240020306
49. World Health Organization (WHO). Monitoring and Evaluating
Digital Health Interventions: A Practical Guide to Conducting
Research and Assessment. WHO; 2016. Accessed June 14,2023.
https://apps.who.int/iris/handle/10665/252183
50. Waugaman A. From principle to practice: implementing the princi-
ples for digital development. Proceedings of the Principles for Digital
Development Working Group. 2016;4.
Peer Reviewed
Received: September 20, 2022; Accepted: June 13, 2023; First published online: July 27, 2023.
Cite this article as: Beynon F, Guérin F, Lampariello R, et al. Digitalizing clinical guidelines: experiences in the development of clinical decision support
algorithms for management of childhood illness in resource-constrained settings. Glob Health Sci Pract. 2023;11(4):e2200439. https://doi.org/
10.9745/GHSP-D-22-00439
© Beynon et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly cited. To view a
copy of the license, visit https://creativecommons.org/licenses/by/4.0/. When linking to this article, please use the following permanent link: https://
doi.org/10.9745/GHSP-D-22-00439
Digitalizing Clinical Guidelines for Childhood Illness in Resource-Constrained Settings www.ghspjournal.org
Global Health:Science and Practice 2023 | Volume 11 | Number 4 12
