Figure - available from: PLOS Digital Health
This content is subject to copyright.
Considering algorithm performance in regards to pre-test probability (disease prevalence) of the condition Health care workers are confronted with two major questions at primary care health facilities: 1) Does the child need to be referred? For which an algorithm must evaluate sensitivity and specificity in relation to the severity of disease. 2) Does the child require specific treatment (most often an antibiotic)? For which the disease prevalence of a bacterial illness needs to be considered when evaluating the sensitivity and specificity of such an algorithm.
Source publication
Electronic clinical decision support algorithms (CDSAs) have been developed to address high childhood mortality and inappropriate antibiotic prescription by helping clinicians adhere to guidelines. Previously identified challenges of CDSAs include their limited scope, usability, and outdated clinical content. To address these challenges we develope...
Citations
... We believe that work overload may affect health professionals' ability to perform their duties at a high quality, including consistently using the tools and tests provided, which may result in inaccurate diagnoses and inappropriate treatment. These compromises have also been reported in other settings of low-and middleincome countries [65]. Importantly, the development process is not complete with the launch of a CDSS; its use needs to be solidly embedded into working routines and local training curricula. ...
... While several studies have found that using these digital CDSAs improve adherence to IMCI, a noteworthy research gap is that many of these investigations were conducted in controlled study settings, and most lacked randomization [13][14][15][16][17][18][19][20]. ePOCT+, a digital CDSA, was developed based on insights from two previous generations of CDSAs [21,22], specifically addressing challenges by our CDSAs and others, such as limited scope and information technology difficulties [23]. The present study aimed to assess whether this CDSA associated with point-of-care tests, training, and mentorship, would improve the quality of care for sick children compared to usual care, by comparing adherence to IMCI in a pragmatic cluster randomized trial. ...
... The intervention involved equipping health facilities with ePOCT+, an electronic clinical decision support algorithm on an Android based tablet (Fig 1), along with associated point-of-care tests (C-Reactive Protein, Hemoglobin, pulse oximetry), training, and mentorship. ePOCT+ prompts the healthcare provider to answer questions about demographics, symptoms, signs, and tests [23]. Based on the answers, ePOCT+ proposes one or more diagnoses, treatments, and management plans including referral recommendation. ...
... The tool allowed some signs to be estimated (temperature, respiratory rate) or based on recent measurements (weight). Detailed description on the development process and features of ePOCT+ and the medAL-reader application can be found in separate publications [23,29]. ...
... While several studies have found that using these digital CDSAs improve adherence to IMCI, a noteworthy research gap is that many of these investigations were conducted in controlled study settings, and most lacked randomization [13][14][15][16][17][18][19][20]. ePOCT+, a digital CDSA, was developed based on insights from two previous generations of CDSAs [21,22], specifically addressing challenges by our CDSAs and others, such as limited scope and information technology difficulties [23]. The present study aimed to assess whether this CDSA associated with point-of-care tests, training, and mentorship, would improve the quality of care for sick children compared to usual care, by comparing adherence to IMCI in a pragmatic cluster randomized trial. ...
... The intervention involved equipping health facilities with ePOCT+, an electronic clinical decision support algorithm on an Android based tablet (Fig 1), along with associated point-of-care tests (C-Reactive Protein, Hemoglobin, pulse oximetry), training, and mentorship. ePOCT+ prompts the healthcare provider to answer questions about demographics, symptoms, signs, and tests [23]. Based on the answers, ePOCT+ proposes one or more diagnoses, treatments, and management plans including referral recommendation. ...
... The tool allowed some signs to be estimated (temperature, respiratory rate) or based on recent measurements (weight). Detailed description on the development process and features of ePOCT+ and the medAL-reader application can be found in separate publications [23,29]. ...
Digital clinical decision support tools have contributed to improved quality of care at primary care level health facilities. However, data from real-world randomized trials are lacking. We conducted a cluster randomized, open-label trial in Tanzania evaluating the use of a digital clinical decision support algorithm (CDSA), enhanced by point-of-care tests, training and mentorship, compared with usual care, among sick children 2 to 59 months old presenting to primary care facilities for an acute illness in Tanzania (ClinicalTrials.gov NCT05144763). The primary outcome was the mean proportion of 14 major Integrated Management of Childhood Illness (IMCI) symptoms and signs assessed by clinicians. Secondary outcomes included antibiotic prescription, counseling provided, and the appropriateness of antimalarial and antibiotic prescriptions. A total of 450 consultations were observed in 9 intervention and 9 control health facilities. The mean proportion of major symptoms and signs assessed in intervention health facilities was 46.4% (range 7.7% to 91.7%) compared to 26.3% (range 0% to 66.7%) in control health facilities, an adjusted difference of 15.1% (95% confidence interval [CI] 4.8% to 25.4%). Only weight, height, and pallor were assessed statistically more often when using the digital CDSA compared to controls. Observed antibiotic prescription was 37.3% in intervention facilities, and 76.4% in control facilities (adjusted risk ratio 0.5; 95% CI 0.4 to 0.7; p<0.001). Appropriate antibiotic prescription was 81.9% in intervention facilities and 51.4% in control facilities (adjusted risk ratio 1.5; 95% CI 1.2 to 1.8; p = 0.003). The implementation of a digital CDSA improved the mean proportion of IMCI symptoms and signs assessed in consultations with sick children, however most symptoms and signs were assessed infrequently. Nonetheless, antibiotics were prescribed less often, and more appropriately. Innovative approaches to overcome barriers related to clinicians’ motivation and work environment are needed.
... 32,37,50,52,55,63,64,72 The other studies were quasi-experimental (n¼4), 36,39,48,51 diagnostic accuracy studies (n¼2), 34,57 observational studies (n¼2), 43,58 1 cost-analysis study, 61 and 1 stakeholder analysis. 66 We also retrieved 6 reviews that were important for contextualizing the development and use of these digital tools: 2 focused on ALgorithms for the MANagement of Acute CHildhood illnesses (ALMANACH) 33,54 development, one on ePOCT, 45 2 on ICT integration in IMCI 25,35 and 1 on integrated e-diagnostic approach (IeDA) adoption in Burkina Faso. 42 The research spanned 16 LMICs, with a major focus on Africa (n¼36 studies) and Asia (n¼3). ...
... 81 On the basis of the experience gained with ALMANACH, the Swiss TPH developed ePOCT, which combines advanced algorithms with an oximeter and includes a C-reactive protein point-of-care test that helped safely reduce antibiotic prescriptions and improve confidence in management. 45 eCare eCare aims to support Médecins sans Frontières's HWs. It has shown promise in reducing antibiotic prescriptions to 25% and covering 90% of clinical situations. ...
... The CDSA, comprising the clinical algorithm (ePOCT+) and software platform (medAL-suite), described in more detail elsewhere [45,46], uses decision logic to guide healthcare providers through consultations based on demographic and clinical information they enter about an individual child. The algorithms are drafted by country-specific clinical algorithm development groups in consultation with MoH, based on national IMCI (0-2 and 2-59 month modules) and other relevant child health guidelines. ...
Effective and sustainable strategies are needed to address the burden of preventable deaths among children under-five in resource-constrained settings. The Tools for Integrated Management of Childhood Illness (TIMCI) project aims to support healthcare providers to identify and manage severe illness, whilst promoting resource stewardship, by introducing pulse oximetry and clinical decision support algorithms (CDSAs) to primary care facilities in India, Kenya, Senegal and Tanzania. Health impact is assessed through: a pragmatic parallel group, superiority cluster randomised controlled trial (RCT), with primary care facilities randomly allocated (1:1) in India to pulse oximetry or control, and (1:1:1) in Tanzania to pulse oximetry plus CDSA, pulse oximetry, or control; and through a quasi-experimental pre-post study in Kenya and Senegal. Devices are implemented with guidance and training, mentorship, and community engagement. Sociodemographic and clinical data are collected from caregivers and records of enrolled sick children aged 0-59 months at study facilities, with phone follow-up on Day 7 (and Day 28 in the RCT). The primary outcomes assessed for the RCT are severe complications (mortality and secondary hospitalisations) by Day 7 and primary hospitalisations (within 24 hours and with referral); and, for the pre-post study, referrals and antibiotic. Secondary outcomes on other aspects of health status, hypoxaemia, referral, follow-up and antimicrobial prescription are also evaluated. In all countries, embedded mixed-method studies further evaluate the effects of the intervention on care and care processes, implementation, cost and cost-effectiveness. Pilot and baseline studies started mid-2021, RCT and post-intervention mid-2022, with anticipated completion mid-2023 and first results late-2023. Study approval has been granted by all relevant institutional review boards, national and WHO ethical review committees. Findings will be shared with communities, healthcare providers, Ministries of Health and other local, national and international stakeholders to facilitate evidence-based decision-making on scale-up. Study registration: NCT04910750 and NCT05065320.
... We developed ePOCT+, a new CDSA with point-of-care tests, to address these challenges 28 . The scope of ePOCT+ was expanded from previous versions of the CDSA 20,29 to include infants under 2 months and children up to age 14 years, and to address syndromes and diagnoses not considered by other CDSAs 30 . ...
... First and foremost, the clinical algorithm of ePOCT+ differs from other CDSAs. It notably has a wider scope including additional conditions and point-of-care tests such as C-reactive protein (CRP), not included in the Integrated Management of Childhood Illness (IMCI) 28 . A randomized controlled trial comparing two different CDSAs found differences in the impact of antibiotic stewardship due to the addition of CRP and other algorithm modifications, demonstrating that not all CDSAs are equal 20 . ...
... CRP point-of-care rapid tests and hemoglobin point-of-care tests were integrated as per usual laboratory procedures (that is, in health facilities where point-of-care tests are usually performed and interpreted in the laboratory by a laboratory technician, tests were performed in the laboratory; in health facilities where tests are usually done in the consultation room, they were done by the health-care provider). The development process and details of the ePOCT+ CDSA and the medAL-reader Android-based application used to deploy ePOCT+ have been described in detail previously 28 . In summary the clinical algorithm of ePOCT+ is based on previous-generation CDSAs (ALMANACH and ePOCT) 20,29 , international and national clinical guidelines, and input from national and international expert panels, and was adapted based on piloting and health-care provider feedback 28 . ...
Excessive antibiotic use and antimicrobial resistance are major global public health threats. We developed ePOCT+, a digital clinical decision support algorithm in combination with C-reactive protein test, hemoglobin test, pulse oximeter and mentorship, to guide health-care providers in managing acutely sick children under 15 years old. To evaluate the impact of ePOCT+ compared to usual care, we conducted a cluster randomized controlled trial in Tanzanian primary care facilities. Over 11 months, 23,593 consultations were included from 20 ePOCT+ health facilities and 20,713 from 20 usual care facilities. The use of ePOCT+ in intervention facilities resulted in a reduction in the coprimary outcome of antibiotic prescription compared to usual care (23.2% versus 70.1%, adjusted difference −46.4%, 95% confidence interval (CI) −57.6 to −35.2). The coprimary outcome of day 7 clinical failure was noninferior in ePOCT+ facilities compared to usual care facilities (adjusted relative risk 0.97, 95% CI 0.85 to 1.10). There was no difference in the secondary safety outcomes of death and nonreferred secondary hospitalizations by day 7. Using ePOCT+ could help address the urgent problem of antimicrobial resistance by safely reducing antibiotic prescribing. Clinicaltrials.gov Identifier: NCT05144763
... 30 The 4 CDSSs, detailed in Table 1, all aim to support health care providers to manage sick children in primary care to contribute to reducing morbidity and mortality in children aged younger than 5 years and improving the rational use of resources. [23][24][25]28,[31][32][33][34][35][36][37][38][39][40][41][42] The developing and implementing organizations also acknowledged that they aimed to leverage CDSS potential to: (1) be updated more readily than paper-based guidelines; (2) contribute to health worker development of knowledge and skills through onthe-job training; and (3) enhance the quality and accessibility of data for decision-making and feedback to health care providers. ...
Clinical decision support systems (CDSSs) can strengthen the quality of integrated management of childhood illness (IMCI) in resource-constrained settings. Several IMCI-related CDSSs have been developed and implemented in recent years. Yet, despite having a shared starting point, the IMCI-related CDSSs are markedly varied due to the need for interpretation when translating narrative guidelines into decision logic combined with considerations of context and design choices. Between October 2019 and April 2021, we conducted a comparative analysis of 4 IMCI-related CDSSs. The extent of adaptations to IMCI varied, but common themes emerged. Scope was extended to cover a broader range of conditions. Content was added or modified to enhance precision, align with new evidence, and support rational resource use. Structure was modified to increase efficiency, improve usability, and prioritize care for severely ill children. The multistakeholder development processes involved syntheses of recommendations from existing guidelines and literature; creation and validation of clinical algorithms; and iterative development, implementation, and evaluation. The common themes surrounding adaptations of IMCI guidance highlight the complexities of digitalizing evidence-based recommendations and reinforce the rationale for leveraging standards for CDSS development, such as the World Health Organization's SMART Guidelines. Implementation through multistakeholder dialogue is critical to ensure CDSSs can effectively and equitably improve quality of care for children in resource-constrained settings.
... However, it is impossible for any clinician to memorise the increasingly complex, evolving, and sometimes conflicting probabilistic clinical guidelines [2], which has driven the need for Clinical Decision Support Systems (CDSS) that summarise guidance into simple rulebased decision trees [3][4][5]. The digitalization of some commonly used CDSS into mobile apps has shown promise in increasing access and adherence to guidelines while laying the foundation for more systematic data collection [6][7][8][9][10]. ...
Clinical Decision Support Systems (CDSS) have the potential to improve and standardise care with probabilistic guidance. However, many CDSS deploy static, generic rule-based logic, resulting in inequitably distributed accuracy and inconsistent performance in evolving clinical environments. Data-driven models could resolve this issue by updating predictions according to the data collected. However, the size of data required necessitates collaborative learning from analogous CDSS’s, which are often imperfectly interoperable (IIO) or unshareable. We propose Modular Clinical Decision Support Networks (MoDN) which allow flexible, privacy-preserving learning across IIO datasets, as well as being robust to the systematic missingness common to CDSS-derived data, while providing interpretable, continuous predictive feedback to the clinician. MoDN is a novel decision tree composed of feature-specific neural network modules that can be combined in any number or combination to make any number or combination of diagnostic predictions, updatable at each step of a consultation. The model is validated on a real-world CDSS-derived dataset, comprising 3,192 paediatric outpatients in Tanzania. MoDN significantly outperforms ‘monolithic’ baseline models (which take all features at once at the end of a consultation) with a mean macro F1 score across all diagnoses of 0.749 vs 0.651 for logistic regression and 0.620 for multilayer perceptron (p < 0.001). To test collaborative learning between IIO datasets, we create subsets with various percentages of feature overlap and port a MoDN model trained on one subset to another. Even with only 60% common features, fine-tuning a MoDN model on the new dataset or just making a composite model with MoDN modules matched the ideal scenario of sharing data in a perfectly interoperable setting. MoDN integrates into consultation logic by providing interpretable continuous feedback on the predictive potential of each question in a CDSS questionnaire. The modular design allows it to compartmentalise training updates to specific features and collaboratively learn between IIO datasets without sharing any data.
... Developing countries like Rwanda, Uganda, South Africa, Ethiopia, Tanzania, and other countries in Sub-Saharan Africa have already embraced NBIC technologies by using drones to deliver on time blood products, vaccines and emergency medical equipment (Amukele, 2022;Griffith et al., 2023;McCall, 2019;Nisingizwe et al., 2022); by developing electronic algorithms for clinical decision support and digital platforms for ambulatory pediatric patients (Tan et al., 2023) and by instituting online training for the development of health professionals (Byungura et al., 2022;Bälter et al., 2022). Digital technologies have already demonstrated their effectiveness in responding to the internal challenges of large African cities, by improving the quality of public services through the digitization of "cadastres" (land registries), the real-time management of public transport; by increasing the security of sites and people through video surveillance; by promoting the provision of new services to users within the framework of e-governance, which saves time and reduces the burden on public services (Ndaguba et al., 2023;Ogbodo et al., 2022). ...
In the context of protracted conflict, severe droughts and health system constraints, children under-five in Somalia face one of the highest mortality rates in the world. The WHO Integrated Management of Childhood Illness (IMCI) guidance targets the main causes of morbidity and mortality, but adherence is low. We implemented the ALgorithm for the MANAgement of CHildhood illness (ALMANACH), a digital clinical decision support system, with the aim of improving IMCI adherence whilst promoting antibiotic stewardship in South-Central Somalia. Alongside, we evaluated health service delivery and ALMANACH acceptability and impact to inform design and roll-out. A pre-post assessment involving direct observation of consultations with sick children (2–59 months) based on the Demographic and Health Surveys Service Provision Assessment, complemented by exit interviews with caregivers and feedback from healthcare staff and stakeholders. Over 600 consultations were observed in each assessment period, in seven health facilities. ALMANACH had a significant impact on antibiotic prescription (reduction from 58.1% pre- to 16.0% post-implementation). This was particularly pronounced among certain conditions such as upper respiratory tract infections (30-fold reduction, RR = 0.03). Large differences in guideline adherence were observed (danger signs: 1.3% pre- to 99% post-implementation; counselling on follow-up: 12% pre- to 94% post-; and Vitamin A supplementation need checked: 19.9% pre- to 96.1% post-implementation). ALMANACH was found to be acceptable to caregivers, healthcare providers and stakeholders, with reports of positive impact on perceived quality of care. Implementation of ALMANACH in primary healthcare in Somalia significantly improved quality of care and guideline adherence, supporting the use of ALMANACH and similar tools to improve healthcare in fragile and resource-constrained settings.
RESUMEN
En un contexto de conflicto prolongado, sequías severas, y limitaciones en el sistema de salud, los niños menores de 5 años en Somalia sufren una de las tasas de mortalidad más altas del mundo. La estrategia Atención Integrada a las Enfermedades Prevalentes de la Infancia (AIEPI) de la OMS incluye recomendaciones alrededor de las causas principales de morbilidad y mortalidad, pero la adherencia a esta guía es pobre. Implementamos el algoritmo para la gestión de enfermedades de la infancia ALMANACH (ALgorithm for the MANAgement of CHildhood illness), un sistema digital de apoyo para las decisiones clínicas, a fin de mejorar el cumplimiento de la AIEPI durante un esfuerzo de promoción de la correcta administración de antibióticos en el centro-sur de Somalia. De manera paralela, evaluamos la prestación de servicios de salud, y la aceptabilidad e impacto de ALMANACH, para informar su diseño y lanzamiento. Evaluación antes-después de la implementación del algoritmo, derivada de la observación directa de consultas médicas para niños enfermos (de 2 a 59 meses), basada en la Evaluación de Provisión de Servicios (SPA, por sus siglas en inglés) de DHS (Demographic and Health Surveys, Encuestas Demográficas y de Salud), complementada con encuestas de salida a los cuidadores, y retroalimentación del personal de salud y partes interesadas. Se observaron más de 600 consultas en cada periodo de evaluación, en 7 instalaciones de salud. ALMANACH mostró tener un impacto significativo en la prescripción de antibióticos (con una reducción de 58.1% antes de la implementación, a 16.0% después). Esto fue particularmente pronunciado con ciertas condiciones, como las infecciones de vías respiratorias superiores (ocurriendo 30 veces menos, RR = 0.03). Se observaron grandes cambios en la adherencia a las recomendaciones (atención a signos de peligro: de 1.3% antes de la implementación, a 99% después; orientación acerca del seguimiento: de 12%, antes, a 94% después; y prueba de necesidad de vitamina A suplementaria: de 19.9%, antes, a 96.1% después). El ALMANACH le resultó aceptable a los cuidadores, al personal de salud y a las partes interesadas, con reportes de impacto positivo en la calidad percibida del cuidado. La implementación de ALMANACH en la atención primaria de salud en Somalia resultó en una calidad de cuidados y adherencia a las recomendaciones significativamente mayores, favoreciendo el uso de ALMANACH y herramientas semejantes en el mejoramiento del cuidado de la salud en entornos frágiles y de recursos limitados.
RESUMO
No contexto de conflitos prolongados, secas graves e limitações do sistema de saúde, as crianças com menos de cinco anos na Somália enfrentam uma das taxas de mortalidade mais elevadas do mundo. As orientações da OMS sobre a Gestão Integrada das Doenças da Infância (GIDI) visam as principais causas de morbilidade e mortalidade, mas a adesão é baixa. Implementámos o ALgorithm for the MANAgement of CHildhood illness (ALMANACH), um sistema digital de apoio à decisão clínica, com o objetivo de melhorar a adesão à IMCI, promovendo simultaneamente a gestão de antibióticos no centro-sul da Somália. Paralelamente, avaliámos a prestação de serviços de saúde, e a aceitabilidade e o impacto do ALMANACH para informar a sua conceção e implementação. Uma pré/pós-avaliação que envolveu a observação direta de consultas com crianças doentes (2–59 meses) com base na Avaliação da Prestação de Serviços do DHS, complementada por entrevistas à saída com os prestadores de cuidados e feedback dos profissionais de saúde e das partes interessadas. Foram observadas mais de 600 consultas em cada período de avaliação, em 7 unidades de saúde. O ALMANACH teve um impacto significativo na prescrição de antibióticos (redução de 58,1% antes da implementação para 16,0% após a implementação). Este impacto foi particularmente pronunciado em determinadas doenças, como as infeções do trato respiratório superior (redução de 30 vezes, RR = 0,03). Foram observadas grandes diferenças na adesão às directrizes (sinais de perigo: 1,3% antes da implementação para 99% após a implementação; aconselhamento no seguimento: 12% antes para 94% depois; e necessidade de controlo da suplementação com vitamina A: 19,9% antes da implementação para 96,1% após a implementação. O ALMANACH foi considerado aceitável pelos cuidadores, prestadores de cuidados de saúde e partes interessadas, com relatos de um impacto positivo na perceção da qualidade dos cuidados. A implementação do ALMANACH nos cuidados de saúde primários na Somália melhorou significativamente a qualidade dos cuidados e a adesão às directrizes, apoiando a utilização do ALMANACH e de ferramentas semelhantes para melhorar os cuidados de saúde em contextos frágeis e com recursos limitados.
RÉSUMÉ
Dans le contexte d’un conflit prolongé, de graves sécheresses et de contraintes du système de santé, les enfants de moins de cinq ans en Somalie sont confrontés à l’un des taux de mortalité les plus élevés au monde. Les lignes directrices de l’OMS sur la prise en charge intégrée des maladies de l’enfant (PCIME) ciblent les principales causes de morbidité et de mortalité, mais leur observance est faible. Nous avons mis en œuvre ALgorithm for the MANAgement of CHildhood illness (ALMANACH), un système numérique d’aide à la décision clinique, dans le but d’améliorer l’observance à la PCIME tout en promouvant la gestion responsable des antibiotiques dans le centre-sud de la Somalie. Parallèlement, nous avons évalué la prestation de services de santé, ainsi que l’acceptabilité et l’impact d’ALMANACH pour éclairer la conception et le déploiement. Une évaluation pré-post impliquant l’observation directe des consultations des enfants malades (2–59 mois) basée sur l’Évaluation des prestations de services de l’EDS, complétée par des entretiens de sortie avec les soignants et les commentaires du personnel de santé et des parties prenantes. Plus de 600 consultations ont été observées au cours de chaque période d’évaluation, dans 7 formations sanitaires. ALMANACH a eu un impact significatif sur la prescription d’antibiotiques (réduction de 58,1% avant la mise en œuvre à 16,0% après la mise en œuvre). Cela était particulièrement prononcé dans certaines affections telles que les infections des voies respiratoires supérieures (réduction de 30 fois, RR = 0,03). De grandes différences dans le respect des lignes directrices ont été observées (signes de danger: 1,3% avant à 99% après la mise en œuvre; conseils sur le suivi: 12% avant à 94% après la mise en œuvre; et vérification du besoin de supplémentation en vitamine A: 19,9% avant 96,1% après la mise en œuvre). ALMANACH s’est avéré acceptable pour les soignants, les prestataires de soins de santé et les parties prenantes, avec des rapports faisant état d’un impact positif sur la qualité perçue des soins. La mise en œuvre d’ALMANACH dans les soins de santé primaires en Somalie a considérablement amélioré la qualité des soins et le respect des lignes directrices, encourageant l’utilisation d’ALMANACH et d’outils similaires pour améliorer les soins de santé dans des contextes fragiles et aux ressources limitées.
