Implementation of quality management for clinical bacteriology in low-resource settings

Abstract

Background:
The declining trend of malaria and the recent prioritization of containment of antimicrobial resistance have created a momentum to implement clinical bacteriology in low-resource settings (LRS). Successful implementation relies on guidance by a quality management system (QMS). Over the past decade, international initiatives were launched towards implementation of QMS in HIV/AIDS, tuberculosis and malaria.

Aims:
To describe the progress towards accreditation of medical laboratories and to identify the challenges and "Best practices" for implementation of QMS in clinical bacteriology in LRS.

Sources:
Published literature, online reports and websites related to the implementation of laboratory QMS, accreditation of medical laboratories and initiatives for containment of antimicrobial resistance.

Content:
Apart from the limitations of infrastructure, equipment, consumables and staff, QMS are challenged with the complexity of clinical bacteriology and the healthcare context in LRS (small-scale laboratories, attitudes and perception of staff, absence of laboratory information systems). Likewise, most international initiatives addressing laboratory health strengthening have focused on public health and outbreak management rather than on hospital based patient care. "Best Practices" to implement quality-assured clinical bacteriology in LRS include alignment with national regulations and public health reference laboratories, participating in external quality assurance programmes, support from the hospital's management, starting with attainable projects, conducting error review and daily bench-side supervision, looking for locally adapted solutions, stimulating ownership and extending existing training programmes to clinical bacteriology.

Implications:
The implementation of QMS in clinical bacteriology in hospital settings will ultimately boost a culture of quality to all sectors of healthcare in LRS.

Full text

Review
Implementation of quality management for clinical bacteriology in
low-resource settings
B. Barb
e
1
,
*
, C.P. Yansouni
2
, D. Affolabi
3
, J. Jacobs
1
,
4
1)
Institute of Tropical Medicine, Antwerp, Belgium
2)
JD MacLean Centre for Tropical Diseases, McGill University Health Centre, Montreal, Canada
3)
Clinical Microbiology, University Hospital Hubert Koutoukou Maga, Cotonou, Benin
4)
Department of Microbiology and Immunology, KU Leuven, Leuven, Belgium
article info
Article history:
Received 17 February 2017
Received in revised form
28 April 2017
Accepted 7 May 2017
Available online xxx
Editor: Gilbert Greub
Keywords:
Clinical bacteriology
Laboratory quality management
Laboratory strengthening
Low-resource setting
Quality management system
Sub-Saharan Africa
abstract
Background: The declining trend of malaria and the recent prioritization of containment of antimicrobial
resistance have created a momentum to implement clinical bacteriology in low-resource settings. Suc-
cessful implementation relies on guidance by a quality management system (QMS). Over the past decade
international initiatives were launched towards implementation of QMS in HIV/AIDS, tuberculosis and
malaria.
Aims: To describe the progress towards accreditation of medical laboratories and to identify the chal-
lenges and best practices for implementation of QMS in clinical bacteriology in low-resource settings.
Sources: Published literature, online reports and websites related to the implementation of laboratory
QMS, accreditation of medical laboratories and initiatives for containment of antimicrobial resistance.
Content: Apart from the limitations of infrastructure, equipment, consumables and staff, QMS are
challenged with the complexity of clinical bacteriology and the healthcare context in low-resource
settings (small-scale laboratories, attitudes and perception of staff, absence of laboratory information
systems). Likewise, most international initiatives addressing laboratory health strengthening have
focused on public health and outbreak management rather than on hospital based patient care. Best
practices to implement quality-assured clinical bacteriology in low-resource settings include alignment
with national regulations and public health reference laboratories, participating in external quality
assurance programmes, support from the hospital's management, starting with attainable projects,
conducting error review and daily bench-side supervision, looking for locally adapted solutions, stim-
ulating ownership and extending existing training programmes to clinical bacteriology.
Implications: The implementation of QMS in clinical bacteriology in hospital settings will ultimately
boost a culture of quality to all sectors of healthcare in low-resource settings. B. Barb
e, Clin Microbiol
Infect 2017;▪:1
©2017 The Authors. Published by Elsevier Ltd on behalf of European Society of Clinical Microbiology and
Infectious Diseases. This is an open access article under the CC BY-NC-ND license (http://
creativecommons.org/licenses/by-nc-nd/4.0/).
Introduction
The relevance of clinical bacteriology laboratories in low-
resource settings is increasingly recognized in light of the reduc-
tion of malaria burden [1,2] and the crisis of antimicrobial
resistance (AMR) [3,4]. In contrast to HIV/AIDS, tuberculosis (TB)
and malaria, clinical bacteriology does not benefit from disease-
specific control programmes and advances towards implementing
quality systems are conspicuously few. This review describes the
current state of laboratory quality management in clinical bacte-
riology in sub-Saharan Africa and reflects on the challenges and
best practices for moving forward. The target setting is a referral
hospital in sub-Saharan Africa with a ‘moderate infrastructure’(i.e.
including a basically equipped laboratory) [5], where clinical
bacteriology (culture-based detection, identification and antibiotic
susceptibility testing of bacterial pathogens) is either available or
planned. Although this review focuses on sub-Saharan Africa, the
recommendations and best practices are applicable to the general
context of low-resource settings.
*Corresponding author. B. Barb
e, Institute of Tropical Medicine, Department of
Clinical Sciences, Nationalestraat 155, 2000 Antwerp, Belgium.
E-mail address: bbarbe@itg.be (B. Barb
e).
Contents lists available at ScienceDirect
Clinical Microbiology and Infection
journal homepage: www.clinicalmicrobiologyandinfection.com
http://dx.doi.org/10.1016/j.cmi.2017.05.007
1198-743X/©2017 The Authors. Published by Elsevier Ltd on behalf of European Society of Clinical Microbiology and Infectious Diseases. This is an open access article under
the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Clinical Microbiology and Infection xxx (2017) 1e8
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

Quality, Quality Management Systems (QMS) and
accreditation
Quality in medical laboratories can be defined as accuracy, reli-
ability and timeliness of reported results [6] and a QMS describes the
approach to meet the quality objectives [7]. QMS for medical labo-
ratories are described in international standards [8]. Among them,
International Organization for Standardization (ISO) 15189 [9] and
Clinical and Laboratory Standards Institute (CLSI) QMS01-A4 [7] are
the most widely used and are comparable. They bothcover the three
laboratory phases: (a) preexamination (indication and test selection,
sample collection, transport, reception and accessioning), (b) ex-
amination (analysis and quality control) and (c) postexamination
(interpretation, reporting, record keeping and notification). In
addition CLSI QMS01-A4 introduced 12 Quality System Essentials
(Fig. 1). Accreditation is the procedure to formal recognition that a
medical laboratory is competent to carry out specific tasks [6].Na-
tional regulations either formulate own standards for accreditation
or refer to existing QMS standards. For example, ISO 15189 is the
standard for accreditation of medical laboratories in Europe.
Progress towards accreditation of medical laboratories in
Sub-Saharan Africa
A decade ago an assessment of medical laboratories in sub-
Saharan Africa portrayed a failing system with unreliable analyses
leading to compromised patient care, unnecessary expenditures
and distrust from clinicians and health authorities. The dysfunc-
tional system was declared a ‘barrier to healthcare in Africa’[10,11].
There were urgent calls to do better.
Starting with the Maputo declaration in 2008, successive
landmark events induced global efforts to strengthen national
laboratory health systems in low-resource settings (Tab le 1 ). Ini-
tiatives from HIV/AIDS and TB control programmes extended to
strengthen the general health laboratory system [13,14,15,16].An
unprecedented increase in international funding supported these
efforts [12e14,17e19]. Accreditation according to ISO 15189 qual-
ity standards was pledged [15]. In 2009 the World Health Orga-
nization regional office for Africa (WHO AFRO) launched the
Stepwise Laboratory Quality Improvement Towards Accreditation
(SLIPTA) programme, which prepares clinical laboratories for ISO
15189 accreditation (Annual ASLM Newsletter 2016) [20].In
addition, WHO-AFRO developed the Strengthening Laboratory
Management Toward Accreditation (SLMTA) toolkit to support
implementation of SLIPTA [20] (https://www.slmta.org/toolkit/
english). In 2011 the African Society of Laboratory Medicine
(ASLM) was created to advocate for the critical role and needs of
laboratory medicine and networks throughout Africa [13].Bythe
end of 2016, SLMTA had been implemented by 1103 laboratories in
47 countries worldwide, with Kenya, Ethiopia and Uganda as top
three countries (Annual ASLM Newsletter 2016). Of those, 23 Af-
rican laborator ies currently achieved accred itation to international
standards (K. Yao, personal communication, February 22, 2017). In
addition, the total number of medical laboratories accredited to
international standards in sub-Saharan Africa has increased from
380 in 12 countries by May 2013 [21] to 485 in 18 countries by
April 2017 (T. Mekonen, personal communication, April 24, 2017)
(Fig. 2).
Despite these efforts, the 2014e2015 West African Ebola
outbreak highlighted the role of weak diagnostic infrastructure in
the affected countries. As a response, the Global Health Security
Agenda (GHSA) was launched to promote global health security as
an international priority [22].
Challenges to implement QMS in clinical bacteriology in sub-
Saharan Africa
The strengthening of the general health laboratory system has fallen
short
The intention to leverage HIV-networks and the SLMTA pro-
gramme to boost general health laboratory systems has yielded
Organization
Management, Laboratory quality manual
Customer focus
Facilities and safety
Personnel
Job qualifications, Orientation, Competence assessment,
Continuing education, Performance evaluation
Purchasing and inventory
Inventory management, Inspection and verification,
Storage and handling
Equipment
Equipment qualifications,
Calibration and maintenance program
Process management
Sample management, Process validation, Quality control
Information management
Paper-based versus computer-based, Confidentiality
Fig. 1. Twelve quality system essentials of CLSI document QMS01-A4: Quality Management System: A Model for Laboratory Services [7]. General themes displayed horizontally,
transversal themes vertically.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e82
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

limited results. Though there has been some success at integration
of HIV with TB services, other examples are few [19]. The encour-
aging data on the uptake of SLMTA belie the fact that the vast
majority of accredited laboratories are HIV and TB reference labo-
ratories [16,20] with few (4/23, 17%) accredited SLMTA laboratories
in sub-Saharan Africa performing clinical bacteriology. Moreover,
most external quality assurance (EQA) programmes centre on HIV,
TB and malaria [23e28].
Antimicrobial resistance enters the scene, but tools are lacking and
performance is poor
Although the critical need for diagnosis of bacterial in-
fections is long established [10], clinical bacteriology has only
recently achieved prominence with policymakers, prompted by
the increasing recognition of the looming crisis of AMR
(Table 2). In October 2015, the Freetown declaration launched a
framework to establish functional tiered public health labora-
tory networks to AMR surveillance in Africa [22,58]. In addition,
the United Nations General Assembly recently launched several
internationally coordinated actions on AMR. Of note, most of
these initiatives focus on public health implicationsdparticu-
larly outbreak management (in line with the International
Health Regulations [29]) rather than on individual patient care.
The predicted impact of the AMR crisis casts a new urgency on
the need to improve performance levels. In 2012 a publication of an
EQA session among reference laboratories in sub-Saharan Africa
revealed serious shortcomings in bacterial identification and anti-
microbial susceptibility [30]. Adding to the difficulty is the lack of
appropriate tools for training and implementation of QMS in clin-
ical bacteriology. For instance, the SLMTA toolkit includes only few
examples about clinical bacteriology.
Table 1
Landmarks towards accreditation of medical laboratories in sub-Saharan Africa
No. Landmark Outcome
1 Maputo declaration on Strengthening of Laboratory Systems, Maputo,
January 2008
‘Maputo declaration,’issued by 33 countries together with WHO, World
Bank and Global Fund for AIDS, TB and Malaria, calls on low-resource
countries to develop national laboratory strategic plans and policies to
strengthen laboratory services and systems as integral part of overall
health system (tiered lab networks for multiple diseases) [12].
2 Joint WHO-CDC Conference on Health Laboratory Quality Systems, Lyon,
April 2008
Statement issued calling for countries with limited resources to develop
quality laboratory systems using staged approach leading to
accreditation. It was suggested that national laboratory standards
establish minimum requirements, and national reference laboratories
were encouraged to meet international standards (ISO 15189).
3 58th session of WHO Regional Committee for Africa, Yaound
e,
September 2008
During the 58th session of the WHO Regional Committee for Africa,
member states adopted resolution AFR/RC58/R2 to strengthen public
health laboratories in WHO African region at all levels of the healthcare
system.
4 5th meeting of Regional HIV/AIDS network for Public Health
Laboratories, Dakar, September 2008
It was agreed that the network should broaden its scope beyond HIV/
AIDS and associated diseases to become an integrated network
encompassing all labs, without limitation of a disease-specific
designation.
5 WHO-AFRO Stepwise Laboratory Accreditation preparedness scheme,
Kigali, July 2009
WHO-AFRO launched Stepwise Laboratory Quality Improvement
Towards Accreditation (SLIPTA) and Strengthening Laboratory
Management Toward Accreditation (SLMTA), in presence of its partners
and government health officials from 13 African countries.
6 59th session of WHO Regional Committee for Africa, Kigali, September
2009
During the 59th session of the WHO Regional Committee for Africa,
member states adopted resolutions AFR/RC59/R2 and AFR/RC59/WP/3,
calling for strengthening of public health laboratories and other centres
of excellence to improve disease prevention and control.
7 African Society of Laboratory Medicine (ASLM), Addis Ababa, March
2011, http://www.aslm.org/
This pan-African professional body aims to set up integrated laboratory
services, to develop national laboratory policies and strategic plans for
tiered laboratory networks and to improve quality systems and
accreditation preparedness.
8 Ministerial Call for Action on Strengthening of Laboratory Services in
Africa, Cape Town, December 2012
During the first international ASLM conference in Cape Town, this
ministerial call was undersigned by six African countries. By November
2014, this call was signed by 13 African countries.
9 Global Health Security Agenda (GHSA), Washington, DC, February 2014,
https://www.ghsagenda.org/
GHSA is collaborative effort from governments, international
organizations and civil society to promote global health security as
international priority. GHSA currently includes almost 50 countries and
is coordinated by a multilateral steering group of 10 countries together
with international organizations such as WHO, FAO, OIE, Interpol,
ECOWAS, UNISDR and the European Union.
10 Freetown declaration on Developing Resilient Laboratory Networks for
GHSA, Freetown, October 2015
Freetown declaration was issued by more than 20 African countries,
together with ASLM and WHO AFRO. It calls for a new framework for
functional tiered laboratory networks into disease surveillance systems
and public health institutes.
Adapted from Alemnji [13] and Andiric and Massambu [14].
ASLM, African Society of Laboratory Medicine; CDC, Centers for Disease Control and Prevention; ECOWAS, Economic Community of West African States; FAO, Food and
Agriculture Organization of the United Nations; GHSA, Global Health Security Agenda; HIV, Human Immunodeficiency virus; MOH, Ministry of Health; OIE, World Organi-
zation for Animal Health; SLIPTA, Stepwise Laboratory Quality Improvement Towards Accreditation; SLMTA, Strengthening Laboratory Management Toward Accreditation;
TB, Tuberculosis; UNISDR, United Nations Office for Disaster Risk Reduction; WHO, World Health Organization; WHO-AFRO, World Health Organization Regional Office for
Africa.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e83
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

Clinical bacteriology and QMS: a difficult fit
Clinical bacteriology deals with multiple specimens, indications
and requirements for sampling and transport. Culture-based bacte-
riology requires skilled operators, is difficult to automate and leaves
much to the individual laboratorian's discretion. In the post-
examinationphase, there are issues of interpretation (contaminating
or colonizing flora) and reporting (e.g. preliminary results of Gram
stain, cultures in progress) and application of expert rules [31e33].
Clinical bacteriology is therefore less amenable to QMS than other
laboratory disciplines, which have single targets and generate
automated quantitative results that allow for statistical monitoring.
Particular challenges in sub-Saharan Africa
Adapting QMS to local realities proves difficult in sub-Saharan
Africa [34]. Besides the well-known limitations of infrastructure,
equipment, consumables and staff, additional issues merit
comment. Most laboratories in sub-Saharan African hospitals are
small [34] and provide haematology, clinical chemistry, parasi-
tology and sometimes blood transfusion services on a 24/7 basis.
Staff not familiar with QMS may associate quality with goodness
rather than with conformance to requirement [6]. In view of the
small scale and daily burdens, quality essentials such as assess-
ments, customer focus and process management may be perceived
as superfluous, and the concept of prospective risk management
may be too unfamiliar [34]. Further, the digitalization of laboratory
and hospital information, automation and bar code trackingd-
which have dramatically reduced errors at all examination phases
[6,35]dare not yet implemented in sub-Saharan laboratories and
the requirement to validate and secure laboratory information
software (LIS) tools is difficult to meet [34]. An additional challenge
in many low-resource settings is the lack of unique identifiers for
patients with naming practices leading to many people with the
same name or names that change over time, lack of identity cards
and exact birth dates.
MOZAMBIQUE
2
MADAGASCAR
SOUTH
AFRIC A
383
LESOTHO
SWAZI LAND
1
BOTSWANA
10
NAMIBIA
11
ANGOLA
ZAMBIA
TANZANIA
10
KENYA
31
BURUNDI
RWANDA
ZIMBABWE
5
MALAWI
ERITREA
SOMALIA
ETHIOPIA
8
DJIBOUTI
DEMOCRATIC
REPUBLIC
OF THE
CONGO
UGANDA
9
SUDAN
NIGER
CHAD
NIGERIA
3
SENEGAL
1
THE GAMBIA
1
SIERRA
LEONE
CAMEROON
1
EQUATORIAL
GUINEA GABON
CONGO
WESTERN
SAHARA
SOUTH
SUDAN
MAURITANIA
MALI
1
BENIN
TOGO
2
GHANA
1
BURKINA
FASO
LIBERIA
CÔTE
D'IVOIRE
GUINEA
MAURITIUS
5
GUINEA-BISSAU
CENTRAL AFRICAN
REPUBLIC
Fig. 2. Map of sub-Saharan Africa showing countries with medical laboratories that have been accredited to internationally recognized standards by April 2017. Numbers below
countries' names refer to number of accredited laboratories in that country.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e84
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

Implementing QMS in clinical bacteriology in low-resource
settings: best practices
Implementation of a QMS comprises all 12 Quality System Es-
sentials, but approaches and priorities may vary according to the
local situation [6]. Below, we present some best practices about
how to implement QMS in clinical bacteriology on site. They are
compiled from existing literature (though mainly conducted in
high-resource settings) complemented with our own experience.
Connect to national regulations and public health laboratories
Alignment with national regulations is imperative and will
guide towards legal requirements such as participation in EQA
programmes [36]. Much can be adopted from the expertise of the
disease-specific control programmes of malaria, HIV/AIDS and TB
and their respective reference laboratories: they provide logistic
support (equipment and quality-assured consumables and stains)
and dedicated procedures, job descriptions, forms and logbooks in
local languages [37]. In addition, they organize trainings, external
quality controls, on-site supervision visits and meetings with the
laboratory heads. They further offer advanced techniques, coordi-
nate surveillance activities and create national laboratory networks
for diffusing of knowledge and competencies [22,38].
Participate in EQA programmes
EQA programmes improve laboratory performance (provided
implementation of corrective actions in case of failures) and are
cost-effective [30,39]. They allow the participants to benchmark
and monitor their competence and to verify their methods; in
addition, they provide didactic and educational stimulus [6,40].
Beyond that, EQA programmes provide health authorities with
information about overall competence of laboratories, in vitro di-
agnostics' performance and training needs [6,28,39,41]. EQA pro-
grammes further generate excellent opportunities to integrate
vertical disease control programmes, for instance by sharing dis-
tribution canals and increasing nationwide coverage [28,39]. Short
turnaround times, swift communication (e.g. sending correct an-
swers directly after closure), well-structured didactic reports
anddin our experiencedpersonalized feedback and EQA-oriented
trainings are instrumental to a productive EQA session [39].
Observe and understand the local scene, find support and allies
Support from the healthcare facility management at all levels is
essential to success, as is a well-functioning relationship between
clinicians and laboratory staff. A strategy that has proved powerful
in antibiotic stewardship activities is the ‘engagement of peer
Table 2
Overview of main international initiatives dedicated to containment of antimicrobial resistance
No. Initiative Scope
1 Advisory Group on Integrated Surveillance of Antimicrobial Resistance
(AGISAR), 2008 (http://www.who.int/foodsafety/areas_work/
antimicrobial-resistance/agisar/en/)
WHO AGISAR consists of a multidisciplinary team of over 30
internationally renowned experts and supports WHO's effort to
minimize the public health impact of antimicrobial resistance
associated with the use of antimicrobials in food animals.
2 Global Antibiotic Resistance Partnership (GARP), 2009 (http://www.
cddep.org/garp/home)
GARP is a project of CDDEP and is funded by the Bill &Melinda Gates
Foundation. GARP has supported the creation of multisectoral national-
level working groups in eight selected low- and middle-income
countries whose mandate is to understand and document antibiotic use
and antibiotic resistance in the human and animal population and to
develop evidence-based proposals to encourage introduction of
measures to contain antimicrobial resistance.
3 Global Health Security Agenda (GHSA), 2014 (https://www.ghsagenda.
org/)
a
One of the 11 GHSA action packages focuses on AMR, and includes
development of a national action plan based on a ‘one health’approach;
development and implementation of guidelines and standards for
infection prevention; development and use of guidelines for antibiotic
use; access to at least one reference laboratory for each country capable
of identifying at least three of seven WHO priority AMR pathogens.
b
4 Global Action Plan on Antimicrobial Resistance (GAP-AMR), 2015
(http://www.who.int/antimicrobial-resistance/global-action-plan/en/)
This WHO action plan sets out five strategic objectives: (a) improve
awareness and understanding of AMR, (b) strengthen knowledge
through surveillance and research, (c) reduce incidence of infection, (d)
optimize use of antimicrobial medicines in human and animal health,
(e) ensure sustainable investment in countering AMR. The action plan
underscores the need for a ‘one health’approach, involving human and
veterinary medicine, agriculture, finance, environment and well-
informed customers.
5 Global Antimicrobial Resistance Surveillance System (GLASS), 2015
(http://www.who.int/drugresistance/surveillance/glass-enrolment/en/)
GLASS, developed by WHO, is a platform for global data sharing on AMR
worldwide, initially focusing on eight priority bacterial pathogens in
humans.
c
Currently more than 30 countries are participating.
6 United Nations General Assembly, 2016 Global leaders met at the United Nations General Assembly to commit to
fighting antimicrobial resistance together.
This was only the 4th time in the history of the UN that a health topic
was discussed at the General Assembly (HIV, noncommunicable
diseases and Ebola were others).
AGISAR, Advisory Group on Integrated Surveillance of Antimicrobial Resistance; AMR, antimicrobial resistance; CDDEP, Center for Disease Dynamics, Economics and Policy;
GAP-AMR, Global Action Plan on Antimicrobial Resistance; GARP, Global Antibiotic Resistance Partnership; GHSA, Global Health Security Agenda; GLASS, Global Antimicrobial
Resistance Surveillance System; UN, United Nations.
a
Refer to Table 1 for a general description of GHSA.
b
WHO list of AMR pathogens of concern includes [3]:Escherichia coli, resistance to third-generation cephalosporins (ESBL) and to fluoroquinolones; Klebsiella pneumoniae,
resistance to third-generation cephalosporins (ESBL) and to carbapenems; Staphylococcus aureus, methicillin resistance (MRSA); Streptococcus pneumoniae, resistance
(nonsusceptibility) to penicillin; nontyphoidal Salmonella (NTS), resistance to fluoroquinolones; Shigella species, resistance to fluoroquinolones; Neisseria gonorrhoeae,
decreased susceptibility to third-generation cephalosporins.
c
GLASS priority pathogens for surveillance are: Escherichia coli, Klebsiella pneumoniae, Acinetobacter baumannii, Staphylococcus aureus, Streptococcus pneumoniae, Salmonella
spp., Shigella spp., Neisseria gonorrhoeae.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e85
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

champion advocates’[42], which means engaging individual lab-
oratorians, clinicians and administrators to promote a culture of
quality. Clinicians can play a role at the pre- and postanalytical
processes, for example, by promoting de-escalation of antibiotics
based on culture results. Insights in the perceptions, attitudes and
interactions of the clinical, nursing and laboratory staff are there-
fore helpful to guide and prepare interventions [43,44]. In line with
what has proved to be successful in malaria treatment, joint
training for clinicians and laboratory staff are recommended [45].
Further, the overarchingQMS of the hospital can be used to connect
for support in logistics (transport, procurement, maintenance,
biosafety and waste management) and staff management (training,
orientation and evaluation).
Start softlydchoose for feasibility and impact
Before formally starting-up a QMS, time to observe and analyse
the laboratory activities must be taken, to get acquainted with the
workflow, terminology and vocabulary, product names and ab-
breviations used as well as with laboratory and hospital staff and
the healthcare's context. As noted elsewhere, personal interactions
provides a solid basis for collaboration [46].
Starting with a project that can be easily accomplished and has a
high impact is recommended [6]. Topics can be found close to the
work space and include for instance biosafety (rational use of
gloves and masks), labelling of shelves and storage space and
standardization of labelling of consumables. In addition, ‘fewer are
better,’which means that no more than one topic every six months
should be addressed [6].
Keep documents and communication simple and straightforward
Communication must be efficient and adapted to the size and
complexity of the laboratory [6]. Standard operating procedures
(SOPs) will be among the first documents to be written or updated.
A good SOP is presented in an appropriate font type and lay-out
(legible), easy to understand (readable) and conveys clear and
unequivocal information (comprehensible); it presents sufficient
procedural details [47] is pretested and verified [6,48]. Given the
extended templates of QMS-recommended SOPs (CLSI QMS02-A6
template lists 17 sections [49]), so-called bench aids or job aids are
welcome as a quick reference at the work placedprovided docu-
ment control [6,47,48]. The urge for legibility, readability and
comprehensibility also applies to other documents particularly if
used at the workplace: stock cards, check lists, flow diagrams and
graphs. In the African context, the importance and impact of
respectful verbal communication cannot be overemphasized and
occasions for face-to-face interactions with clinicians and other
stakeholders should be fully exploited [47,48].
Learn from your mistakes
Reducing medical errors ranks first among the benefits of a
successful QMS implementation and error review has been the
oldest approach of the ‘Nonconforming Event Management’[7].
Error review consists of tracking errors up to the identification of
root cause(s) and subsequent design and implementation of
corrective and preventive actions [50]. Most errors occur in the pre-
and postexamination processes (about 60% and 25% respectively)
[6,51], but given its manual and subjective workup, clinical bacte-
riology is also vulnerable to errors in the examination phase [52].
Apart from ‘expected’and easily detectable errors (e.g. failing to
comply with SOP, not acting on out-of-range quality controls), there
are hidden errors such as failures of Gram stain reading, species
identification and antimicrobial susceptibility testing [32,53].
In our experience, error review is an excellent portal of entry to
activate QMS and it can be applied from the early phase of
implementation of the QMS, whereas alternative approaches such
as monitoring of quality indicators and internal audits require a
more advanced implementation. Of note, most analytical errors in
clinical bacteriology proved to be related to knowledge and skills
and therefore can be efficiently addressed by teaching, training
and learning [52]. Educational programmes should be straight-
forward, explain the science behind the error and stress the ‘do's
and don'ts'; moreover, overlap and redundancy are appropriate
[47].
Be present at the bench and organize daily supervisory review
Supervision of culture workup is another traditional approach
towards laboratory quality and is particularly relevant for clinical
bacteriology [54]. Supervision at the workplacedin preference at a
daily basis and at a fixed timedis a valuable tool for bench-side
teaching, tuning of operator-subjective examinations (such as
Gram stain) and boosting compliance with SOPs. Daily supervision
also allows for timely detection of procedural deviations such as
unapproved modifications, ill-advised shortcuts and use of
outdated products inserts [47].
Look for locally adapted solutions
A creative mind helps to design local solutions on the road to
ISO 15189. Low-cost low-tech, feasibility and acceptability are key.
As an example, printed request forms may guide the prescriber
towards harmonized indications and relevant requests. Stan-
dardized abbreviations may be agreed upon and recorded. Risks to
sample mismatch and mislabelling of subcultures can be mitigated
by arranging the physical space and workflow at the bench and
sticking to the one-by-one rule used at specimen transfer and
distribution [55]. A particular risk is inoculating multiple isolates
on a single petri agar, which should be limited and well controlled.
Electronic sign-off of documents may be replaced by hand-signing
before or after laboratory meetings. Other tools of traceability
include writing in different colours according to the day of incu-
bation, use of standardized laboratory forms and implementing a
journal passing along information between staff covering night
shifts [6].
Stimulate ownership and create a positive climate
A QMS must be visible in the laboratory and must express
progress and achievements. A clear workplace such as described in
the SLMTA toolkit (Module 1dProductivity management) with
well-designed bench aids displayed is conducive to QMS but also
attractive for staff, trainees and visitors. Staff should be invited to
SOP and document writing and verification, but should not be
overloaded. A constructive and critical attitude including reporting
and reviewing errors should be encouraged. As stated above, lab-
oratory management must assure clear commitment and involve-
ment in the implementation of the QMS, amongst others by visible
presence at meetings and trainings, respecting short feedback
loops and keeping good and timely records [6,50].
Needs and opportunities to extend QMS and training tools to clinical
bacteriology
Existing tools to facilitate QMS (Table 3), such as the WHO AF-
RO's SLMTA toolkit and the WHO, Centers for Disease Control and
Prevention and CLSI Laboratory QMS Training Handbook and
Toolkit can be easily extended with real-life cases of clinical
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e86
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

bacteriologydfor instance in the SLMTA's ‘Meet the Clinician’ac-
tivities and the ‘Clinicians Handbook,’or its modules of specimen
collection, work area management and test result reporting.
Although hardware and network costs remain barriers, imple-
mentation of a LIS has benefits at all three examination processes;
LIS further provides opportunities to educate prescribers, guide
therapeutic decisions and detect hospital infection outbreaks in
real-time. LIS supports other Quality System Essentials such as
purchasing, inventory and assessments (e.g. analysis of corrected
reports as a proxy for near errors) [6,52]. Currently, free-of-charge
LIS are made available (Table 3).
Conclusion
Across sub-Saharan Africa and over the last decade amazing
strides have been made to implement QMS in the laboratory
diagnosis of HIV, malaria and TB. It is now time to extend this
success to clinical bacteriology, given the momentum generated by
the declining burden of malaria and the need to contain the
emergent AMR. Taking into account the particularities of both
clinical bacteriology and the context of low-resource settings,
existing national networks can be strengthened towards compe-
tences in clinical bacteriology and training tools can be adapted to
integrate clinical bacteriology. Best practices can facilitate the
implementation of QMS in clinical bacteriology in hospital settings.
Given the extensions of clinical bacteriology to antibiotic stew-
ardship and infection prevention, this step will ultimately boost a
culture of quality to all sectors of healthcare in low-resource
settings.
Acknowledgements
The authors acknowledge T. Mekonen, ASLM, for providing the
list of accredited laboratories in Africa, K. Yao for sharing the number
of laboratories in Africa that have implemented SLMTA and C. Kiyan,
Institute of Tropical Medicine, for help designing the figures.
Transparency Declaration
All authors report no conflicts of interest relevant to this article.
References
[1] D'Acremont V, Kilowoko M, Kyungu E, Philipina S, Sangu W, Kahama-Maro J,
et al. Beyond malariadcauses of fever in outpatient Tanzanian children.
N Engl J Med 2014;370:809e17.
[2] Crump JA, Morrissey AB, Nicholson WL, Massung RF, Stoddard RA,
Galloway RL, et al. Etiology of severe non-malaria febrile illness in Northern
Tanzania: a prospective cohort study. PLoS Negl Trop Dis 2013;7:e2324.
[3] World Health Organization. Antimicrobial resistance. Global report on sur-
veillance. Geneva: World Health Organization; 2014.
[4] World Health Organization. Worldwide country situation analysis: response
to antimicrobial resistance. Geneva: World Health Organization; 2015.
[5] Girosi F, Olmsted SS, Keeler E, Hay Burgess DC, Lim YW, Aledort JE, et al.
Developing and interpreting models to improve diagnostics in developing
countries. Nature 2006;444(Suppl 1):3e8.
[6] World Health Organization. Laboratory quality management system hand-
book. Geneva: World Health Organization; 2011.
[7] Clinical and Laboratory Standards Institute. Quality management system: a
model for laboratory services. Approved guideline. CLSI document QMS01-A4.
4th ed. Wayne, PA: Clinical and Laboratory Standards Institute; 2011.
[8] Datema TA, Oskam L, Klatser PR. Review and comparison of quality standards,
guidelines and regulations for laboratories: original research. Afr J Lab Med
2012;1:1e7.
Table 3
Examples of open-access tools for the implementation of laboratory quality management systems
No. Tool Scope
1 WHO/NCID External Quality Assessment Programme (EQAP) in Africa,
2002 (http://www.who.int/ihr/publications/policy_procedures_eqa/en/)
WHO/NCID EQAP tests the proficiency of microbiologic testing for
epidemic-prone diseases by laboratories in the African region.
2 WHO-CDC-CLSI Laboratory Quality Management System Training Toolkit,
2009 (http://www.who.int/ihr/training/laboratory_quality/doc/en/)
Training toolkit intended to provide comprehensive materials that will
allow for designing and organizing training workshops for all
stakeholders in health laboratory processes.
3 Quality Manual Template, 2013 (http://www.who.int/ihr/training/
laboratory_quality/quality_manual/en/)
Quality manual template supplement to the laboratory quality
management system training toolkit, Module 16dDocuments and
records.
4 WHO-CDC-CLSI Laboratory Quality Management System Training
Handbook, 2011 (http://www.who.int/ihr/publications/lqms/en/)
Handbook that covers topics essential for quality management of public
health or clinical laboratory. They are based on both ISO 15189 and CLSI
GP26-A3 documents. The handbook is linked to the training toolkit on
laboratory quality management system.
5 WHO Laboratory Assessment Tool (LAT), 2012 (http://www.who.int/ihr/
publications/laboratory_tool/en/)
Document that describes the general process for assessing laboratories
and provides two questionnaires to help assess national laboratory
systems (Annex 1) and individual laboratories (Annex 2).
6 WHO-AFRO Guide for Establishing Laboratory Based Surveillance for
Antimicrobial Resistance, 2013 (http://apps.who.int/medicinedocs/
documents/s20135en/s20135en.pdf)
Guide that provides background information and defines key steps for
countries to conduct AMR surveillance for meningitis, bacteraemia and
common enteric epidemic-prone diseases in national bacteriology
reference laboratory.
7 Laboratory Quality Stepwise Implementation (LQSI) Tool, 2015 (https://
extranet.who.int/lqsi/)
Web-based tool that provides a stepwise plan to guide medical
laboratories towards implementation of quality management system in
accordance with requirements of ISO 15189. Developed by Royal
Tropical Institute Netherlands for World Health Organization [56].
8 Basic Laboratory Information System (BLIS) (http://blis.cc.gatech.edu/
index.php)
Open-source system to track patient specimens and laboratory results,
developed by C4G (Computing for Good) and implemented through
AFENET [57].
9 WHONET (http://www.whonet.org/) Free software developed by WHO Collaborating Centre for Surveillance
of AMR for management and analysis of microbiology laboratory data
with special focus on analysis of antimicrobial susceptibility test results.
WHONET 5.6 with 2016 breakpoints is the most commonly used around
the world and offers support for 27 languages. WHONET 2017 with
GLASS support is currently only available in English.
AFENET, African Field Epidemiology Network; BLIS, Basic Laboratory Information System; AMR, antimicrobial resistance; CDC, Centers for Disease Control and Prevention;
CLSI, Clinical and Laboratory Standards Institute; EQAP, External Quality Assessment Programme; GLASS, Global Antimicrobial Resistance Surveillance System; LAT, Laboratory
Assessment Tool; LQSI, Laboratory Quality Stepwise Implementation Tool; NCID, National Institute for Communicable DiseasesdSouth Africa; WHO, World Health
Organization.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e87
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007

[9] International Organization for Standardization. Medical laborator-
iesdrequirements for quality and competence. International standard ISO
15189. 3rd ed. Geneva: ISO Copyright Office; 2012.
[10] Petti CA, Polage CR, Quinn TC, Ronald AR, Sande MA. Laboratory medicine in
Africa: a barrier to effective health care. Clin Infect Dis 2006;42:377e82.
[11] Bates I, Maitland K. Are laboratory services coming of age in sub-Saharan
Africa? Clin Infect Dis 2006;42:383e4.
[12] Nkengasong JN. Strengthening laboratory services and systems in resource-
poor countries. Am J Clin Pathol 2009;131:774.
[13] Alemnji GA, Zeh C, Yao K, Fonjungo PN. Strengthening national health labo-
ratories in sub-Saharan Africa: a decade of remarkable progress. Trop Med Int
Health 2014;19:450e8.
[14] Andiric LR, Massambu CG. Laboratory quality improvement in Tanzania. Am J
Clin Pathol 2015;143:566e72.
[15] Peter TF, Rotz PD, Blair DH, Khine AA, Freeman RR, Murtagh MM. Impact of
laboratory accreditation on patient care and the health system. Am J Clin
Pathol 2010;134:550e5.
[16] Yao K, Luman ET. Evidence from 617 laboratories in 47 countries for SLMTA-
driven improvement in quality management systems. Afr J Lab Med 2014;3(3).
[17] Nkengasong JN, Nsubuga P, Nwanyanwu O, Gershy-Damet GM, Roscigno G,
Bulterys M, et al. Laboratory systems and services are critical in global health:
time to end the neglect? Am J Clin Pathol 2010;134:368e73.
[18] Ndihokubwayo JB, Kasolo F, Yahaya AA, Mwenda J. Strengthening public
health laboratories in the WHO African region: a critical need for disease
control. African Health Monitor 2010;12:47e52.
[19] Parsons LM, Birx D, Nkengasong J, Somoskovi A, Lee E, Paramasivan CN, et al.
Global health: integrating national laboratory health systems and services in
resource-limited settings: original research. Afr J Lab Med 2012;1:1e5.
[20] Ndihokubwayo JB, Maruta T, Ndlovu N, Moyo S, Yahaya AA, Coulibaly SO, et al.
Implementation of the World Health Organization Regional Office for Africa
stepwise laboratory quality improvement process towards accreditation. Afr J
Lab Med 2016;5:1e8.
[21] Schroeder LF, Amukele T. Medical laboratories in sub-Saharan Africa that
meet international quality standards. Am J Clin Pathol 2014;141:791e5.
[22] Best M, Sakande J. Practical recommendations for strengthening national and
regional laboratory networks in Africa in the Global Health Security era. Afr J
Lab Med 2016;5:1e10.
[23] Audu RA, Onubogu CC, Okoye RN, Nwokoye NN, Onwuamah CK, Musa AZ,
et al. Proficiency testing for HIV, tuberculosis and malaria diagnosis in clinical
laboratories in Nigeria: original research. Afr J Lab Med 2014;3:1e9.
[24] Boeras DI, Peeling RW, Onyebujoh P, Yahaya AA, Gumede-Moeletsi HN,
Ndihokubwayo JB. The WHO AFRO external quality assessment programme
(EQAP): linking laboratory networks through EQA programmes: lessons from
the field. Afr J Lab Med 2016;3:1e6.
[25] Cham F, Belabbes EH, Gershy-Damet GM, Maleka M, Masango M, Goetsch E,
et al. The World Health Organization African region external quality assessment
scheme for anti-HIV serology: original research. Afr J Lab Med 2012;1:1e6.
[26] Chongo P, Sitoe N, Viegas S, Pinto I, Macave A, Sitoe F, et al. Quality assurance
for point-of-care testing in Mozambique's National Health Service: country
profile. Afr J Lab Med 2016;3:1e3.
[27] Meyers AF, Sandstrom P, Denny TN, Hurlston M, Ball TB, Peeling RW, et al.
Quality assurance for HIV point-of-care testing and treatment monitoring
assays. Afr J Lab Med 2016;5:1e7.
[28] Mukadi P, Gillet P, Lukuka A, Atua B, Kahodi S, Lokombe J, et al. External
quality assessment of malaria microscopy in the Democratic Republic of the
Congo. Malar J 2011;10:308.
[29] World Health Organization. International health regulations. 2nd ed. Geneva:
World Health Organization; 2005.
[30] Frean J, Perovic O, Fensham V, McCarthy K, von Gottberg A, de Gouveia L, et al.
External quality assessment of national public health laboratories in Africa,
2002e2009. Bull World Health Organ 2012;90. 191Ae99A.
[31] Hidri N, Marzuck V. L'accr
editation au laboratoire de bact
eriologie. In: Denis F,
Ploy M, Martin C, Bingen E, Quentin R, editors. Bact
eriologie m
edicale. Tech-
niques usuelles. 2nd ed. Issy-les-Moulineaux: Elsevier Masson; 2011.
p. 109e16.
[32] Chandler L. Challenges in clinical microbiology testing. In: Dasgupta A,
Sepulveda JL, editors. Accurate results in the clinical laboratory. A guide to
error detection and correction. Amsterdam: Elsevier; 2013. p. 315e26.
[33] Wilson ML. Assuring the quality of clinical microbiology test results. Clin
Infect Dis 2008;47:1077e82.
[34] Bouchet N. ISO 151892012: what changes for African laboratories? [opinion
paper]. Afr J Lab Med 2015;4:1e3.
[35] Snyder SR, Favoretto AM, Derzon JH, Christenson RH, Kahn SE, Shaw CS, et al.
Effectiveness of barcoding for reducing patient specimen and laboratory
testing identification errors: a laboratory medicine best practices systematic
review and meta-analysis. Clin Biochem 2012;45:988e98.
[36] Sakande J, Nikiema A, Kabre E, Sawadogo C, Nacoulma EW, Sanou M, et al.
Implementation of a national external quality assessment program for med-
ical laboratories in Burkina Faso: challenges, lessons learned, and perspec-
tives. Am J Clin Pathol 2014;141:181e7.
[37] Rieder HL, Van Deun A, Kam KM, Kim SJ, Chonde TM, Tr
ebucq A, et al. Pri-
orities for tuberculosis bacteriology services in low-income countries. 2nd ed.
Paris: International Union Against Tuberculosis and Lung Disease; 2007.
[38] Affolabi D, Tanimomo-Kledjo B, Anyo G, Johnson RC, Anagonou SY, Portaels F.
Setting up a national reference laboratory for Buruli ulcer: the case of Benin.
Trop Med Int Health 2008;13:365e8.
[39] Mukadi P, Lejon V, Barbe B, Gillet P, Nyembo C, Lukuka A, et al. Performance of
microscopy for the diagnosis of malaria and human African trypanosomiasis
by diagnostic laboratories in the Democratic Republic of the Congo: results of
a nation-wide external quality assessment. PLoS One 2016;11:e0146450.
[40] WHO Prequalification Team: Diagnostics. World Health Organization. WHO
manual for organizing a national external quality assessment programme for
health laboratories and other testing sites. Geneva: World Health Organiza-
tion; 2016.
[41] Kettelhut MM, Chiodini PL, Edwards H, Moody A. External quality assessment
schemes raise standards: evidence from the UKNEQAS parasitology sub-
schemes. J Clin Pathol 2003;56:927e32.
[42] Barlam TF, Cosgrove SE, Abbo LM, MacDougall C, Schuetz AN, Septimus EJ,
et al. Implementing an antibiotic stewardship program: guidelines by the
Infectious Diseases Society of America and the Society for Healthcare Epide-
miology of America. Clin Infect Dis 2016;62:e51e77.
[43] Tuijn CJ, Van den Broek A, Msoka E, Sumari-de Boer M, Chilongola J, Mushi DL.
The interface between clinicians and laboratory staff: a field study in northern
Tanzaniadoriginal research. Afr J Lab Med 2014;3:1e7.
[44] Van den Broek A, Tuijn CJ, Van't Klooster L, Oskam L, Msoka E, Sumari-de
Boer M, et al. Understanding the interface between clinical and laboratory
staff: original research. Afr J Lab Med 2014;3:1e6.
[45] Namagembe A, Ssekabira U, Weaver MR, Blum N, Burnett S, Dorsey G, et al.
Improved clinical and laboratory skills after team-based, malaria case man-
agement training of health care professionals in Uganda. Malar J 2012;11:44.
[46] Bouza E, Sousa D, Munoz P, Rodriguez-Creixems M, Fron C, Lechuz JG.
Bloodstream infections: a trial of the impact of different methods of reporting
positive blood culture results. Clin Infect Dis 2004;39:1161e9.
[47] Amsterdam D, Barenfanger J, Campos J, Cornish N, Daly JA, Della-Latta P, et al.
Cumitech 41. Detection and prevention of clinical microbiology laboratory-
associated errors. Washington, DC: ASM Press; 2004.
[48] Barb
e B, Verdonck K, Mukendi D, Lejon V, Lilo Kalo JR, Alirol E, et al. The art of
writing and implementing standard operating procedures (SOPs) for labora-
tories in low-resource settings: review of guidelines and best practices. PLoS
Negl Trop Dis 2016;10:e0005053.
[49] Clinical and Laboratory Standards Institute. Quality management system:
development and management of laboratory documents. Approved guideline.
CLSI document QMS02-A6. 6th ed. Wayne, PA: Clinical and Laboratory Stan-
dards Institute; 2013.
[50] Agarwal R. Measurement of errors in clinical laboratories. Indian J Clin Bio-
chem 2013;28:227e34.
[51] Plebani M. Exploring the iceberg of errors in laboratory medicine. Clin Chim
Acta 2009;404:16e23.
[52] Yuan S, Astion ML, Schapiro J, Limaye AP. Clinical impact associated with cor-
rected results in clinicalmicrobiology testing. J Clin Microbiol2005;43:2188e93.
[53] Stratton CW. Clinical microbiology. In: Laposata M, editor. Clinical diagnostic
tests. How to avoid errors in ordering tests and interpreting results. New
York: Demos Medical Publishing; 2016. p. 191e257.
[54] Goodyear N, Ulness BK, Prentice JL, Cookson BT, Limaye AP. Systematic
assessment of culture review as a tool to assess errors in the clinical micro-
biology laboratory. Arch Pathol Lab Med 2008;132:1792e5.
[55] Kinkus CA. Laboratory management. In: Laposata M, editor. Clinical diagnostic
tests. How to avoid errors in ordering tests and interpreting results. New
York: Demos Medical Publishing; 2016. p. 259e86.
[56] Perrone LA, Voeurng V, Sek S, Song S, Vong N, Tous C, et al. Implementation
research: a mentoring programme to improve laboratory quality in Cambodia.
Bull World Health Organ 2016;94:743e51.
[57] Masanza MM, Nqobile N, Mukanga D, Gitta SN. Laboratory capacity building
for the International Health Regulations (IHR[2005]) in resource-poor coun-
tries: the experience of the African Field Epidemiology Network (AFENET).
BMC Public Health 2010;10(Suppl 1):S8.
[58] Regional global health security agenda consulation for laboratory strength-
ening in Africa. Executive summary. 16/10/2015.
B. Barb
e et al. / Clinical Microbiology and Infection xxx (2017) 1e88
Please cite this article in press as: Barb
e B, et al., Implementation of quality management for clinical bacteriology in low-resource settings,
Clinical Microbiology and Infection (2017), http://dx.doi.org/10.1016/j.cmi.2017.05.007