ArticlePDF Available

Improving health worker motivation and performance to deliver adolescent sexual and reproductive health services in the Democratic Republic of Congo: study design of implementation research to assess the feasibility, acceptability, and effectiveness of a package of interventions



Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
Global Health Action
ISSN: (Print) (Online) Journal homepage:
Improving health worker motivation and
performance to deliver adolescent sexual
and reproductive health services in the
Democratic Republic of Congo: study design of
implementation research to assess the feasibility,
acceptability, and effectiveness of a package of
Sheri Bastien, Erin Ferenchick, Symplice Mbola Mbassi, Marina Plesons &
Venkatraman Chandra-Mouli
To cite this article: Sheri Bastien, Erin Ferenchick, Symplice Mbola Mbassi, Marina Plesons
& Venkatraman Chandra-Mouli (2022) Improving health worker motivation and performance
to deliver adolescent sexual and reproductive health services in the Democratic Republic
of Congo: study design of implementation research to assess the feasibility, acceptability,
and effectiveness of a package of interventions, Global Health Action, 15:1, 2022280, DOI:
To link to this article:
© 2022 The Author(s). Published by Informa
UK Limited, trading as Taylor & Francis
Published online: 20 Jan 2022.
Submit your article to this journal View related articles
View Crossmark data
Improving health worker motivation and performance to deliver adolescent
sexual and reproductive health services in the Democratic Republic of Congo:
study design of implementation research to assess the feasibility,
acceptability, and effectiveness of a package of interventions
Sheri Bastien
, Erin Ferenchick
, Symplice Mbola Mbassi
, Marina Plesons
and Venkatraman Chandra-Mouli
Department of Public Health Science, Norwegian University of Life Sciences, Ås, Norway;
Cumming School of Medicine, Department of
Community Health Sciences, University of Calgary, Calgary, Canada;
Technical Advice and Partnership Department, Geneva,
Regional Office for Africa, World Health Organization, Geneva, Switzerland;
Department of Sexual and Reproductive
Health and Research, World Health Organization, Geneva, Switzerland
During its last funding cycle from 2018–2020, the Global Fund in collaboration with the
Ministry of Health, World Health Organization, and implementing partners Cordaid and Santé
Rural (SANRU), implemented a multi-sectoral, contextualized approach to improve the sexual
and reproductive health of adolescent girls and young women in two regions in the
Democratic Republic of the Congo, which included community-based, school-based and
health facility-based actions. This implementation research focuses on the health-facility
component. The objective of this research is to evaluate the feasibility, acceptability, and
effectiveness of a package of interventions to improve health workers’ knowledge, skills, and
attitudes in providing sexual and reproductive health services to adolescents, whilst con-
comitantly creating an enabling work environment for building health workers’ motivation.
The package includes a combination of job descriptions, training and refresher training, desk
reference tools, and collaborative learning. The package did not focus on improving ame-
nities, providing or repairing equipment, or providing medicines and supplies. The underlying
theoretical framework informing the project and the implementation research draws from
Social Network Theory, Diffusion of Innovations and Normalization Process Theory. Qualitative
and quantitative process and outcome data from in-depth interviews and focus group
discussions with health workers and health managers, field notes, monitoring reports, costing
sheets, and health worker surveys, adolescent mystery client assessments, and exit interviews
with adolescents will be collected as part of a time-series study. The findings from this
implementation research will be utilized to inform future adaptations and/or scale-up of
the package of interventions to improve health worker motivation and performance in the
Democratic Republic of the Congo and elsewhere. The findings will also contribute to
advancing the use of theoretical approaches within the field of implementation research.
Received 22 October 2021
Accepted 20 December 2021
Stig Wall
Adolescent sexual and
reproductive health; health
worker training;
collaborative learning;
implementation research;
heath worker motivation;
health worker performance
Promotive, preventive, and curative health services
are an important component of a package of inter-
ventions to improve the sexual and reproductive
health (SRH) of adolescents. A crucial ingredient
of a high quality, friendly, and responsive health
service is a capable and empathic health worker.
Yet inadequate competency (i.e. knowledge and
skills to carry out a task), judgmental attitudes,
and low motivation leading to poor clinical and
interpersonal performance are widespread. This
has negative implications for the provision and
utilization of health services, especially SRH ser-
vices, for adolescents [1]. Delivering accessible,
high quality adolescent sexual and reproductive
health (ASRH) services that meet the needs and
preferences of adolescents could lead to increased
uptake of services and improved sexual and repro-
ductive health outcomes.
Evidence suggests that factors operating at the
individual, institutional and social levels affect health
worker performance, and that these factors need to be
addressed to improve health worker competencies,
attitudes (including biases) and motivation [2].
Likewise, it suggests that attention to the context is
crucial, and that local conditions must be addressed
by interventions. In terms of improving competen-
cies, evidence suggests that training alone is insuffi-
cient to lead to sustained improvements in health
worker performance, and that comprehensive
approaches that combine interactive and participa-
tory training, job aids, supportive supervision and
CONTACT Sheri Bastien Norwegian University of Life Sciences, Ås, Norway
2022, VOL. 15, 2022280
© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
collaborative learning are more effective than piece-
meal approaches in building competencies, positive
attitudes and motivation and thereby improving per-
formance [3,4]. However, in many instances training,
often one-off training that generally uses didactic
methods, is the only performance-improvement
approach that is used in large-scale programmes [5].
Similarly, much attention is given to the impor-
tance of financial incentives in sustaining health
worker motivation and performance. Whilst financial
incentives play a role in sustaining health worker
motivation and performance, a systematic review
that assessed the effectiveness of strategies to improve
health-care provider practices in low- and middle-
income countries found that interventions using
group problem solving plus training had a larger
effect size than those addressing financial incentives
[2]. Likewise, a study in India found that health
worker motivation and performance can be improved
through non-financial incentives, such as teamwork
and recognition [6]. Finally, a study in Benin and
Kenya found that non-financial incentives and
human resource management tools to provide an
enabling environment play an important role in
increasing health worker motivation [7].
While our knowledge of efficacious and effective
interventions to improve ASRH has improved, gaps
remain in our knowledge of how to deliver these
interventions effectively at scale in resource-
constrained settings, while ensuring quality and
equity. It is clear that interventions that have shown
to be effective in research or small-scale project con-
texts have, in many cases, not been effective when
implemented in large-scale programmes. However, it
is not always clear why this is so and what can be
done to remedy this. Implementation research can
help us to understand how and why implementation
is going well or not well, and to test approaches to
improve the situation in specific contexts [8].
In this manuscript, we describe the broader context in
which the project was implemented in, with a focus on
the implementation research on the health-facility com-
ponent which aims to assess the feasibility, acceptability,
and effectiveness of a package of interventions to
improve health worker motivation and performance
using a time-series design. The first and second rounds
of data collection are planned to be completed by
December 2021, with additional rounds planned in
2022. Here we describe the protocol for the implementa-
tion research. Forthcoming publications will describe the
implementation process and findings from the study.
In the Democratic Republic of the Congo (DRC),
adolescents in general and adolescent girls in
particular are at risk of poor sexual and reproduc-
tive health outcomes for a host of reasons at the
individual, community, and structural levels. Early
sexual debut, unprotected sexual activity, transac-
tional sex, and violence, including sexual violence
coerced sex are widespread [9,10]. The HIV epi-
demic in DRC is generalized with a prevalence of
0.7% in the general population (15 to 49 years) and
notably higher among women (1.1%) than men
(0.4%) [11]. Among young women, the prevalence
is 0.4% and among young men it is 0.2%. The low
HIV prevalence at the national level in the DRC in
comparison to neighboring countries masks geo-
graphic disparities within the country with preva-
lence higher in urban areas. Further, the proportion
of ever-partnered women aged 15–49 years who
have reportedly experienced intimate partner phy-
sical and/or sexual violence at least once in their
lifetime is 51% [10], although even this figure is
likely an underestimate given that a substantial
proportion of survivors do not report or seek care
and support.
Objectives of the overall project
To address these issues and challenges, the Global
Fund to Fight AIDS, Tuberculosis and Malaria
(Global Fund) in collaboration with the Ministry
of Health, World Health Organization, and imple-
menting partners Cordaid and Santé Rural
(SANRU), implemented a multi-sectoral, contex-
tualized approach to improving the SRH of adoles-
cent girls and young women (AGYW) in two
regions in DRC (Kasai Oriental and Kinshasa),
which included community-based, school-based
and health facility-based actions. The objectives of
the project were: i) to increase the knowledge and
understanding of SRH, HIV, human rights and
GBV among AGYW; (ii) to reduce GBV cases in
the school and the community; and iii) to improve
the provision of adolescent-responsive health ser-
vices in terms of both quality and access. The
selection criteria for the geographical distribution
of the study sites participating in the project
included HIV prevalence rates among AGYW
aged 15–24 years, as well as the presence of health
services already being supported by the GFATM
through existing grants. Intervention activities
within health facilities focused on 3 health zones
in each of the two regions were intended to include
coordination strengthening, supportive supervision,
the provision of in-service training, job aids, and
collaborative learning sessions. They aimed to build
the capacity of health workers to respond to ado-
lescent clients more effectively and with sensitivity
and empathy, while ensuring that health workers
have access to necessary supports (i.e. health
products financed out of the grants) to enhance
health worker performance through an enabling
work environment.
Formative phase (2017-2019)
In the first phase of the project, formative research
was undertaken to assess the quality of health service
provision to adolescents and to identify actions to
build on strengths and address areas of weakness.
A baseline assessment was carried out in 30 health
facilities (5 facilities in each of the 3 health zones in
Kasai Oriental and Kinshasa, respectively) in
November 2017, consisting of semi-structured inter-
views with health workers and adolescent clients, as
well as systematic observation of health facilities
[12,13]. The reports from these assessments note
that facilities had basic equipment and essential med-
icines, and that they were staffed and functioning
staff in place. While the findings in both Kasai
Oriental and Kinshasa indicate that the majority of
health workers had not been trained on the provision
of SRH services to adolescents, health workers
reported that they respect adolescents and young
people, that they feel confident in providing services
to them, and that they do not judge them. However,
the reports highlight that none of the facilities offered
a complete health service package or educational
materials tailored to the needs of adolescents; or
specific activities related to this group. In addition,
almost all facilities lacked preferential pricing for
adolescents and young people. Finally, adolescent
clients themselves reported dissatisfaction with for
example the length of waiting time to receive services
In terms of actions, an initial two-day health
worker training in Kinshasa was conducted in 2017
by the World Health Organization Regional Office
for Africa (WHO AFRO) and subsequently in Mbuji-
Mayi, Kasai Oriental in early 2018, with a focus on
developing a shared understanding of the period of
adolescence, adolescents’ specific health needs, and
how to deliver SRH services to adolescents with dig-
nity and respect. Following this, desk reference tools
were distributed to them.
In March 2018, WHO-AFRO facilitated a four-day
meeting to sensitize the Ministry of Health, specifi-
cally the National HIV/AIDS Control Programme
(Programme National de Lutte contra le VIH/Sida,
PNLS) and the National Adolescent Health
Programme (Programme National de Santé de
l’Adolescent, PNSA), implementing partners, and
key stakeholders to the collaborative learning
approach. This meeting was also used as the forum
for validating a draft operational manual prepared by
WHO AFRO for the collaborative learning model to
be piloted in the project. Collectively the participants
reviewed and adapted this draft, ultimately develop-
ing a national ‘Collaborative Learning Guide for
Improving the Performance of Health Service
Providers’ specific for the context of DRC.
Subsequently, in April 2018, a five-day workshop
to train facilitators from Kinshasa and Kasai Oriental
on the collaborative learning approach was organized
by the Ministry of Health under the leadership of
PNLS and PNSA, with the financial support of
CORDAID, the primary recipient of the GFATM
grant. WHO AFRO provided the technical support
and facilitated the workshop and the GFATM pro-
vided additional support. The sessions were attended
by 30 participants across all levels of the health sys-
tem from six target health zones in this project with
the intention to form a team of facilitators comprised
of experts at the national and provincial levels, as well
as members of the health zone management teams
[14]. This pool of facilitators with expertise in colla-
borative learning strategies and ASRH has been
responsible for supporting the collaborative learning
sessions with health workers and managers from the
identified health facilities. In addition, this pool of
facilitators includes national experts to generate own-
ership and support systems to contribute to sustain-
ability. Between June 2018 and May 2019, 32
collaborative learning sessions were arranged by
Cordaid and PNLS, with logistical support from
Réseau National des ONG pour le Développement
de la Femme (RENADEF), a local civil society orga-
nization, and technical support from the WHO and
the Global Fund. During 2019 and 2020 there were 56
collaborative learning sessions held in Kinshasa and
Mbuji-Mayi. There are plans for additional sessions
to take place during 2021–2022 with funding from
the WHO.
Alongside efforts to put in place evidence-based
and scalable approaches with proven health and
social benefits, this implementation research is being
undertaken in order to understand how to tailor and
deliver interventions that have shown to be effective
in research studies and pilot projects. It thus presents
a unique opportunity to address the pressing need for
evidence in the area of improving health worker
knowledge, skills, and attitudes and how this influ-
ences their motivation and performance in relation to
the provision of SRH services to adolescents in this
The package of interventions
A package of interventions to enhance health work-
ers’ performance, tailored to the local context, was
targeted primarily to health workers, but also to some
extent involving health managers at participating
health facilities and the health zone management
team. This package draws on the available evidence
of effective interventions, including a combination of
job descriptions, training related to adolescence and
adolescent health issues, quality adolescent and youth
friendly health services, including refresher training,
monitoring and desk reference tools and collaborative
learning [3]. In addition, a platform was created for
health workers to contribute to ongoing discussions
on how to improve working conditions, which aimed
to contribute to increased motivation and sustainabil-
ity of efforts to strengthen the quality of services.
Protocols for training, refresher training, and colla-
borative learning, which are aligned with existing
PNSA/DRC standards as well as WHO quality stan-
dards and job descriptions, were developed. The
package of interventions did not focus on improving
amenities, providing or repairing equipment, or pro-
viding medicines and supplies.
An overview of the components of the package of
interventions to improve health worker competen-
cies, attitudes, motivation, and performance at the
health facilities is provided below. In addition to
these components, routine supervision was regularly
provided by the Ministry of Health and on occasion
Despite early intentions of the project to strengthen
routine supervision, this aspect was not fully realized.
An in-depth overview of the collaborative learning
approach is provided in Box 1 below.
The implementation research at the
health-facility level
This implementation research is nested within the
Global Fund supported project and focuses on the
health facility-based actions. The objectives of the pre-
sent study are to evaluate the feasibility, acceptability,
and effectiveness of a package of interventions to:
(a) improve health workers’ knowledge and skills
in providing SRH services to adolescents and
to build positive attitudes regarding the provi-
sion of SRH services to adolescents; and
(b) create an enabling work environment for
building health workers’ motivation to apply
their competencies and positive attitudes
towards the provision of SRH services for
Together, these actions are intended to improve and
sustain improvements in health workers’ performance.
Theoretical framework
Public health interventions tend to be more effective
if they are grounded in social and behavioural science
theory, which may help predict or explain the
pathway to a desired outcome [15]. Within this
study, the specific focus on the collaborative learning
approach to promote group problem solving and
improve health worker competencies, attitudes, prac-
tices, and motivation concerning the provision of
SRH services to adolescents draws on a cluster of
interrelated theories: Social Network Theory [16],
Diffusion of Innovations [17,18], and Normalization
Process Theory [19]. Each of these theories contri-
butes to understanding how attitudes, norms and
practices become embedded in different social set-
tings, including health facilities.
Social Network Theory (SNT) focuses on the
structure of relations among individuals with regard
to their network and the impact this has on attitudes
and practices from the individual to the collective and
systems levels. The density and distribution of ties are
considered important to understand the spread of
information and influence according to the theory,
as is social support and the extent to which contacts
within the network are homophilous. Within imple-
mentation science, SNT can be useful for understand-
ing a programme’s adoption, implementation, and
sustainability. From this perspective, it is essential to
consider three interrelated aspects of an intervention
or programme: partnerships between researchers,
practitioners, and the wider community, as well as
stakeholders such as policy makers; the characteristics
of those who implement interventions as well as the
recipients; and the wider context in which the inter-
vention is implemented [20].
Diffusion of Innovations also focuses on social
networks and is most often used to investigate the
adoption or rejection of new innovations or practices,
especially the rate of adoption or rejection and how
innovations diffuse through social systems to influ-
ence social norms [18]. Diffusion is characterized by
five stages: knowledge, persuasion, decision, imple-
mentation, and confirmation. This theory suggests
that innovations typically spread following an ‘S’
shaped curve, whereby uptake by early adopters is
eventually followed by the majority, at which point
the innovation becomes commonplace.
Normalization Process Theory (NPT) focuses on
mechanisms that influence how a new norm or inter-
vention can become embedded and normalized in
everyday practice in institutional and other social con-
texts and settings, and which factors enable or hinder
this process [19]. A recent systematic review of the use
of NPT in complex healthcare interventions found that
it can be useful in all programming phases, from
intervention development and implementation plan-
ning, to evaluation of complex implementation pro-
cesses [21]. According to NPT, practice is routinely
integrated into social contexts as a result of the indi-
vidual and collective work done in connection with the
implementation [21]. Within health care settings,
implementation of interventions is highly complex due
to organizational structures and how they interface
with the social contexts in which they are embedded.
NPT can be useful for mapping and explaining the
relationship between actors, norms, practices, and con-
texts in relation to the underlying mechanisms of
implementation processes.
These three theories are particularly relevant to
this study given their emphasis on:
(i) the importance of communication in social
networks (in the context of this study, through
group problem solving and collaborative learn-
ing among colleagues), to catalyse change,
(ii) the role of modelling, opinion leaders and
imitation of others who have successfully
adopted newly introduced practices, and,
(iii) the necessity of an enabling environment and
organizational culture that is supportive of
evidence-based approaches such as collabora-
tive learning, and
(iv) the tailoring of approaches to suit the cul-
tural, economic, political, and social context
of the setting.
This study aims to build knowledge and skills and
change potentially deep-seated attitudes among health
workers through the introduction of the package of
interventions, and particularly the collaborative learn-
ing approach. Given this, drawing on this cluster of
theories both strengthens and guides the implementa-
tion research at all phases, from inception through to
analysis of the role of the intervention components in
shifting practices and norms, as well as the nature and
extent of the impact of the intervention components on
the provision of SRH services to adolescents.
Study setting
As described above, the health facility-based compo-
nent of the project is being implemented in 3 health
zones in both Kinshasa and Kasai Oriental. Kinshasa is
the country’s capital city and is one of the continents
fastest growing cities, with an estimated population of
14,970,000, whereas Mbuji-Mayi, the capital city of
Kasai Oriental, has approximately 2,643,000 inhabi-
tants [22]. Kinshasa is linguistically diverse with inha-
bitants speaking French and other local languages such
as Lingala, Kikongo, Tshiluba, and Swahili. Kinshasa’s
main economic activities include food processing,
manufacturing, construction, and provision of ser-
vices, and it is an important logistical and transporta-
tion hub within Africa. Kasai Oriental is also
linguistically and culturally diverse, and the socio-
economic setting in the province is influenced by the
presence of a diamond industry, which provides
employment for a large segment of the population.
Study design and methods
The study is based on a mixed-methods time-series
design. As noted above, a package of interventions is
being implemented, and ongoing collaborative learning
sessions will take place through 2022. An overview of
the package of interventions is provided in Table 1
below. Also mentioned previously, the first two rounds
of data collection are planned to take place by the end of
December 2021, with 3–4 additional rounds of data
collection planned in 2022. The package of interven-
tions will be assessed through process and outcome
evaluations. The University of Kinshasa is leading the
process and outcome evaluation (2021–2023), with
technical support from the WHO. This section
describes the implementation research objectives,
research questions and data collection methods that
will be common across each round of data collection,
with an overview provided in Table 2 below.
Across all data collection rounds, data will be collected
by local research assistants recruited by the University of
Kinshasa and trained by the University of Kinshasa and
the WHO. All data collection tools were translated from
English to French, pilot tested among health workers and
health workers at non-participating facilities and as role
Table 1. The package of interventions.
component Description and implementation plan Potential benefits
Job descriptions that set out functions of health managers and
workers, in line with existing PNSA documents. The
competencies required to carry out the functions are clearly
stated and in line with national standards and the WHO ‘Core
competencies in adolescent health and development for primary
care providers’ (WHO, 2015).
Improved clarity around objectives, responsibilities, authority
and lines of accountability in relation to adolescent-
responsive SRH/HIV services.
Training Competency-based in-service training for health managers and
workers with a focus on the provision of SRH, including GBV,
services to adolescents. Training is followed up with periodic
Improved knowledge and skills in providing adolescent-
responsive SRH/HIV services.
Job aids Desk reference tools to assist health workers in providing high
quality SRH/HIV services to adolescents.
Assurance that everyday work is in line with Standard
Operating Procedures.
Collaborative learning, including sharing of information with peers
and learning from and with them. Support for group problem
identification and solving.
Learning from others’ experiences and motivation. A cordial
and supportive social environment.
play simulations for the adolescent tools among research
team members and revised in April/May 2021. In-depth
interviews and focus group discussions with health
worker and health managers will be conducted in
French. In the case of mystery clients and adolescent
exit interviews, discussions will be conducted in which-
ever local language the adolescent prefers. Systematic
debriefs will take place among the research team after
each data collection to ensure richness of the data is not
lost. The details of each type of process and outcome data
collected are provided below.
Process evaluation
The objective of the process evaluation is to develop an
understanding of the feasibility and acceptability of the
package of interventions. The qualitative process eva-
luation consists of a mix of methods to allow for
triangulation, including in-depth interviews, focus
group discussions, as well as field notes, monitoring
and collaborative learning reports, and costing sheets.
The process evaluation consists of an analysis of fac-
tors related to intervention package including its deliv-
ery (feasibility and cost) and the perceptions of those
targeted (adoption and acceptability). This is essential
to understand contextual factors affecting the effective-
ness of the intervention package and lessons learned in
the implementation process in a ‘real world’ setting.
In-depth interviews and focus group discussion
with health workers and health managers
The sampling strategy across all rounds of data col-
lection will be purposive and stratified by health zone
to ensure that rural and urban facilities are
adequately covered, with the final number of in-
depth interviews and focus group discussions deter-
mined by data saturation.
In-depth interviews and focus group discussions
across all rounds of data collection will consist of
a convenience sample of health workers and managers
from participating health facilities who have received
training and participated in collaborative learning ses-
sions to gain insight into their perspectives and experi-
ences. Specifically, at least two in-depth interviews will
be conducted (one with the manager and one health
worker at half of participating facilities) for a total of
approximately 30 in each round of data collection.
Two focus group discussions will be held with health
workers in both Kinshasa and Kasai Oriental for
a total of four in each round of data collection, to
gain insight into attitudes, beliefs, group norms, and
dynamics, as well as perceptions of change as a result
of the packages of interventions.
The main themes explored in the discussions
include: the extent to which the intervention package
met their needs, their perceptions, and experiences
with the collaborative learning approach, their ability
to implement in practice what they learned, and bar-
riers and facilitators that affect their ability to translate
their knowledge, skills, and attitudes into practice.
Field notes
The research team is responsible for taking detailed
field notes using a template developed for the pro-
ject’s data collection manual to ensure standardiza-
tion and consistency after all data collection activities
to document other relevant initiatives in the
Table 2. Overview of objectives, research questions, and data collection methods.
Objectives Evaluation/research questions
Data collection methods and
To evaluate the feasibility, acceptability, and
effectiveness of a package of interventions to:
a) improve health workers’ knowledge and skills in
providing SRH services to adolescents and judgment
on where/when/how to apply their them (together
contributing to their competence), and to build
positive attitudes regarding the provision of SRH
services to adolescents; and
b) create an enabling work environment for building
health workers’ motivation to apply their
competencies and attitudes for SRH services for
How did the health managers perceive the package of
How did they perceive each component of the
package, and especially the collaborative learning
(in terms of meeting the needs of health workers with
respect to feasibility, and in terms of acceptability)
In-depth interviews with health
Reports of collaborative learning
sessions, supervision, and
monitoring reports.
Field notes.
How much did the delivery of the intervention
package cost?
Costing sheets.
How did the health workers perceive the package of
How did the health workers perceive each component
of the package of interventions, and especially the
collaborative learning sessions?
(in terms of meeting their needs and in terms of
In-depth interviews with health
Focus group discussions with
health workers.
Field notes.
How effective was the package of interventions in
improving health workers’:
a. competencies in responding to the SRH needs of
adolescent clients,
b. attitudes towards meeting these needs,
c. motivation to apply their competencies and
attitudes to the best of their ability, and
d. performance (i.e. clinical and interpersonal
In-depth interviews with health
workers including hypothetical
scenarios and role plays.
Mystery client assessments.
Exit interviews with adolescents
and young women.
Health worker surveys.
participating health facilities and communities. This
will ensure that a broader understanding of the con-
text can be taken into account during the data ana-
lysis process and will assist in understanding
attribution of any potential changes in key outcomes
to the project.
Monitoring of implementation
The implementation of the package of interventions
will be monitored using a simple reporting form
designed to collect information on what was done
as well as reflections on the sessions, what went
well, and what could be improved in future sessions.
Collaborative learning sessions and other monitoring
reports will be reviewed during each of the planned
rounds of data collection through 2022 to provide
information that can be triangulated on evolution of
changes, if any.
Cost analysis
An activity-based costing (ABC) approach will be
used to determine how much it costs to deliver the
package of interventions, simple cost analysis of the
package of interventions will be undertaken to assess
feasibility of scale-up, and to inform budgeting for
future efforts. This involves a review of documenta-
tion from implementing partners RENADEF and
Cordaid of the costs of each activity to contribute to
an understanding of the cost of each component of
the packages of interventions. Initial startup costs for
the collaborative learning component of the project
have been detailed elsewhere [23].
Outcome evaluation
The objective of the outcome evaluation is to assess
the effectiveness of the package of interventions in
improving health workers’ knowledge, skills, atti-
tudes, and motivation. A quasi-experimental design
using survey data collected at four time points, each
six months apart will be conducted. Additionally,
mystery client assessments and adolescent client exit
interviews which will also be conducted at six- month
intervals to understand adolescent experiences and
perceptions of services received from health workers
trained as part of the project.
Health worker survey
The full details of the validated measures and scales
used in the survey will be provided in subsequent
publications. The survey was initially piloted in 2021
on a sample of 20 health workers at health facilities
not participating in the intervention and revised
accordingly. Across each round of data collection,
approximately 60 health workers who have received
training as part of the project will complete the sur-
vey. The survey will be administered to the same
sample of participants four more times, at 6-month
intervals. Attrition may reduce the numbers of
respondents in the follow-up surveys due to staff
turnover and other factors; however, efforts will be
made to ensure that as many health workers as pos-
sible who received training and ongoing support
complete the follow-up surveys.
Mystery client assessments
Mystery client assessments will be conducted in each
round of data collection to provide an in-depth
understanding of adolescent client experiences of
receiving services from health workers supported as
part of the intervention. The perspectives of adoles-
cent mystery clients are essential to developing
a comprehensive, authentic understanding of how
young clients experience receiving health services.
This assessment will allow for triangulation with
adolescent client exit interviews to assess the percep-
tions and experiences of adolescents, and the impact
of the package of interventions on improving health
workers’ performance. The use of mystery clients has
been shown to be reliable, valid, feasible, and accep-
table for assessing the quality of health services [24].
Ten females between the ages of 18–25 (5 in
Kinshasa and 5 in Kasai Oriental) will be recruited
by senior members of the research team from
a youth organization, the Congolese Youth
Association Network (RACOJ). Selection criteria
for these positions will emphasize interpersonal skills
and professionalism, literacy, field note taking as
assessed in a practice session, and non-residence in
the vicinity of a participating health facility.
Potential mystery clients will be screened to identify
those who have previously experienced GBV-related
trauma. If they reported they have experienced GBV,
they will be referred for counselling and excluded
from the selection process to ensure psychosocial
harm does not occur as a result of participation in
the study. To the extent that it is possible, the same
mystery clients will be used in each round of data
collection, but they will visit different health facilities
in each round.
Mystery clients will receive a 3-day training pro-
vided by the University of Kinshasa to conduct the
mystery client assessments. The training will be based
on approaches that are standardized for training mys-
tery clients, and approaches which were identified as
best practices in a related systematic review [24].
Training modules will include sessions on ethics
and confidentiality, what to expect in terms of stan-
dard of care, how to engage in role play (including
rehearsal of possible scenarios), how to ask for infor-
mation and advice concerning SRH issues, how to
conduct observations and memory techniques, and
how to complete the checklists and reporting require-
ments. A refresher training will be provided to mys-
tery clients before each round of data collection.
After training, mystery clients will conduct random
visits on different days of the week and different times
of day to facilities to enact different scenarios, such as
requesting information on STIs including HIV, con-
doms, or contraceptive methods, or requesting infor-
mation on how to support a friend who has
experienced sexual harassment. They will be given
a pseudonym and fictional birth date, and they will
carry a letter of participation in case they face any
questions about their role in the study. They will be
instructed to specify that they are only seeking infor-
mation, and not to allow any invasive procedures such
as pelvic exams or blood tests. A trained staff member
from the University of Kinshasa will be available on
site when mystery client visits are conducted in case
any assistance is needed. The mystery clients will be
required to take detailed field notes concerning their
visit and participate in a debrief with the local project
manager. The assessment and debrief will focus on
whether mystery clients are satisfied with the services
they received and perceive that they have been treated
with respect and empathy and in a non-judgmental
manner. Mystery clients will be reimbursed for travel
costs, as well as a stipend and per diem for their time
for participating in the training and subsequent visits
to the health facilities.
Adolescent client exit interviews
The purpose of the adolescent client exit interviews is to
generate complementary data to the mystery client
assessments, which will provide an in-depth understand-
ing of adolescent client experiences of receiving services
from health workers supported as part of the project.
Only adolescents over the age of 18 years are
eligible to participate in exit interviews and in all
instances are required to sign an informed consent
form. Across each round of data collection, we aim to
conduct approximately 10 exit interviews (5 in
Kinshasa, 5 in Kasai Oriental) with adolescent clients
recruited immediately after they have received infor-
mation and services at participating health facilities
by trained members of the research team.
Key themes explored in the exit interviews with ado-
lescent clients include domains for assessing adolescent-
responsive health services in line with the WHO global
standards for quality health care services for adolescents,
such as equity and non-discrimination [25].
This mixed-methods study systematically employs quan-
titative and qualitative methods at all phases of the project
and across both the process and the outcome evaluation
components. The different types of data to be collected
have been selected to carefully align with the objectives of
the implementation research, to ensure complementarity,
reduce reporting bias, and enhance triangulation and
validity of the data. Using different methods will also
allow for mitigating the weaknesses associated with rely-
ing on one type of evidence. This is particularly important
given that satisfaction measures, for instance, may be
unreliable as adolescent clients may not be aware of
their rights, what constitutes quality care, and in general
are subject to social desirability bias. Additionally, a more
comprehensive and complete understanding of the
packages of interventions will be obtained by ensuring
adequate coverage of the range of perspectives about the
intervention among key stakeholders.
Planned data analyses
Qualitative data from in-depth interviews, focus group
discussions, mystery client assessments, and exit inter-
views across all rounds of data collection will be coded
thematically based both on a priori and emerging
themes by two members of the research team to
ensure quality control and consistency in coding in
Atlas.ti. Relevant constructs from the cluster of the-
ories that underpin the design and evaluation of the
project will also be used when analyzing in-depth
interviews and focus group discussions with health
workers and health managers to shed light on the
process of change in the provision of services to ado-
lescents as a result of the intervention. Among health
workers and health managers, themes to be explored
include: knowledge, skills, and attitudes related to the
provision of SRH services to adolescents; facilitators,
and barriers to providing quality care to adolescents
including the provision of care to those reporting
having experienced violence; factors influencing health
worker motivation, and performance; and perceptions
of relevance and adequacy of the support provided as
part of the Project. From the adolescent client perspec-
tive (both mystery clients and exit interviews), the
analysis will focus on whether they perceive that they
were treated with respect, sensitivity, dignity, and
empathy. Qualitative data will be transcribed in the
language in which the discussion was conducted and
translated into French if necessary. The accuracy of the
translations will be verified independently through
a process of back-translation.
Survey data will be exported from the ODK tablets
to Excel and after verification exported to Stata 14
analysis software. Statistical analyses will consist of
descriptive statistics including frequency tables, as
well as bivariate analyses to classify frequency distri-
butions of attitudes and practices according to knowl-
edge and skills levels. Logistic regression and time
series analyses may be used to examine associations
and look at changes over time.
Stakeholder engagement and dissemination
During the evaluation phase relevant stakeholders at
the local and national level such as the Ministry of
Health through PNSA, PNLS and health zone man-
agement teams, and local civil society bodies repre-
senting adolescents including RACOJ, RENADEF,
and CORDAID will be involved in a Technical
Advisory Committee (TAC) to ensure widespread
support and understanding of the project activities
and main findings.
The findings will be disseminated in national and
international fora and at relevant policy-level events
to ensure that the results promote policy dialogue
and improve the health sector response to the provi-
sion of services and care to adolescents.
Ethical considerations
The study adheres to the ethical and safety guidelines
laid out in the Standards and operational guidance for
ethics review of health-related research with human
participants [26] and the WHO guidance on ethical
considerations in planning and reviewing research stu-
dies on sexual and reproductive health in adolescents
[27]. The protocol has been approved by the WHO’s
Ethical Review Committee and the University of
Kinshasa’s Ethics Review Board.
All participants including adolescents (all of legal
age of majority i.e. over 18 years) participating in
mystery client assessments, adolescent exit interviews,
as well as health managers and health workers parti-
cipating in in-depth interviews, focus group discus-
sions, and completing the survey, are required to
provide written consent. Mystery client assessments,
adolescent exit interviews, in-depth interviews and
focus group discussions will be conducted in
a private setting to be set up at the health facilities.
Participants will be informed about the confiden-
tiality procedures as part of the consent process. To
ensure confidentiality, a code is assigned to each
participant instead of individuals’ names on data
collection tools, in transcripts and reports. No infor-
mation that can be used to identify individuals will be
transcribed or reported (for example indirect identi-
fiers such as names of other persons mentioned, place
names etc.). Any information, including potentially
sensitive information about participants will be kept
confidential and data analyses will be done on de-
individualized samples. An electronic master list of
names and codes will be managed by the PIs of the
project. Hard copies of consent forms will be kept in
a secure, locked location at Kinshasa School of Public
Health. All transcripts, translations, field notes, and
databases will be locked either physically or electro-
nically, with only members of the research team able
to access data. All confidential materials will be acces-
sible only to senior members of the research team.
Audio-recordings will be deleted once transcripts and
reports summarizing the findings have been finalized.
In-depth training has been given to data collectors on
issues related to confidentiality and providing refer-
rals in case of distress and refresher training will be
provided prior to each round of data collection.
The forthcoming findings of this implementation
research will shed light on the perspectives and
experiences of health workers and health managers
who have participated in the study, and adolescents
who have received services from health workers
trained through the project. We anticipate the find-
ings will also contribute to the growing body of
evidence of what works in terms of improving health
workers’ knowledge, skills, and attitudes in providing
sexual and reproductive health services to adoles-
cents. In particular, the study will generate findings
on the feasibility, acceptability and effectiveness of
the collaborative learning model as an approach to
train health workers (which could be relevant for
other target groups such as schoolteachers) to jointly
identify, define and solve issues related more broadly
to adolescent health issues ranging from non-
communicable diseases (NCDs), to communicable
diseases and a host of other issues such as early/
unwanted pregnancy, violence and mental health.
This implementation research will contribute new
knowledge related to several of the Sustainable
Development Goals (SDGs) and their achievement,
for instance the crucial role of the health workforce in
delivering on SDG goal 3 (health and well-being) and
SDG goal 5 (gender equality) in particular.
Competent and motivated health workers deployed
in the right places in adequate numbers, receiving full
support from health authorities and partners are
necessary for a strong primary health care system
and for making progress towards universal health
coverage, and specifically for promoting SRH and
preventing early pregnancies, and sexually trans-
mitted infections including HIV infection, and
detecting and responding to challenges when they
occur [28]. This forthcoming findings from this
study may also contribute to progress related to
SDG goals 3 and 5 by generating evidence of what
works and in which settings in terms of improving
and maintaining improvements in health worker
knowledge, skills, and positive attitudes towards the
provision of sexual and reproductive health care and
services to adolescents. Developing and rigorously
evaluating a package of interventions to train and
support health workers and create an enabling envir-
onment for them to apply their competencies and
attitudes and evaluating the effectiveness of these
interventions on key outcomes, is crucial in efforts
to improve health and well-being [29].
Additionally, the study findings may also contri-
bute new knowledge which addresses the issue of
SRH of adolescents, a cross-cutting theme in the
SDGs, through targets on ensuring universal access
to services, reducing the number of early pregnancies,
new HIV infections, eliminating violence against
women and girls, and reducing maternal mortality.
The inclusion of mystery client assessments and exit
interviews with adolescent clients will ensure their
perspectives and experiences of receiving health care
and services from trained providers are taken into
consideration in the planning of future interventions
and programmes.
Finally, this study, which aims to systematically
apply a cluster of interrelated theories from social
and behavioral science, may contribute to advancing
understandings of how new practices and norms
become adopted in health facilities. This is particu-
larly relevant given the study’s focus on introducing
the collaborative learning approach. The application
of a theoretical lens may also assist in identifying
barriers to uptake and provide important insights
for potential future adaptation and scale-up of the
package of interventions to other settings.
We do however acknowledge that there are limita-
tions to our study. Firstly, this implementation
research is nested within an ongoing initiative
which did not have an evaluation plan built into the
project at the outset. This also presents a unique
learning opportunity however, since it reflects the
reality of many similar projects, and we will be able
to demonstrate that it is possible to establish a robust
process and outcome evaluation in the midst of
implementation. Secondly, the study is only con-
ducted in two provinces in the DRC which may
limit the generalizability of findings to other contexts
and areas in the country. On the other hand, there is
one urban area and one rural area represented.
Thirdly, the sample size of health workers completing
the survey is relatively small thereby limiting our
statistical power. However, we have enrolled every
health worker who is a part of the initiative which
will provide a comprehensive overview of participant
perspectives in the project. An additional limitation
related to the study design is that the implementation
research lacks a comparison group which hinders our
ability to demonstrate a causal relationship between
the intervention and any outcomes. To address this
shortcoming, we may consider inclusion of
a comparison group in subsequent rounds of data
collection. Finally, as with any quality improvement
initiative, effects and sustainable change tend to be
slow to emerge and may be affected by health worker
or health manager turnover, competing priorities in
a complex health system, and other contextual fac-
tors. Nevertheless, our mixed-methods time series
design will allow us to triangulate data collected
using different methods to reduce bias, as well as
rigorously monitor change and document informa-
tion on any other initiatives aside from the interven-
tion that may influence reported outcomes and
provide important information for future studies
and efforts to scale-up.
Interventions which aim to move beyond one-off train-
ings and didactic approaches and towards comprehen-
sive and collaborative learning approaches to improve
health workers’ knowledge, skills, and attitudes in pro-
viding sexual and reproductive health services to ado-
lescents, alongside creating an enabling work
environment are urgently needed. We anticipate that
the findings from this mixed methods time-series
implementation research will be useful in subsequent
adaptations and/or scale-up of the package of interven-
tions to improve health worker motivation and perfor-
mance in the Democratic Republic of the Congo and
beyond. More broadly, the findings from the study will
also contribute to evidence base of how to apply theo-
retical approaches within the field of implementation
We would like to thank all local stakeholders including
PNLS and PNSA the health workers and adolescent clients,
Sylvie Olela Odimba, and the research team led by Eric
Mafuta at the University of Kinshasa, School of Public
Health. Sylvie Olela Odimba, and the research team at the
University of Kinshasa, School of Public Health.
Author contributions
VC (Venkatraman Chandra-Mouli) conceived the paper.
VC and MP (Marina Plesons) engaged SB (Sheri Bastien)
to lead the preparation of the paper. SB led the develop-
ment of the paper. Erin Ferenchick (EF), and Symplice
Mbola-Mbassi (SM) provided background information on
the project, relevant documents, and inputs from the per-
spectives of those who designed the initiative, oversaw its
execution, and supported it technically. VC, MP, and EF
contributed to the development of the paper with detailed
inputs on drafts. However, all authors contributed to the
paper as it evolved, and approved the final version.
Disclosure statement
No potential conflict of interest was reported by the
Ethics and consent
Ethical approval has been obtained by the WHO-ERC
(ERC.0003228) and the University of Kinshasa Ethics
Review Board. No primary data are reported in this study
design article.
Funding information
This research has received funding from the Global Fund
and the World Health Organization.
Paper context
There is limited knowledge of how to deliver effective inter-
ventions at scale in resource constrained settings. This study
describes the development, implementation, and evaluation
plan to assess the feasibility, acceptability, and effectiveness of
a package of interventions aimed at improving health worker
motivation and performance in the Democratic Republic of
the Congo. The findings will contribute to the evidence base
of how interventions can move beyond piecemeal efforts and
towards comprehensive approaches.
Sheri Bastien
Erin Ferenchick
Symplice Mbola Mbassi
Marina Plesons
Venkatraman Chandra-Mouli
[1] Newton-Levinson A, Leichliter JS, Chandra-Mouli V.
Sexually transmitted infection services for adolescents
and youth in Low- and Middle-Income Countries:
perceived and experienced barriers to accessing care.
J Adoles Health. 2016;59:7–16.
[2] Rowe AK, Rowe SY, Peters DH, et al. Effectiveness of
strategies to improve health-care provider practices in
low-income and middle-income countries: a systematic
review. Lancet Glob Heal. 2018 1;6(11):e1163–75.
[3] Dieleman M, Gerretsen B, van der Wilt GJ. Human
resource management interventions to improve health
workers‘ performance in low and middle income
countries: a realist review. Health Res Policy Syst.
[4] Bailey C, Blake C, Schriver M, et al. A systematic
review of supportive supervision as a strategy to
improve primary healthcare services in Sub-Saharan
Africa. Int J Gynecol Obstet. 2016;132(1):117–125.
[5] Chandra-Mouli V, Akwara E. Improving access to and
use of contraception by adolescents: what progress has
been made, what lessons have been learnt, and what
are the implications for action? Best Practice &
Research Clinical Obstetrics & Gynaecology.
[6] Grant C, Nawal D, Guntur SM, et al. ’We pledge to
improve the health of our entire community’: improv-
ing health worker motivation and performance in
Bihar, India through teamwork, recognition, and non-
financial incentives. PLoS One. 2018 1;13(8):e0203265.
[7] Mathauer I, Imhoff I. Health worker motivation in Africa:
the role of non-financial incentives and human resource
management tools. Human Res Health. 2006;29:4.
[8] Peters DH, Tran NT, Taghreed A. Implementation
research in health: a practical guide. World Health
Org. 2013;67.
[9] Decker MR, Latimore AD, Yasutake S, et al. Gender-
based violence against adolescent and young adult
women in low- and middle-income countries.
J Adoles Health. 2015 1;56(2):188–196.
[10] Ministère du Plan et Suivi de la Mise en oeuvre de la
Révolution de la Modernité (MPSMRM) M de la SP
(MSP) and II. Democratic Republic of Congo
Demographic and Health Survey 2013-14: key
Findings. Rockville; Maryland; 2014.
[11] UNAIDS. Country factsheets: Democratic Republic of
the Congo. Geneva: UNAIDS; 2019.
[12] Ministere de la Sante Publique. Rapport final des etats
des lieux de 15 formations sanitaires des zones de
sante de Kalamu I, Makala et Kitambo. Kinshasa:
Province de Kinshasa; 2017.
[13] Ministere de la Sante Publique. Rapport final des etats
des lieux de 15 formations sanitaires des zones de
sante de Diulu, Kansele et Nzaba. Kinshasa: Province
de Kasai Oriental; 2017.
[14] Ministere de la Sante Publique. Rapport de la forma-
tion des facilitateurs du niveau central, intermediare et
ECZS sur l’apprentissage collaboratif de la prise en
charge des adolescents dans les fosa. Kinshasa:
Ministere de la Sante Publique; 2018.
[15] Glanz K, Bishop DB. The role of behavioral science
theory in development and implementation of public
health interventions. Annu Rev Public Health.
[16] Smith K, Christakis N. Social networks and health.
Annu Rev Sociol. 2008;31(1):405–429.
[17] Coleman J, Katz E, Menzel H. Medical innovation:
a diffusion study. New York: Bobbs-Merrill; 1996.
[18] Rogers E. Diffusion of innovations. 5th ed. New York:
Free Press; 2003.
[19] May C. A rational model for assessing and evaluating
complex interventions in health care. BMC Health
Serv Res. 2006;7:6.
[20] Valente TW, Palinkas LA, Czaja S, et al. Social net-
work analysis for program implementation. PLOS
One. 2015 25;10(6):e0131712.
[21] May CR, Cummings A, Girling M, et al. Using nor-
malization process theory in feasibility studies and
process evaluations of complex healthcare interven-
tions: a systematic review. Implementation Science:
IS. 2018 7;13(1). DOI:10.1186/s13012-018-0758-1.
[22] United Nations. World Population Prospects 2019.
[23] Odimba SO, Squires F, Ferenchick E, et al.
A collaborative learning approach to improving health
worker performance in adolescent sexual and repro-
ductive health service provision: a descriptive feasibil-
ity study in six health zones in the Democratic
Republic of the Congo. Glob Health Action. 2021;14
(1). DOI:10.1080/16549716.2021.1985228.
[24] Chandra-Mouli V, Lenz C, Adebayo E, et al.
A systematic review of the use of adolescent mystery
clients in assessing the adolescent friendliness of
health services in high, middle, and low-income
countries. Glob Health Action. 2018;11(1):1536412.
[25] WHO/UNAIDS. Global standards for quality health-
care services for adolescents: a guide to implement
a standards-driven approach to improve the quality of
health-care services for adolescents. Geneva: WHO/
UNAIDS; 2015.
[26] WHO. Standards and operational guidance for ethics
review of health-related research with human partici-
pants. Geneva: WHO; 2011.
[27] WHO. Guidance on ethical considerations in plan-
ning and reviewing research studies on sexual and
reproductive health in adolescents. Geneva: WHO;
[28] WHO. Framing the health workforce agenda for the
sustainable development goals: biennium report
2016–2017 WHO health workforce. Geneva;
[29] Lesco G, Squires F, Babii V, et al. The feasibility and
acceptability of collaborative learning in improving
health worker performance on adolescent health: find-
ings from implementation research in Moldova. BMC
Health Serv Res. 2019 28;19(1). DOI:10.1186/s12913-
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Poor performance among health service providers is a key barrier to high quality, adolescent-responsive health services. Collaborative learning has been shown to strengthen health service provider performance, but few studies have examined its implementation in adolescent health services. In this paper, we describe a collaborative learning approach for adolescent health service providers implemented as part of a project aiming to prevent HIV in adolescent girls and young women in the Democratic Republic of the Congo (DRC) and explore its feasibility, acceptability, benefits and challenges. To do so, we reviewed plans, budgets and progress reports, as well as nested implementation research related to the project. We also carried out a quantitative analysis of the number, location, participants and topics of collaborative learning sessions conducted as part of this initiative, and thematic analysis to synthesise findings on perceived benefits and challenges. Under the project, 32 collaborative learning sessions of approximately two-hour duration were held with up to 15 participants, most of whom were nurses, between June 2018 and May 2019. The project cost was approximately USD 135,000 over three years. Reported benefits included improving health service provider attitudes, knowledge and skills; ensuring delivery of non-judgemental, empathic and confidential health services; and improving communication and collaboration between health services and adolescents together with their parents. While the novelty of the approach in this context presented challenges, our results suggest that collaborative learning in adolescent health services is feasible, acceptable, and inexpensive. It may help strengthen the knowledge and skills of health service providers, build positive attitudes and motivation, and improve their performance and thereby the adolescent-responsiveness of health services. Further research is needed to confirm these results in other settings and to examine the impact of collaborative learning on the acceptability and uptake of health services.
Full-text available
This paper sets out the progress that has been made in reducing levels of adolescent childbearing and in meeting adolescent contraceptive needs, over the last 25 years, and also makes the public health, economic and human rights rationale for continued attention to and investment in these areas. Using an analytic framework that covers the perspectives of both the use and the provision of contraception, it further examines the factors that make it difficult for adolescents to obtain and use contraceptives to avoid unintended pregnancies, and outlines what could be done to address these factors, drawing from research evidence and programmatic experience. In doing this, the paper provides concrete examples from low- and middle-countries that have made tangible progress in these areas.
Full-text available
Background Collaborative learning has been shown to be effective in improving health worker performance, but relatively little is known about the feasibility or acceptability of collaborative learning in youth-friendly health services (YFHS). This paper describes the characteristics, feasibility and acceptability of a collaborative learning approach implemented in YFHS in Moldova as part of a national scaling up process. Methods We gathered and analysed data on the number, location, themes, and participants of sessions, as well as benefits and challenges of collaborative learning, using two information sources: 1) formal reports on collaborative learning sessions, and 2) two questionnaires conducted with participants and moderators. Results Collaborative learning sessions have been implemented in 30 out of 35 YFHS in Moldova. In 2016, 464 collaborative learning sessions were conducted. Sessions were conducted one to three times per month, had a mean of 15 participants and an average duration of two - three hours. 74.3% of participants (n = 6942) were from rural areas and 55.1% were health professionals. The most common topics in 2016 were adolescent health and YFHS (159 of 464 sessions), sexual and reproductive health (103 sessions), and violence (76 sessions). Reported benefits for participants of collaborative learning fell into three categories: 1) improved knowledge on adolescent health / development and use of evidence-based resources; 2) strengthened teamwork and cooperation; and 3) empowerment to provide high quality, youth-friendly care. Moderators identified benefits for the quality, youth-friendliness, and positioning of YFHS as centres of excellence on adolescent health. Challenges included the time and resources required to start and maintain the program, developing a constructive multi-disciplinary learning culture, and ensuring the involvement of stakeholders from outside YFHS. Conclusion This study confirms that collaborative learning within YFHS is feasible and acceptable, and offers benefits to both participants and YFHS. Collaborative learning may be a valuable strategy to improve the quality and youth-friendliness of services. It may also be relevant to key challenges in scaling up YFHS such as increasing utilisation and achieving long-term sustainability. Further research is required to confirm our results in other settings and to examine the effects of collaborative learning at the outcome and impact level.
Full-text available
Background: Mystery client methodology is a form of participatory research that provides a unique opportunity to monitor and evaluate the performance of health care providers or health facilities from the perspective of the service user. However, there are no systematic reviews that analyse the use of mystery clients in adolescent sexual and reproductive health (ASRH) research and monitoring and evaluation of programmes. Objective: To assess the use of adolescent mystery clients in examining health care provider and facility performance in providing ASRH services in high, middle, and low-income countries. Methods: We carried out a systematic review of published journal articles and reports from the grey literature on this topic from 2000 to 2017 (inclusive). Thirty research evaluations/studies were identified and included in the analysis. We identified common themes through thematic analysis. Results: The findings reveal that researchers and evaluators used mystery client methodology to observe client-provider relationships, and to reduce observation bias, in government or private health facilities, NGOs, and pharmacies. The mystery clients in the evaluations/studies were young people who played varying roles; in most cases, they were trained for these roles. Most reported good experiences and friendly providers; however, some reported lack of privacy and confidentiality, lack of sufficient written/verbal information, and unfavourable experiences such as sexual harassment and judgmental comments. Female mystery clients were more likely than males to report unfavourable experiences. Generally, the methodology was considered useful in monitoring and evaluating the attitudes of health service providers and ASRH service provision. Conclusions: The research evaluations/studies in this review highlight the usefulness of mystery clients as a method to gain insight, from an adolescent perspective, on the quality of ASRH services for research and monitoring and evaluation of programmes.
Full-text available
Background Inadequate health-care provider performance is a major challenge to the delivery of high-quality health care in low-income and middle-income countries (LMICs). The Health Care Provider Performance Review (HCPPR) is a comprehensive systematic review of strategies to improve health-care provider performance in LMICs. Methods For this systematic review we searched 52 electronic databases for published studies and 58 document inventories for unpublished studies from the 1960s to 2016. Eligible study designs were controlled trials and interrupted time series. We only included strategy-versus-control group comparisons. We present results of improving health-care provider practice outcomes expressed as percentages (eg, percentage of patients treated correctly) or as continuous measures (eg, number of medicines prescribed per patient). Effect sizes were calculated as absolute percentage-point changes. The summary measure for each comparison was the median effect size (MES) for all primary outcomes. Strategy effectiveness was described with weighted medians of MES. This study is registered with PROSPERO, number CRD42016046154. Findings We screened 216 477 citations and selected 670 reports from 337 studies of 118 strategies. Most strategies had multiple intervention components. For professional health-care providers (generally, facility-based health workers), the effects were near zero for only implementing a technology-based strategy (median MES 1·0 percentage points, IQR −2·8 to 9·9) or only providing printed information for health-care providers (1·4 percentage points, −4·8 to 6·2). For percentage outcomes, training or supervision alone typically had moderate effects (10·3–15·9 percentage points), whereas combining training and supervision had somewhat larger effects than use of either strategy alone (18·0–18·8 percentage points). Group problem solving alone showed large improvements in percentage outcomes (28·0–37·5 percentage points), but, when the strategy definition was broadened to include group problem solving alone or other strategy components, moderate effects were more typical (12·1 percentage points). Several multifaceted strategies had large effects, but multifaceted strategies were not always more effective than simpler ones. For lay health-care providers (generally, community health workers), the effect of training alone was small (2·4 percentage points). Strategies with larger effect sizes included community support plus health-care provider training (8·2–125·0 percentage points). Contextual and methodological heterogeneity made comparisons difficult, and most strategies had low quality evidence. Interpretation The impact of strategies to improve health-care provider practices varied substantially, although some approaches were more consistently effective than others. The breadth of the HCPPR makes its results valuable to decision makers for informing the selection of strategies to improve health-care provider practices in LMICs. These results also emphasise the need for researchers to use better methods to study the effectiveness of interventions. Funding Bill & Melinda Gates Foundation, CDC Foundation.
Full-text available
Background Motivation is critical to health worker performance and work quality. In Bihar, India, frontline health workers provide essential health services for the state’s poorest citizens. Yet, there is a shortfall of motivated and skilled providers and a lack of coordination between two cadres of frontline health workers and their supervisors. CARE India developed an approach aimed at improving health workers’ performance by shifting work culture and strengthening teamwork and motivation. The intervention—“Team-Based Goals and Incentives”—supported health workers to work as teams towards collective goals and rewarded success with public recognition and non-financial incentives. Methods Thirty months after initiating the intervention, 885 health workers and 98 supervisors completed an interviewer-administered questionnaire in 38 intervention and 38 control health sub-centers in one district. The questionnaire included measures of social cohesion, teamwork attitudes, self-efficacy, job satisfaction, teamwork behaviors, equitable service delivery, taking initiative, and supervisory support. We conducted bivariate analyses to examine the impact of the intervention on these psychosocial and behavioral outcomes. Results Results show statistically significant differences across several measures between intervention and control frontline health workers, including improved teamwork (mean = 8.8 vs. 7.3), empowerment (8.5 vs. 7.4), job satisfaction (7.1 vs. 5.99) and equitable service delivery (6.7 vs. 4.99). While fewer significant differences were found for supervisors, they reported improved teamwork (8.4 vs. 5.3), and frontline health workers reported improved fulfillment of supervisory duties by their supervisors (8.9 vs. 7.6). Both frontline health workers and supervisors found public recognition and enhanced teamwork more motivating than the non-financial incentives. Conclusions The Team-Based Goals and Incentives model reinforces intrinsic motivation and supports improvements in the teamwork, motivation, and performance of health workers. It offers an approach to practitioners and governments for improving the work environment in a resource-constrained setting and where there are multiple cadres of health workers.
Full-text available
Background: Normalization Process Theory (NPT) identifies, characterises and explains key mechanisms that promote and inhibit the implementation, embedding and integration of new health techniques, technologies and other complex interventions. A large body of literature that employs NPT to inform feasibility studies and process evaluations of complex healthcare interventions has now emerged. The aims of this review were to review this literature; to identify and characterise the uses and limits of NPT in research on the implementation and integration of healthcare interventions; and to explore NPT's contribution to understanding the dynamics of these processes. Methods: A qualitative systematic review was conducted. We searched Web of Science, Scopus and Google Scholar for articles with empirical data in peer-reviewed journals that cited either key papers presenting and developing NPT, or the NPT Online Toolkit ( ). We included in the review only articles that used NPT as the primary approach to collection, analysis or reporting of data in studies of the implementation of healthcare techniques, technologies or other interventions. A structured data extraction instrument was used, and data were analysed qualitatively. Results: Searches revealed 3322 citations. We show that after eliminating 2337 duplicates and broken or junk URLs, 985 were screened as titles and abstracts. Of these, 101 were excluded because they did not fit the inclusion criteria for the review. This left 884 articles for full-text screening. Of these, 754 did not fit the inclusion criteria for the review. This left 130 papers presenting results from 108 identifiable studies to be included in the review. NPT appears to provide researchers and practitioners with a conceptual vocabulary for rigorous studies of implementation processes. It identifies, characterises and explains empirically identifiable mechanisms that motivate and shape implementation processes. Taken together, these mean that analyses using NPT can effectively assist in the explanation of the success or failure of specific implementation projects. Ten percent of papers included critiques of some aspect of NPT, with those that did mainly focusing on its terminology. However, two studies critiqued NPT emphasis on agency, and one study critiqued NPT for its normative focus. Conclusions: This review demonstrates that researchers found NPT useful and applied it across a wide range of interventions. It has been effectively used to aid intervention development and implementation planning as well as evaluating and understanding implementation processes themselves. In particular, NPT appears to have offered a valuable set of conceptual tools to aid understanding of implementation as a dynamic process.
Full-text available
Purpose: Access to sexual and reproductive health (SRH) services is vital for sexually active adolescents; yet, their SRH care needs are often unmet. Methods: We conducted a qualitative systematic review of mixed methods studies to assess adolescent and provider views of barriers to seeking appropriate medical care for sexually transmitted infection (STI) services for adolescents. We searched peer-reviewed literature for studies published between 2001 and 2014 with a study population of youth (aged 10-24 years) and/or health service providers. Nineteen studies were identified for inclusion from 15 countries. Thematic analyses identified key themes across the studies. Results: Findings suggest that youth lacked knowledge about STIs and services. In addition, youth experienced barriers related to service availability and a lack of integration of services. The most reported barriers were related to acceptability of services. Youth reported avoiding services or having confidentiality concerns based on provider demographics and some behaviors. Finally, experiences of shame and stigma were common barriers to seeking care. Conclusions: Adolescents in low- and middle-income countries experience significant barriers in obtaining STI and SRH services. Improving uptake may require efforts to address clinic systems and provider attitudes, including confidentiality issues. Moreover, addressing barriers to STI services may require addressing cultural norms related to adolescent sexuality.
Full-text available
Background: It may be assumed that supportive supervision effectively builds capacity, improves the quality of care provided by frontline health workers, and positively impacts clinical outcomes. Evidence on the role of supervision in Sub-Saharan Africa has been inconclusive, despite the critical need to maximize the workforce in low-resource settings. Objectives: To review the published literature from Sub-Saharan Africa on the effects of supportive supervision on quality of care, and health worker motivation and performance. Search strategy: A systematic review of seven databases of both qualitative and quantitative studies published in peer-reviewed journals. Selection criteria: Selected studies were based in primary healthcare settings in Sub-Saharan Africa and present primary data concerning supportive supervision. Data collection and analysis: Thematic synthesis where data from the identified studies were grouped and interpreted according to prominent themes. Main results: Supportive supervision can increase job satisfaction and health worker motivation. Evidence is mixed on whether this translates to increased clinical competence and there is little evidence of the effect on clinical outcomes. Conclusions: Results highlight the lack of sound evidence on the effects of supportive supervision owing to limitations in research design and the complexity of evaluating such interventions. The approaches required a high level of external inputs, which challenge the sustainability of such models.
Full-text available
This paper introduces the use of social network analysis theory and tools for implementation research. The social network perspective is useful for understanding, monitoring, influencing, or evaluating the implementation process when programs, policies, practices, or principles are designed and scaled up or adapted to different settings. We briefly describe common barriers to implementation success and relate them to the social networks of implementation stakeholders. We introduce a few simple measures commonly used in social network analysis and discuss how these measures can be used in program implementation. Using the four stage model of program implementation (exploration, adoption, implementation, and sustainment) proposed by Aarons and colleagues [1] and our experience in developing multi-sector partnerships involving community leaders, organizations, practitioners, and researchers, we show how network measures can be used at each stage to monitor, intervene, and improve the implementation process. Examples are provided to illustrate these concepts. We conclude with expected benefits and challenges associated with this approach.