Systems Consultation: A Novel Implementation Strategy to Promote Evidence-Based Practice in Primary Care

Abstract

Adoption of evidence-based practices takes place at a glacial place in healthcare. This research will pilot test an innovative implementation strategy – systems consultation –intended to speed the adoption of evidence-based practice in primary care. The strategy is based on tenets of systems engineering and has been extensively tested in addiction treatment. Three innovations have been included in the strategy – translation of a clinical practice guideline into a checklist-based implementation guide, the use of physician peer coaches (‘systems consultants’) to help clinics implement the guide, and a focus on reducing variation in practices across prescribers and clinics. The implementation strategy will be applied to improving opioid prescribing practices in primary care, which may help ultimately mitigate the increasing prevalence of opioid abuse and addiction.
The pilot test will compare four intervention clinics to four control clinics in a matched-pairs design. A leading clinical guideline for opioid prescribing has been translated into a checklist-based implementation guide in a systematic process that involved experts who wrote the guideline in consultation with implementation experts and primary care physicians. Two physicians with expertise in family and addiction medicine are serving as the systems consultants. Each systems consultant will guide two intervention clinics, using two site visits and follow-up communication by phone and email, to implement the translated guideline. Mixed methods will be used to test the feasibility, acceptability, and preliminary effectiveness of the implementation strategy in an evaluation that meets standards for ‘fully developed use’ of the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance). The clinic will be the primary unit of analysis.
The systems consultation implementation strategy is intended to generalize to the adoption of other clinical guidelines. This pilot test is intended to prepare for a large randomized clinical trial that will test the strategy against other implementation strategies, such as audit/feedback and academic detailing, used to close the gap between knowledge and practice. The systems consultation approach has the potential to shorten the famously long time it takes to implement evidence-based practices and clinical guidelines in healthcare.

Full text

S T U D Y P R O T O C O L Open Access
Systems consultation: protocol for a novel
implementation strategy designed to
promote evidence-based practice in
primary care
Andrew Quanbeck
1*
, Randall T Brown
2
, Aleksandra E Zgierska
2
, Roberta A Johnson
1
, James M Robinson
3
and Nora Jacobson
4
Abstract
Background: Adoption of evidence-based practices takes place at a glacial place in healthcare. This research will pilot
test an innovative implementation strategy –systems consultation –intended to speed the adoption of evidence-
based practice in primary care. The strategy is based on tenets of systems engineering and has been extensively tested
in addiction treatment. Three innovations have been included in the strategy –translation of a clinical practice
guideline into a checklist-based implementation guide, the use of physician peer coaches (‘systems consultants’) to
help clinics implement the guide, and a focus on reducing variation in practices across prescribers and clinics. The
implementation strategy will be applied to improving opioid prescribing practices in primary care, which may help
ultimately mitigate the increasing prevalence of opioid abuse and addiction.
Methods/Design: The pilot test will compare four intervention clinics to four control clinics in a matched-pairs
design. A leading clinical guideline for opioid prescribing has been translated into a checklist-based
implementation guide in a systematic process that involved experts who wrote the guideline in consultation with
implementation experts and primary care physicians. Two physicians with expertise in family and addiction
medicine are serving as the systems consultants. Each systems consultant will guide two intervention clinics,
using two site visits and follow-up communication by phone and email, to implement the translated guideline.
Mixed methods will be used to test the feasibility, acceptability, and preliminary effectiveness of the
implementation strategy in an evaluation that meets standards for ‘fully developed use’of the RE-AIM framework
(Reach, Effectiveness, Adoption, Implementation, Maintenance). The clinic will be the primary unit of analysis.
Discussion: The systems consultation implementation strategy is intended to generalize to the adoption of other
clinical guidelines. This pilot test is intended to prepare for a large randomized clinical trial that will test the
strategy against other implementation strategies, such as audit/feedback and academic detailing, used to close
the gap between knowledge and practice. The systems consultation approach has the potential to shorten the
famously long time it takes to implement evidence-based practices and clinical guidelines in healthcare.
Keywords: Clinical guideline adoption, Implementation strategies, Systems engineering
* Correspondence: andrew.quanbeck@chess.wisc.edu
1
Center for Health Enhancement Systems Studies, University of
Wisconsin-Madison, 1513 University Ave., Madison, WI 53706, USA
Full list of author information is available at the end of the article
© 2016 Quanbeck et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Quanbeck et al. Health Research Policy and Systems (2016) 14:8
DOI 10.1186/s12961-016-0079-2
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Background
Context
This project addresses the urgent need to promote the
adoption of evidence-based practices (EBPs) in healthcare
by pilot-testing an innovative implementation strategy
named the Systems Consultation Strategy (SCS). The SCS
is based on an evidence-based quality improvement
approach with roots in systems engineering. This ap-
proach, named NIATx (the Network for the Improvement
of Addiction Treatment) [1-3], has been widely tested in
addiction treatment. The SCS extends elements of the
NIATx approach to reducing variation in opioid prescrib-
ing in primary care. The proposed approach is intended to
be a generalizable approach to EBP adoption, used in this
proposal for a specific problem (opioid prescribing prac-
tices) and setting (primary care).
The standard approach to improving medical practice
includes developing and disseminating clinical guide-
lines. Developing the guidelines involves panels of ex-
perts systematically reviewing the literature, achieving
consensus, and publishing the results in a medical jour-
nal intended for the clinical audience [4]; this approach
leaves a great gap between clinical knowledge and clin-
ical practice [5-8]. Various approaches have been tried to
narrow this gap, such as providing educational materials,
audit/feedback [9], and academic detailing [10], with
mixed success; about 30–40% of patients do not receive
evidence-based care, and about 20–25% of care given is
unnecessary or potentially harmful [8, 11]. Clinicians tend
to continue to do what is comfortable, and value personal
experience and familiar practice routines over scientific
evidence [12].
Rationale for proposed pilot study
This protocol was funded under the United States
National Institute of Health’s R34 funding mechanism,
the purpose of which is to “provide support for the initial
development of a clinical trial or research project.”In
this case, the funding will support a pilot study of the
SCS to prepare for a large randomized control trial that
will test the SCS against alternative approaches to EBP
adoption in primary care. The SCS (1) teams clinical
guideline writers with implementation specialists and
primary care physicians to translate guidelines into a
checklist-based implementation guide; (2) selects, trains,
and deploys physician peer coaches (systems consul-
tants) to help primary care clinicians implement EBPs
using evidence-based tools of systems engineering; and
(3) focuses on process as a cause of variation in out-
comes. Although the SCS is rooted in established theory
and empirical research, the approach has not been for-
mally tested. This pilot test is intended to answer ques-
tions about the feasibility, acceptability, and preliminary
effectiveness of the approach by studying adaptations
that will be made to tailor the NIATx approach to pri-
mary care. In the future randomized trial, we will assess
the costs and effects of the SCS versus other implemen-
tation strategies (such as audit/feedback or academic de-
tailing) to determine the most cost-effective approach
for reducing variation in opioid prescribing practices.
Primary care was chosen as the setting because, at a
broad level, a strategy for improving the adoption of
EBPs in primary care could apply to diverse patient pop-
ulations and outcomes, and because primary care physi-
cians are the main prescribers of opioids [13]. If clinical
practice can be changed by simply informing physicians
of their opioid prescribing levels and how they compare
to peers, then audit/feedback may suffice to reduce vari-
ation in opioid prescribing. If education beyond audit/
feedback is required, nurses or other healthcare profes-
sionals may deliver it relatively inexpensively through
academic detailing visits. However, evidence from prior
quality improvement research [1] and theory on
organizational and individual change [14,15] suggests that
changing clinical practice may require a more comprehen-
sive strategy.
The prescription opioid crisis
This research focuses on a specific clinical practice in
need of change –opioid prescribing for chronic pain.
Opioid analgesics have been increasingly used to treat
chronic non-cancer pain [16], despite mounting evi-
dence of their adverse effects [17]. In the past, phys-
ician education encouraged physicians to listen to and
treat patients’pain complaints and (1) held that
addiction was rare when pain medications were taken
as prescribed, and (2) said that, if subjectively well-
tolerated, opioids did not cause end-organ damage,
and, hence, no ceiling [18] existed for dose increases
[19,20]. These two tenets have been accompanied by
alarming increases in the prescribing of opioids, along
with prescription opioid misuse, addiction, and diver-
sion. Prescribing in daily doses exceeding the equiva-
lent of 100–120 mg of morphine equivalent daily
dose (MEDD) is now associated with increased risk of
addiction and overdose [21-23].
The increase in opioid-related harms in the past two
decades has led to efforts to curb prescription opioid-
related harms [24,25]. Although examples of positive
change in opioid prescribing can be found in the litera-
ture, few (if any) approaches have been systematically
studied using experimental design as proposed in the
long-term strategy described here. The literature on
practice change in healthcare has repeatedly shown that
system-level changes occur at a glacial pace [26]. This
research seeks to promote system-level change by pilot-
testing a pioneering systems engineering approach to
improving opioid prescribing practices in primary care.
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 2 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

NIATx and NIATx 200
Since 2003, NIATx has been addressing quality issues in
addiction treatment using a model based on evidence-
based principles of organizational change [27]. Early
evaluation showed the effectiveness of the NIATx model
on reducing waiting time and increasing early engage-
ment [2,3]. More than 3,500 organizations in the spe-
cialty addiction treatment field have implemented the
NIATx model. In 2007, the National Institute on Drug
Abuse funded a cluster-randomized trial called NIATx
200 that was designed to identify the “active ingredient”
in the NIATx model. NIATx 200 (5 R01 DA020832-05)
was among the largest trials of its kind ever conducted
in healthcare. Of the approaches tested, namely interest
circle calls (group teleconferences), learning sessions
(large in-person meetings), coaching (one-on-one con-
sulting with experts), and the combination of all three,
coaching emerged as the most cost-effective for reducing
patient waiting time and increasing clinics’annual num-
ber of admissions.
Coaches in NIATx 200 helped clinics improve access
through an initial site visit and regular email and phone
communication with each clinic’s designated change
leader and other clinic staff members. Coaches helped
clinics conduct a walkthrough exercise and interpret its
results; assess workflows and identify opportunities for
improvement using systems-engineering tools such as
flowcharting and the nominal group technique [28]; and
implement changes using Plan-Do-Study-Act cycles [29].
Although individual coaching sounds expensive, it
proved more effective and substantially less expensive
than learning sessions (periodic, large face-to-face meet-
ings that are common in healthcare improvement efforts)
in the NIATx 200 study. The costs associated with con-
vening healthcare professionals from various locations
(lodging, food, mileage, etc.) add up quickly. Sending a
qualified coach to visit a clinic and to provide other con-
sulting via phone and email proved less expensive and
more effective than learning sessions. Although coaching
has been well tested in the specialty addiction field, the
approach must be adapted to fit within primary care [30].
Early feedback showed that physicians had negative asso-
ciations with the term ‘coaching’, so the term has been re-
placed by ‘consulting', the more familiar term to
physicians.
Evidence gap/opportunity
This proposal introduces an implementation strategy dis-
tinguished by several innovations: (1) a systematic process
for clinical guideline translation; (2) a peer-to-peer consult-
ing intervention that integrates relevant theory with empir-
ical research on organizational change [1,14,15]; and (3) an
explicit focus on process as a cause of variation in out-
comes [29]. The SCS consists of a series of generalizable
steps, including setting the larger context (patient safety);
teaming clinical guideline writers with implementation spe-
cialists and clinical practitioners to distill the essence of the
guideline into a succinct, checklist-based implementation
guide; providing an outside systems consultant (i.e. a coach)
with clinical expertise and experience; actively involving
clinical staff (including primary care physicians, nurses,
and physician assistants) in the implementation
process; giving participants the ability to customize in-
terventions (rather than using a one-size-fits-all ap-
proach); and providing tools, such as the walkthrough
exercise, Plan-Do-Study-Act change cycles, the nominal
group technique, and flowcharting, that promote
problem-solving and rapid, incremental improvements.
Theoretical foundations
The SCS rests on established theories of organizational
change from systems engineering [29,31] and diffusion
of innovations [14], as well as recent empirical research
on quality improvement [1]. Deming established a num-
ber of canonical systems engineering principles in his in-
fluential book, Out of the Crisis [29]. Of these principles,
the following are most important for the current proto-
col: (1) quality problems are almost always caused by
poor processes and systems, and thus lie outside the
purview of individuals to change; improving quality re-
quires systemic approaches rather than singling out indi-
viduals, and (2) measuring and understanding process
variation is critical to improving quality. The goal of the
SCS is to reduce variability in opioid prescribing
practices.
Even though a clinical guideline for opioid prescribing
has been in place for several years [32], most clinicians
do not make adoption decisions based solely on scien-
tific evidence. Rogers’s work on diffusion of innovations
[14] provides a theoretical framework for understanding
why some ideas diffuse throughout systems while others
do not. Although characteristics of ideas are important
in diffusion, social factors largely supersede them. People
(and organizations) are more likely to adopt an idea if
they know of others similar to themselves who have
done so already. Diffusion of innovations theory stresses
homophily between change agents and those whose
behaviour they seek to influence, that is, the similarity
between individuals on characteristics such as education
and social status, suggesting that consulting intended to
affect physician prescribing practices should be done by
consultants who are physicians.
The idea of systems consulting rests on self-determination
theory [15]. The theory holds that satisfying three funda-
mental needs (competence, relatedness, and autonomy)
contributes to individual high functioning. Conditions that
foster meeting these needs lead individuals to move from
extrinsic to intrinsic motivation –achangethat’s
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 3 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

necessary for organizational improvement efforts to suc-
ceed. According to self-determination theory, extrinsic
motivation encourages people to regard the cause of
their actions as being outside themselves rather than
thinking of themselves as agents of change. Extrinsic
motivators (e.g. tangible rewards, deadlines, threats, di-
rectives, imposed goals) have all been shown to reduce
intrinsic motivation [15]. In terms of opioid prescribing,
an extrinsic motivator for physicians would be the threat
of additional regulation. Self-determination theory posits
that behaviour can become self-determined, or intrinsic-
ally motivated, if actions are modelled or valued by others
to whom an individual feels related or attached. People
are more likely to adopt actions when they feel competent
doing the activities, and to adopt changes that relate to
their goals and values.
Contribution of this research
The SCS offers two innovations to promote the adoption
of EBPs. First, panels of experts are commonly convened
to reach consensus and develop guidelines for EBPs, as
was done for opioid prescribing [32]. This proposal adds
a novel step by bringing together guideline writers, im-
plementation experts, and primary care physicians to
translate the guideline into a checklist-based format that
can be implemented more easily than a guideline
appearing in an academic journal. The experts convened
for this proposal include pain physicians from the panel
that developed a leading opioid prescribing guideline;
internationally recognized experts on healthcare quality
improvement and drug policy; and community-based fam-
ily medicine physicians (see Acknowledgments). Collect-
ively, the advisory panel monitors and advises the research
team throughout the pilot test of the implementation
strategy.
Second, peer-to-peer physician consulting is a corner-
stone of the SCS. While many efforts have been made to
increase the use of EBPs among physicians, a literature
review revealed a surprising dearth of research on
models in which physicians consulted other physicians
in primary care. Medical conferences have been used to
promote new practices, but they are generally not effect-
ive [33]. Improvement collaboratives are a common
method in healthcare; these involve clinical teams
(which often include physicians and other clinicians)
learning with and from one another [34]. More targeted
types of physician education (such as coaching, facilita-
tion, and academic detailing) often are conducted by
health professionals such as nurses, physician assistants,
or others with master’s degrees in business administra-
tion, public health, counselling, public administration, or
social work [9,35-37]. In our literature review, the Physi-
cians’Clinical Support System −Buprenorphine came
close to offering a formal physician-to-physician
consulting model [38], though it did not include on-site
visits and implementation support, and little program
evaluation took place beyond statistics on the frequency
and type of contact with mentors. Dartmouth Medical
School developed a consulting model for medical stu-
dents in which they received guidance and career devel-
opment advice from senior faculty members [39], a
model that took place only in an academic setting and
did not target clinical practice.
Methods/Design
Preliminary studies
In preparation for the grant submission, we analyzed
preliminary data from the set of family medicine clinics
that form the recruitment base for this pilot study. Sub-
stantial differences in opioid prescribing practices were
evident. Clinics, and providers within clinics, varied sub-
stantially in the percentage of patients with three or
more opioid prescriptions in the past year and average
MEDD per patient. Compared with other patients, those
with three or more opioid prescriptions more frequently
reported smoking and drinking, were diagnosed more
often with depression, and had substantially more clinic
visits. Those in the highest MEDD group (4th quartile)
had the least favourable profile. Of note, close to 50% of
those in the 4th MEDD quartile reported regular alcohol
consumption, which is discouraged in treatment agree-
ments. These preliminary results align with evidence on
the dose–response relation between MEDD and poor
outcomes, [40] and further suggest that implementing
universal precautions for opioid prescribing can reduce
harms.
Specific Aim 1 consists of guideline translation and
training of the systems consultants. Guideline translation
has been conducted by the research team in consultation
with the advisory panel. We followed a structured group
decision-making approach called the integrative group
process [41], a systematic technique for facilitating meet-
ings of experts that incorporates the nominal group
technique [28], the Delphi process [42], social judgment
analysis [43], and cognitive mapping [44]. We began by
conducting a structured Delphi process [28], in which
we asked each member of the advisory panel to rate
each recommendation in the opioid prescribing clinical
guideline on its measurability, potential to reduce opioid
abuse, and ease of implementation. We conducted follow-
up telephone interviews with each panel member to
understand the ratings they assigned. We compiled the re-
sults of the Delphi process to prioritize the panelists’rec-
ommendations into a preliminary checklist.
Checklists can be effective in improving healthcare
processes and patient safety [45] by serving at least two
purposes: (1) jogging the clinician’s memory and
(2) "making the minimum explicit" [46]. But a checklist
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 4 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

per se is a weak intervention [47] because it does not ad-
dress barriers specific to the setting in which it is to be
used. A checklist can be effective as part of an interven-
tion that creates social connections among clinicians
[26, 47], but not when it functions as the whole inter-
vention. For this reason, our implementation strategy
will include not just a checklist, but implementation
tools and support.
We convened the advisory panel for a one-day, in-
person meeting on November 18, 2014. During the
meeting, the authors presented the initial checklist and
asked panel members to provide feedback and revisions.
We presented a variety of archetypal patient cases (for
instance, a patient complaining of pain for the first time
versus a patient already taking opioids) and asked panel
members to discuss how the checklist might need to be
adapted to fit different circumstances. The advisory
panel has addressed questions such as: How does the
checklist need to be adapted for local context (which re-
search suggests is vital to successful implementation)
[47]? Who needs to be involved in the implementation
process (e.g. clinic medical directors, nurses, administra-
tive staff)?
Teaming panelists who wrote the clinical guideline for
opioid prescribing [32] with implementation experts and
primary care physicians was designed to produce a clear
picture of how implementation should happen. The im-
plementation guide consists of a checklist containing the
essential elements of the guideline and an implementation
approach/timetable designed to guide local customization
of the checklist.
The two systems consultants for this pilot project are
both physician faculty members in the University of
Wisconsin’s Department of Family Medicine and Com-
munity Health (DFMCH) with current clinical practices.
Both are board-certified in family medicine and addic-
tion medicine. They have clinical experience with opioid
therapy management in accordance with opioid pre-
scribing guidelines, and have first-hand experience of in-
tegrating elements of the guideline into clinical
workflows (such as treatment agreements and random
drug testing).
The systems consultants have received training in
systems-engineering fundamentals in sessions taught by
past leaders of the NIATx initiative (on June 30 and
November 6, 2015). These training sessions have included
topics such as fundamentals of quality improvement, what
makes a good consultant, understanding the walk-through
exercise, creating and facilitating change teams, using tools
such as flowcharting and the nominal group technique, and
using Plan-Do-Study-Act cycles.
Specific Aim 2 focuses on field-testing the SCS imple-
mentation strategy. The systems consultants’approach
to interacting with clinics is modelled after the coaching
protocol used in NIATx 200, which included in-person
site visits and regular follow-up through phone and
email communication. Participating clinics will be paired
with a systems consultant to work through the SCS over
a 6-month period. Each clinic will designate one clin-
ician to act as a clinic leader in working with the systems
consultant. The clinic leader will host two site visits by
the systems consultant and communicate with the con-
sultant monthly during the 6-month follow-up consult-
ing period via phone and email. The clinic leader will
work with a clinic’s“change team”—a group of 2 to 6
other staff members involved in opioid prescribing to
identify and implement systems changes to improve opi-
oid prescribing processes. The consultant’s initial site
visit will last about 2 hours. The consultant will start by
presenting the latest research on balancing the benefits
and risks of long-term opioid use. The visit will include
reviewing information gathered by the change leader be-
fore the site visit as a result of conducting a walk-
through exercise, in which the change leader follows the
clinic process for refilling an opioid prescription for a
current patient, step-by-step. The visit will also help the
change team determine the best course for implement-
ing aspects of the checklist, depending on the clinic’s
own workflow. The systems consultant will help
synthesize baseline information from the walk-through
and other activities and facilitate a brainstorming session
with the change team using nominal group technique
[28]. The systems consultant and change team will envi-
sion how to implement the checklist, assessing any sys-
temic barriers to implementation. The systems
consultant and clinic leader will debrief to develop an
implementation plan and schedule a second site visit to
assess progress. Throughout the intervention period, the
systems consultant will help the team implement ideas
for change using Plan-Do-Study-Act change cycles [29]
and maintain monthly email and phone contact with the
clinic leader and change team to monitor implementa-
tion progress and offer advice. The systems consultant
will also be available to discuss patient care issues (e.g.
difficult cases) during monthly phone conferences, with
all identifiable patient data removed beforehand.
Clinic recruitment
Recruitment will focus on primary care clinics that are
part of the DFMCH (n = 20). Clinics offering resident
training will be excluded (n = 6); one clinic will be ex-
cluded from consideration because one of the systems
consultants has an active clinical practice there. We se-
lected the remaining 13 clinics as a recruitment pool to
enable systematic monitoring of opioid prescribing rates
and other clinical data through the clinical data ware-
house housed by the DFMCH. The 13 clinics were first
grouped into two categories (community vs. regional)
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 5 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

and then ranked by the number of patients with long-
term opioid prescriptions (defined as 10+ orders in the
previous 12 months). Within these two categories, the
clinics were paired by “nearest-neighbour”matching
based on number of patients with 10+ opioid prescrip-
tions. One pair of clinics will be recruited from the re-
gional group (2/4 clinics eligible) and three pairs will be
recruited from the community group (6/9 clinics eligible).
Within each pairing, one clinic will be invited to be the
intervention clinic by random selection. If that clinic
agrees to participate (via a discussion between the systems
consultant and the clinic’s medical director), the second
clinic in the pairing will be assigned to the control condi-
tion. If the first clinic declines to participate, the second
clinic in the pair will be invited to be the intervention
clinic. If the second clinic agrees to participate, it will be
assigned to be the intervention clinic and the first clinic
will be assigned to control. If both clinics decline to par-
ticipate, an alternate clinic will be selected within each cat-
egory (there are three alternates among the community
clinics and two alternate clinics in the regional clinics)
until one of the alternate clinics agrees to participate.
Specific Aim 3 uses mixed methods to understand
how the strategy worked, including assessments of feasi-
bility and acceptability (costs, ease of recruitment, fidel-
ity to the protocol, physician acceptance) and
preliminary effectiveness (degree of checklist implemen-
tation, effect of the strategy on variation in opioid pre-
scribing rates and dosing levels). While multiple
prescribers (primary care physicians, nurse practitioners,
physician assistants) potentially practice within each
clinic, the clinic will serve as the primary unit of ana-
lysis. Relevant comparisons for study measures will be
made in two ways. First, historical data are available for
many study measures, permitting time-series analysis of
repeated measures to detect changes in a clinic over
time (pre-intervention vs. post-intervention). Second, we
can compare intervention and control clinics per the
matched-pairs design.
Measures
This proposal uses the RE-AIM model as an organizing
evaluation framework [48] to examine the quality, speed
and impact of implementing the SCS in primary care set-
tings. RE-AIM assesses implementation in five dimen-
sions: Reach, Effectiveness, Adoption, Implementation,
and Maintenance. This evaluation seeks to meet the
standards for ‘fully developed use’of RE-AIM [49] by
employing the RE-AIM checklist available at www.re-
aim.org. We have, however, chosen to omit one RE-
AIM dimension (maintenance at the setting level) be-
cause assessing this dimension would require follow-up
6 months after project funding ends. Specific measures
for each RE-AIM dimension are presented in Table 1.
Quantitative data collection and analysis
We will access many of our RE-AIM measures by tap-
ping into clinics’electronic health records via a data
warehouse maintained in the DFMCH (e.g. reach data
such as number and percentage of patients excluded, ef-
fectiveness and maintenance data such as overall rate of
opioid prescribing by provider, etc.). The representative-
ness of the sample (patients, clinics, and staff ) will be
accessed through administrative databases maintained
by the DFMCH. The quantitative analysis will focus pri-
marily on patient-level opioid prescription rates mea-
sured in MEDD. Changes in outcomes will be assessed
through repeated monthly observations. Data on pre-
scribing will be augmented by other process and out-
come measures shown in Table 1.
Statistical model
To isolate and measure the intervention effect on each
measure of interest, we will fit a mixed-effects model to
the data. The model will contain a fixed effect for a
shared common linear trend; we will test the sensitivity
of the results to other non-linear trends. Fixed effects
will be included for the impact of the intervention on
the measure of interest. Since the intervention activities
will be skewed toward the beginning of the intervention
period, we will model an increasing cumulative effect
that allows the rate of increase to change during the
period. That is, we will use a piecewise linear function of
the intervention duration with ‘knots’at the start and
midway through the 12-month period. At the end of the
intervention period, we will allow for a second linear
progression to capture any continuing effect or any re-
gression back to pre-intervention response levels. Other
fixed effects will be included for observed characteristics
of providers that may have a significant impact on the
response variable (e.g. patient/physician ratio). Random
effects will be included to allow for correlation among
repeated observations within the same clinic, provider,
or patient. Auto-correlated model error terms will be
included to allow for additional correlation among
observations from the same patient in adjacent
months. Appropriate transformations of the response
variable (e.g. logarithms, square roots, etc.) will be
considered to avoid negative fitted values and to bet-
ter match the frequency of outlying values.
Cost analysis
Methods and instruments used for cost data collection
in the NIATx 200 study [1] will be adapted to assess the
costs of administering the SCS. Systems consultants will
keep detailed logs of contacts with clinics (based on an
online tracking system developed for NIATx 200) to as-
sess staff participation and fidelity to the protocol. We
will estimate the cost of the intervention by assessing
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 6 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

time spent by systems consultants and clinicians during
the implementation phase and multiplying these esti-
mates by appropriate wage rates based on averages avail-
able through the Wisconsin Department of Workforce
Development. We will add any relevant non-personnel
costs, such as travel to site visits, the cost of teleconfer-
encing services, etc.
Qualitative data collection and analysis
We will follow NIH guidelines for mixed-methods
inquiry [50] to complement quantitative assessments of
the feasibility, acceptability, quality, and preliminary ef-
fectiveness of the SCS. During the intervention, we will
use a variety of qualitative methods, including document
review, activity logs, debriefing conversations with the
system consultants, and observations of interactions be-
tween the systems consultants and the clinic change
teams. These methods are designed to help us identify
the incentives, scheduling, delivery methods, and other
processes and structures that will make the SCS useful
for participating clinicians and manageable for the sys-
tems consultants. Post-intervention focus groups will be
used to compare the experience of clinics that changed
substantially versus those that did not. We will explore
questions such as (1) What kinds of process changes
were associated with improvement? (2) What factors
helped providers and clinics make changes? (3) What
were the barriers to improvement, and how were they
addressed? (4) When the intervention did not work well,
what was different?
Table 1 RE-AIM measures
Domain Measure
Reach Number and percentage of patients excluded
Reach Number and percentage of patients served by eligible clinics
Reach Characteristics of participating patients versus general patient population
Reach Structured interview with Family Medicine director to qualitatively assess recruitment process
Effectiveness Number and percentage of patients completing urine drug screens
Effectiveness Overall rate of opioid prescribing by clinic and provider
Effectiveness Number and percentage of patients screened for mental health/substance use problems
Effectiveness Overall rate of opioid/benzodiazepine co-prescribing
Effectiveness Number and percentage of patients signing pain agreements
Effectiveness Number and percentage of opioid prescriptions above 120 mg daily morphine equivalent
Effectiveness Number and percentage of providers who drop out of study at 3 months
Effectiveness Structured interview with clinic lead to assess satisfaction, effectiveness, and subgroup differences
Adoption (Setting) Number and percentage of clinics excluded
Adoption (Setting) Number and percentage of clinics that participate
Adoption (Setting) Characteristics of participating clinics versus non-participants
Adoption (Staff) Number and percentage of staff excluded
Adoption (Staff) Number and percentage of staff who participate
Adoption (Staff) Characteristics of participating staff versus non-participants
Implementation Hours of consulting delivered/received per provider
Implementation Adaptations made to consulting protocol during intervention period
Implementation Cost of consulting intervention
Implementation Structured interview with clinic lead to assess consistency of consulting intervention
Maintenance (Indiv.) Number and percentage of patients completing urine drug screens (6-month follow-up)
Maintenance (Indiv.) Overall rate of opioid prescribing by clinic and provider (6-month follow-up)
Maintenance (Indiv.) Number and percentage of patients screened for mental health/substance use problems (6-month follow-up)
Maintenance (Indiv.) Overall rate of opioid/benzodiazepine co-prescribing (6-month follow-up)
Maintenance (Indiv.) Number and percentage of patients signing pain agreements (6-month follow-up)
Maintenance (Indiv.) Number and percentage of opioid prescriptions above 120 mg daily morphine equivalent (6-month follow-up)
Maintenance (Indiv.) Number and percentage of providers who drop out of study (6-month follow-up)
Maintenance (Indiv.) Focus group with clinicians who made substantial changes
Source:Re-aim.org; Measuring the Use of the RE-AIM Model Dimension Items Checklist.
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 7 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

We will apply a combination of directed and traditional
content analysis [51] to answer the questions posed above.
Our analysis will identify themes in the qualitative data
and organize them into a conceptual model that incorpo-
rates a priori elements of self-determination theory (e.g.
relatedness, competence, autonomy) and diffusion of in-
novations theory (e.g. homophily), while simultaneously
seeking new insights inductively.
Our second objective in this mixed methods compo-
nent of our research is to assess fidelity to the interven-
tion, defined as: (1) amount of the intervention received
(i.e. ‘dose’), (2) adherence to the protocol, and (3) quality
of intervention delivery [52]. Assessing the dose of inter-
vention received will rely on quantitative data (the num-
ber of intervention hours delivered to clinic staff )
obtained through logs kept by the systems consultants.
For adherence, we will review the planned protocol with
clinicians and document adaptations made to it at each
site. To assess quality, we will examine both quantitative
data on clinic-wide practices and focus group data on
changes in clinicians’prescribing attitudes and actions.
Milestones and products of pilot test
The primary goal of this project is to obtain pilot data for
a large-scale clinical trial of different approaches to EBP
adoption in healthcare. The research will be critical in de-
veloping (1) specifics of the guideline to be implemented,
and (2) supports provided via the SCS implementation
strategy and the systems consultants who deliver it. The
protocol will be determined to be feasible if we enroll four
intervention clinics, deliver the intended intervention in
all four clinics, follow the clinics for a 6-month period,
and obtain quantitative and qualitative data from partici-
pating clinicians. Acceptability will be gauged by assessing
fidelity to the protocol and analysing qualitative data col-
lected by the research team. We will continue to track ef-
fectiveness measures through the end of the 3-year grant
period using de-identified electronic health records (e.g.
opioid prescribing rate by clinician, number of treatment
agreements signed) to assess the preliminary effectiveness
of the SCS on clinical practice. With eight clinics report-
ing data (four intervention and four control clinics), we
will be able to obtain baseline estimates of means and
standard deviations for RE-AIM outcomes and point esti-
mates of treatment effect sizes, thereby equipping us to
conduct an accurate power calculation for the follow-up
randomized trial.
Potential problems and alternative strategies
We are confident we can recruit four intervention clinics
from our recruitment pool based on successful prior col-
laboration between the University of Wisconsin Depart-
ments of Industrial & Systems Engineering and DFMCH.
If we are unable to recruit four intervention clinics from
the 13 family medicine clinics we are targeting, we will ex-
tend our recruitment efforts to the six DFMCH commu-
nity clinics that train residents (and exclude from analysis
clinicians who are residents).
Prescribing opioids for chronic non-cancer pain is com-
plex and controversial. The potential exists for physicians
to be reluctant to participate in the study because they
fear having their opioid prescribing practices scrutinized.
We will employ several strategies to address this: (1) the
primary unit of analysis will be the clinic; (2) individual
prescribers will not be identified in study databases, and
every precaution will be taken to maintain confidentiality;
(3) the systems consultant will emphasize that aspects of
pain management lack an adequate evidence base, and
that the purpose of the study (and the larger trial) will be
to provide the evidence needed to improve opioid pre-
scribing practices; and (4) the presence of national experts
involved in the research may motivate physicians and
clinics to participate.
Limitations
As pilot implementation research, we are focused on
assessing the feasibility, acceptability, and preliminary ef-
fectiveness of the implementation strategy and not long-
term patient-level outcomes. Secular trends in opioid
prescribing could make it difficult to gauge the effective-
ness of the SCS (i.e. as public awareness of the opioid
crisis increases, rates of prescribing may fall). We will
partially control for this limitation by comparing trends
in intervention versus control clinics.
Future directions
As outlined earlier, we intend to conduct a large-scale
RCT to test the costs and effects of different approaches
to EBP adoption in healthcare, including audit/feedback,
academic detailing, and the SCS. We expect that some
of the physicians who participate in this pilot research
might themselves become systems consultants; indeed, 9
of the 17 coaches in the NIATx 200 study were former
change leaders in addiction treatment clinics who went
on to coach their peers.
Current status
When this manuscript was submitted, the research team
was transitioning from Study Aim 1 (guideline translation
and consultant training) to Study Aim 2 (field testing the
implementation strategy). Results will be published as they
become available.
Ethics approval
The study protocol has been designated minimal risk
and approved by the University of Wisconsin’s Health
Sciences Institutional Review Board (2015-0280-CP003).
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 8 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Discussion
Given that this is pilot research and that implementation
is always dynamic, elements of the research design re-
ported in this paper will undoubtedly change after it is
published. One illustration of this dynamic pertains to
the naming of the implementation strategy. In the grant
proposal, we named the strategy “NIATx-VOP”(i.e. Net-
work for the Improvement of Addiction Treatment –
Variation in Opioid Prescribing). However, initial discus-
sions with community-based primary care physicians
highlighted that the name ‘NIATx’has been primarily
identified with quality improvement in specialty addic-
tion treatment and holds little currency in primary care.
Further, primary care physicians told us that “doctors
don’t like to be coached.”In response, we have tenta-
tively changed the name of the implementation strategy
from ‘NIATx-VOP’to ‘systems consulting strategy’and
the word ‘coach’to ‘systems consultant’. As pointed out
by Proctor et al. [53], naming implementation strategies
is an important aspect of specifying them; thus, it may
be that the name assigned to the strategy will change
again based on feedback gathered during our fieldwork.
Conclusion
Developing an effective strategy for implementing
evidence-based practices for opioid prescribing in primary
care is an urgent matter, given that primary care is the ori-
gin of most opioid prescriptions, the rate at which patients
taking opioids for chronic pain become addicted, and the
increasing number of lives compromised or ended by opi-
oid abuse and addiction. The novel implementation strategy
proposed here (guideline translation and peer consulting
using tools of systems engineering) could readily be gener-
alized to other EBPs, and holds the potential to exert a
powerful and sustained impact not only on opioid prescrib-
ing but perhaps on the implementation of EBPs throughout
healthcare. Such a strategy could potentially be used for
other evidence-based practices, shortening the famously
long time it takes for such practices to be implemented in
healthcare.
Competing interests
Authors Quanbeck and Johnson have a shareholder interest in CHESS Mobile
Health, a small business that develops web-based healthcare technology for
patients and family members. This relationship is extensively managed by the
authors and the University of Wisconsin. Dr. Quanbeck provides consulting
through the NIATx Foundation, a non-profit organization that offers training in
quality improvement. Dr. Zgierska has funding from Pfizer to conduct studies
related to opioid prescribing practices. All the other authors declare no compet-
ing interests.
Authors’contributions
AQ drafted the original manuscript. AQ and RB designed the study. AZ, JR,
and NJ contributed to the design of the study. RJ performed critical revisions
to the manuscript. All authors read, contributed to, and approved the final
manuscript.
Acknowledgements
The National Institute on Drug Abuse (NIDA) is the primary funder of the
study (R34-DA-036720-01). The funder has no role in study design, the
interpretation of data, or the publication of results. Additional support is
provided by grants from the National Institute on Drug Abuse (R01–DA–
034279–01, R01–DA–030431–01, and K01-DA-039336-01). The authors wish
to thank the distinguished experts who took part in translating the clinical
guideline for opioid prescribing for use in this study, including, from pain man-
agement, the experts who developed the guideline for opioid prescribing, Jane
Ballantyne, MD, Roger Chou, MD, and Perry Fine, MD; from healthcare quality
improvement and implementation, David H. Gustafson, PhD, Dennis McCarty,
PhD, and Paul Batalden, MD; and from community-based family medicine, Jo-
nas Lee, MD, Beth Potter, MD, and John Frey, MD. The authors also wish to
thank the expert systems consultant, Lynn M. Madden, who is advising the sys-
tems consultants in the study, and the dedicated staff who helped plan and
conduct the study, Brienna M. Deyo, Wen-Jen Tuan, Esra Alagoz, Ellyn Klaila, and
Judith Ganch.
Author details
1
Center for Health Enhancement Systems Studies, University of
Wisconsin-Madison, 1513 University Ave., Madison, WI 53706, USA.
2
School of
Medicine and Public Health, University of Wisconsin-Madison, Room 3832,
1100 Delaplaine Ct, Madison, WI 53715, USA.
3
Center for Health Systems
Research & Analysis, University of Wisconsin-Madison, 1109c Warf Office
Building, 610 Walnut St., Madison, WI 53726, USA.
4
School of Nursing,
University of Wisconsin-Madison, 5130 Cooper Hall, Signe Skott; 701
Highland Ave, Madison, WI 53705, USA.
Received: 6 January 2016 Accepted: 8 January 2016
References
1. Gustafson DH, Quanbeck AR, Robinson JM, Ford 2nd JH, Pulvermacher A,
French MT, et al. Which elements of improvement collaboratives are most
effective? A cluster-randomized trial. Addiction. 2013;108(6):1145–57.
2. McCarty D, Gustafson DH, Wisdom JP, Ford J, Choi D, Molfenter T, et al. The
Network for the Improvement of Addiction Treatment (NIATx): enhancing
access and retention. Drug Alcohol Depend. 2007;88(2–3):138–45.
3. Hoffman KA, Ford 2nd JH, Choi D, Gustafson DH, McCarty D. Replication and
sustainability of improved access and retention within the network for the
improvement of addiction treatment. Drug Alcohol Depend. 2008;98(1–2):63–
9.
4. Turner T, Misso M, Harris C, Green S. Development of evidence-based
clinical practice guidelines (CPGs): comparing approaches. Implement Sci.
2008;3:45.
5. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge
dissemination, utilization, and integration in public health. Annu Rev Public
Health. 2009;30:151–74.
6. Grimshaw J, Eccles M, Tetroe J. Implementing clinical guidelines: current
evidence and future implications. J Contin Educ Health Prof. 2004;24
Suppl 1:S31–7.
7. Haines A, Jones R. Implementing findings of research. BMJ. 1994;308(6942):
1488–92.
8. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care
in the United States? 1998. Milbank Q. 2005;83(4):843–95.
9. Grol R, Grimshaw J. From best evidence to best practice: effective
implementation of change in patients’care. Lancet. 2003;362(9391):1225–30.
10. van Eijk ME, Avorn J, Porsius AJ, de Boer A. Reducing prescribing of highly
anticholinergic antidepressants for elderly people: randomised trial of group
versus individual academic detailing. BMJ. 2001;322(7287):654–7.
11. Gr ol R. Improving the quality of medical care: building bridges among
professional pride, payer profit, and patient satisfaction. JAMA. 2001;
286(20):2578–85.
12. Baker TB, McFall RM, Shoham V. Current status and future prospects of
clinical psychology: toward a scientifically principled approach to mental
and behavioral health care. Psychol Sci Public Interest. 2008;9(2):67–103.
13. Volkow ND, McLellan TA, Cotto JH, Karithanom M, Weiss SR. Characteristics
of opioid prescriptions in 2009. JAMA. 2011;305(13):1299–301.
14. Rogers EM. Diffusion of innovations. 4th ed. New York, NY: Free Press; 1995.
15. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. Am Psychol. 2000;55(1):68–78.
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 9 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

16. Boudreau D, Von Korff M, Rutter CM, Saunders K, Ray GT, Sullivan MD, et al.
Trends in long-term opioid therapy for chronic non-cancer pain.
Pharmacoepidemiol Drug Saf. 2009;18(12):1166–75.
17. Kronick R. AHRQ data reveal wider impact of opioid overuse. AHRQ Views.
Rockville, MD: Agency for Healthcare Research and Quality; 2014.
18. Portenoy RK. Opioid therapy for chronic nonmalignant pain: a review of the
critical issues. J Pain Symptom Manage. 1996;11(4):203–17.
19. Morgan JP. American opiophobia: customary underutilization of opioid
analgesics. Adv Alcohol Subst Abuse. 1985;5(1–2):163–73.
20. Bennett DS, Carr DB. Opiophobia as a barrier to the treatment of pain. J
Pain Palliat Care Pharmacother. 2002;16(1):105–9.
21. Edlund MJ, Martin BC, Fan MY, Devries A, Braden JB, Sullivan MD. Risks for
opioid abuse and dependence among recipients of chronic opioid therapy:
results from the TROUP study. Drug Alcohol Depend. 2010;112(1–2):90–8.
22. Bohnert AS, Valenstein M, Bair MJ, Ganoczy D, McCarthy JF, Ilgen MA, et al.
Association between opioid prescribing patterns and opioid overdose-related
deaths. JAMA. 2011;305(13):1315–21.
23. Dunn KM, Saunders KW, Rutter CM, Banta-Green CJ, Merrill JO, Sullivan MD,
et al. Opioid prescriptions for chronic pain and overdose: a cohort study.
Ann Intern Med. 2010;152(2):85–92.
24. Trescott CE, Beck RM, Seelig MD, Von Korff MR. Group Health's initiative to
avert opioid misuse and overdose among patients with chronic noncancer
pain. Health Aff (Millwood). 2011;30(8):1420–4.
25. McLellan AT, Turner BJ. Chronic noncancer pain management and
opioid overdose: time to change prescribing practices. Ann Intern Med.
2010;152(2):123–4.
26. Berwick DM. Disseminating innovations in health care. JAMA. 2003;
289(15):1969–75.
27. Gustafson DH, Hundt AS. Findings of innovation research applied to
quality management principles for health care. Health Care Manage
Rev. 1995;20(2):16–33.
28. Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program
planning: a guide to nominal group and Delphi processes. Glenview, IL:
Scott, Foresman; 1975.
29. Deming WE. Out of the crisis. Cambridge, MA: Massachusetts Institute of
Technology, Center for Advanced Engineering Study; 1986.
30. Kaplan HC, Brady PW, Dritz MC, Hooper DK, Linam WM, Froehle CM, et al.
The influence of context on quality improvement success in health care: a
systematic review of the literature. Milbank Q. 2010;88(4):500–59.
31. Berwick DM. Controlling variation in health care: a consultation from Walter
Shewhart. Med Care. 1991;29(12):1212–25.
32. Chou R, Fanciullo GJ, Fine PG, Adler JA, Ballantyne JC, Davies P, et al. Clinical
guidelines for the use of chronic opioid therapy in chronic noncancer pain.
J Pain. 2009;10(2):113–30.
33. Ioannidis JP. Are medical conferences useful? And for whom? JAMA. 2012;
307(12):1257–8.
34. Institute for Healthcare Improvement. The breakthrough series: IHI’s
collaborative model for achieving breakthrough improvement. Diabetes
Spectr. 2004;17:2.
35. Knox L, Taylor EF, Geonnotti K, Machta R, Kim J, Nysenbaum J, et al.
Developing and running a primary care practice facilitation program: a
how-to guide. Rockville, MD: Agency for Healthcare Research and Quality;
2011.
36. Grumbach K, Bainbridge E, Bodenheimer T. Facilitating improvement in
primary care: the promise of practice coaching. Issue Brief (Commonw
Fund). 2012;15:1–14.
37. Nagykaldi Z, Mold JW, Aspy CB. Practice facilitators: a review of the
literature. Fam Med. 2005;37(8):581–8.
38. Egan JE, Casadonte P, Gartenmann T, Martin J, McCance-Katz EF, Netherland
J, et al. The Physician Clinical Support System-Buprenorphine (PCSS-B): a
novel project to expand/improve buprenorphine treatment. J Gen Intern
Med. 2010;25(9):936–41.
39. Homa K, Regan-Smith M, Foster T, Nelson EC, Liu S, Kirkland KB, et al.
Coaching physicians in training to lead improvement in clinical
microsystems: a qualitative study on the role of the clinical coach. Int J Clin
Leader. 2008;16(1):37–48.
40. Ramond A, Bouton C, Richard I, Roquelaure Y, Baufreton C, Legrand E, et al.
Psychosocial risk factors for chronic low back pain in primary care–a
systematic review. Fam Pract. 2011;28(1):12–21.
41. Gustafson DH, Cats-Baril WL, Alemi F. Systems to support health policy
analysis: theory, models and uses. Ann Arbor, MI: Health Administration
Press; 1992.
42. Dalkey N, Helmer O. An experimental application of the Delphi method to
the use of experts. Manage Sci. 1963;9(3):458–67.
43. Rohrbaugh J. Improving the quality of group judgment: social
judgment analysis and the Delphi technique. Organ Behav Hum
Perform. 1979;24(1):73–92.
44. Eden C, Jones S, Sims D. Messing about in problems: an informal structured
approach to their identification and management. New York: Pergamon
Press; 1983.
45. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical
checklists for improved quality of patient care. Int J Qual Health Care. 2008;
20(1):22–30.
46. Gawande A. The checklist. The New Yorker. 2007. http://www.newyorker.
com/magazine/2007/12/10/the-checklist.. Accessed 20 January 2016.
47. Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. Reality check for
checklists. Lancet. 2009;374(9688):444–5.
48. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of
health promotion interventions: the RE-AIM framework. Am J Public Health.
1999;89(9):1322–7.
49. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ.
What does it mean to "employ" the RE-AIM model? Eval Health Prof.
2013;36(1):44–66.
50. Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Best practices for mixed
methods research in the health sciences. Bethesda, MD: National Institutes
of Health; 2011.
51. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis.
Qual Health Res. 2005;15(9):1277–88.
52. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al.
Outcomes for implementation research: conceptual distinctions,
measurement challenges, and research agenda. Adm Policy Ment Health.
2011;38(2):65–76.
53. Proctor EK, Powell BJ, McMillen JC. Implementation strategies:
recommendations for specifying and reporting. Implement Sci. 2013;8:139.
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research
Submit your manuscript at
www.biomedcentral.com/submit
Submit your next manuscript to BioMed Central
and we will help you at every step:
Quanbeck et al. Health Research Policy and Systems (2016) 14:8 Page 10 of 10
Content courtesy of Springer Nature, terms of use apply. Rights reserved.

1.
2.
3.
4.
5.
6.
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:
use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
use bots or other automated methods to access the content or redirect messages
override any security feature or exclusionary protocol; or
share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at
onlineservice@springernature.com