Training, supervision and quality monitoring of the COMBINE Study behavioral interventions.
ABSTRACT Current standards for clinical trials require that behavioral interventions be thoroughly specified, that clinicians be well trained and closely supervised and that performance be carefully monitored to ensure and document treatment fidelity. This article describes procedures developed and implemented for this purpose in the COMBINE Study, a multisite trial combining medications and behavioral interventions for alcohol dependence.
Two behavioral treatments were tested in COMBINE: a Medical Management (MM) protocol to accompany the delivery of trial medications and a Combined Behavioral Intervention (CBI) designed as a comprehensive and flexible evidence-based psychotherapy.
Information is provided concerning (1) screening and qualifications of practitioners, (2) training and certification in the interventions, (3) on-site supervision and coordination of the two treatments, (4) central monitoring and coding of treatment sessions and (5) maintenance and documentation of treatment fidelity.
Both interventions were delivered with fidelity. Problems encountered and important choices made in the implementation of training and quality assurance for CBI and MM are briefly described.
- SourceAvailable from: David Eisenman[Show abstract] [Hide abstract]
ABSTRACT: Posttraumatic stress disorder (PTSD) is a common problem in primary care. Although effective treatments are available, little is known about whether such treatments are effective within the context of Federally Qualified Health Centers (FQHCs) that serve as national "safety nets" for providing primary care for low income and underinsured patients. The Violence and Stress Assessment (ViStA) study is the first randomized controlled trial (RCT) to test the impact of a care management intervention for treating PTSD in FQHCs. To develop a PTSD management intervention appropriate for lower resource FQHCs and the predominantly Latino patients they serve, formative work was conducted through a collaborative effort between researchers and an FQHC practice-based research network. This article describes how FQHC stakeholders were convened to review, assess, and prioritize evidence-based strategies for addressing patient, clinician, and system-level barriers to care. This multi-component care management intervention incorporates diagnosis with feedback, patient education and activation; navigation and linkage to community resources; clinician education and medication guidance; and structured cross-disciplinary communication and continuity of care, all facilitated by care managers with FQHC experience. We also describe the evaluation design of this five-year RCT and the characteristics of the 404 English or Spanish speaking patients enrolled in the study and randomized to either the intervention or to usual care. Patients are assessed at baseline, six months, and 12months to examine intervention effectiveness on PTSD, other mental health symptoms, health-related quality-of-life, health care service use; and perceived barriers to care and satisfaction with care.Contemporary clinical trials 04/2014; · 1.51 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: Background Interventions requiring abstinence from alcohol are neither preferred by nor shown to be highly effective with many homeless individuals with alcohol dependence. It is therefore important to develop lower-threshold, patient-centered interventions for this multimorbid and high-utilizing population. Harm-reduction counseling requires neither abstinence nor use reduction and pairs a compassionate style with patient-driven goal-setting. Extended-release naltrexone (XR-NTX), a monthly injectable formulation of an opioid receptor antagonist, reduces craving and may support achievement of harm-reduction goals. Together, harm-reduction counseling and XR-NTX may support alcohol harm reduction and quality-of-life improvement. Aims Study aims include testing: a) the relative efficacy of XR-NTX and harm-reduction counseling compared to a community-based, supportive-services-as-usual control, b) theory-based mediators of treatment effects, and c) treatment effects on publicly funded service costs. Methods This RCT involves four arms: a) XR-NTX + harm-reduction counseling, b) placebo + harm-reduction counseling, c) harm-reduction counseling only, and d) community-based, supportive-services-as-usual control conditions. Participants are currently/formerly homeless, alcohol dependent individuals (N = 300). Outcomes include alcohol variables (i.e., craving, quantity/frequency, problems and biomarkers), health-related quality of life, and publicly funded service utilization and associated costs. Mediators include 10-point motivation rulers and the Penn Alcohol Craving Scale. XR-NTX and harm-reduction counseling are administered every 4 weeks over the 12-week treatment course. Follow-up assessments are conducted at weeks 24 and 36. Discussion If found efficacious, XR-NTX and harm-reduction counseling will be well-positioned to support reductions in alcohol-related harm, decreases in costs associated with publicly funded service utilization, and increases in quality of life among homeless, alcohol-dependent individuals.Contemporary Clinical Trials 07/2014; · 1.99 Impact Factor
Training, Supervision and Quality Monitoring of the
COMBINE Study Behavioral Interventions*
WILLIAM R. MILLER, PH.D.,t THERESA B. MOYERS, PH.D., LISA ARCINIEGA, PH.D., DENISE ERNST, M.S., M.A.,
AND ALYSSA FORCEHIMES, B.A.
Center on Alcoholism, Substance Abuse, and Addictions, University of New Mexico, Albuquerque, New Mexico
ABSTRACT. Objective: Current standards for clinical trials require that
behavioral interventions be thoroughly specified, that clinicians be well
trained and closely supervised and that performance be carefully moni-
tored to ensure and document treatment fidelity. This article describes
procedures developed and implemented for this purpose in the COM-
BINE Study, a multisite trial combining medications and behavioral in-
terventions for alcohol dependence. Method: Two behavioral treatments
were tested in COMBINE: a Medical Management (MM) protocol to
accompany the delivery of trial medications and a Combined Behav-
ioral Intervention (CBI) designed as a comprehensive and flexible evi-
dence-based psychotherapy. Results: Information is provided concern-
ing (1) screening and qualifications of practitioners, (2) training and cer-
tification in the interventions, (3) on-site supervision and coordination
of the two treatments, (4) central monitoring and coding of treatment
sessions and (5) maintenance and documentation of treatment fidelity.
Conclusions: Both interventions were delivered with fidelity. Problems
encountered and important choices made in the implementation of train-
ing and quality assurance for CBI and MM are briefly described. (J Stud.
Alcohol, Supplement No. 15: 188-195, 2005)
suring and documenting the fidelity of the treatments to be
tested (Carroll et al., 1994). In meeting these challenges in
behavioral treatment research, the integrity of the indepen-
dent variable hangs in the balance. Whereas the ingredients
of a pharmacotherapy are readily assayed and encapsulated,
behavioral therapies are more difficult to specify and con-
tain. This article describes the methods that were used both
to screen, train, certify, supervise and monitor practitioners
delivering two behavioral interventions in the COMBINE
Study and to maintain and document fidelity in the deliv-
ery of those interventions (COMBINE Study Research
At least five functions are crucial to quality control of
behavioral treatments in a multisite trial. A first step is
standardizing the delivery of trial treatments through thera-
pist manuals and training procedures that are consistent
across sites. That is, clinicians who are to see trial clients
should proceed through a specified and standard training
procedure. This does not in itself ensure competency and
adherence, but it does provide a common knowledge and
conceptual base for trial clinicians.
LINICAL TRIALS of behavioral interventions, whether
single or multisite, face significant challenges in en-
*This work was supported, in part, by National Institute on Alcohol Abuse
and Alcoholism grants U 10 AA 11716 and K05-AA00133.
tCorrespondence may be sent to William R. Miller, Department of Psy-
chology, MSC03 2220, I University of New Mexico, Albuquerque, NM
8713 1-0001, or via email at: firstname.lastname@example.org.
Second, a related concern in quality control is the selec-
tion of properly trained and qualified clinicians. This in-
volves initial screening, checking of professional credentials
and evaluating candidates on minimum training criteria.
A third step for quality control is the certification of
clinicians to see trial clients. Again, standardized proce-
dures and criteria should be used to approve clinicians be-
fore they treat clients in a trial.
Fourth, once clinicians have been certified, there is an
ongoing process of monitoring performance so that the con-
tent and quality of counseling do not drift from trial stan-
dards. This requires a standardized review of practice samples.
A fifth crucial function is to document the fidelity of
delivered treatments, so that adherence to and discrimin-
ability of protocols can be demonstrated in scientific peer
review. For the success and credibility of a multisite trial,
it is important to have centralized training, certification,
monitoring and documentation for behavioral interventions,
in addition to local on-site supervision.
COMBINE Behavioral Interventions
The need for a treatment Training and Quality Assur-
ance Center (TQAC) for the COMBINE trial became ap-
parent once the two behavioral interventions had been
chosen and were designed (COMBINE Study Research
Group, 2003a). The initial intent of the funding application
had been to consider two levels of behavioral intervention
to be combined with the pharmacotherapies. The first was
MILLER ET AL.
a moderate intensity behavioral therapy to be delivered in a
specialized treatment setting such as the Cognitive Behav-
ioral Therapy offered in Project MATCH (Kadden et al.,
1992). The second was a minimal behavioral intervention
that would be deliverable in a primary care setting, similar
to the behavioral interventions used to support medication
adherence in previous pharmacotherapy trials.
A Behavioral Interventions Committee was appointed to
recommend and design the two behavioral treatments to be
implemented in the COMBINE trial. The committee in turn
formed two subcommittees: one to focus on the Medical
Management (MM) counseling that would be delivered to
all participants receiving pharmacotherapies, and the other
to choose a state-of-the-art psychotherapy with which the
two pharmacotherapies (and MM) could be combined. Both
subcommittees reached the same conclusion: that previously
developed interventions did not constitute optimal treatments
with sufficient flexibility for this new multisite trial. The
three treatment manuals developed for Project MATCH
(Babor and Del Boca, 2003), for example, sought to isolate
and differentiate the three treatments from each other,
whereas both research and practice indicate the value of
combining these methods. Consequently, each subcommit-
tee set about the task of designing a new intervention, build-
ing on prior methods and what had been learned in their
implementation and testing. The processes by which these
two treatments were designed are described elsewhere in
this volume (Longabaugh et al., this supplement; Pettinati
et al., this supplement). Detailed therapist manuals were
developed for each of the two behavioral interventions,
which have been published by the National Institute on
Alcohol Abuse and Alcoholism (Miller, 2004; Pettinati et
The University of New Mexico Center on Alcoholism,
Substance Abuse and Addictions was designated as the
TQAC for behavioral interventions in the COMBINE trial.
We then set about the process of specifying and imple-
menting the necessary procedures, as described in the re-
mainder of this article.
Screening and qualifications of practitioners
A first decision regards the minimum qualifications re-
quired for a practitioner to be eligible to deliver a particu-
lar behavioral intervention. The two interventions in the
COMBINE Study required different professional compe-
tencies. In the case of the Combined Behavioral Interven-
tion (CBI), therapists also had to demonstrate proficiency
in a specific clinical style deemed to be essential for CBI.
MM. Practitioners delivering the MM intervention rep-
resented medical professions that would provide services
in primary health care, including physicians, nurses, physi-
cian assistants, nurse practitioners and clinical pharmacists.
In addition to the appropriate medical degree and license,
MM practitioners were required to have at least 2 years of
CBL. CBI therapists were required to have at least a
master's degree in counseling, psychology, social work or
a related behavioral health field. In addition to the appro-
priate degree and license, CBI therapists were required to
have at least 2 years of postdegree counseling experience.
CBI was designed to be delivered in the client-centered
style of motivational interviewing (Miller and Rollnick,
1991). Several clinical studies had shown significantly
greater reduction in drinking for clients randomly assigned
to counselors with higher skillfulness in accurate empathy
(Miller and Baca, 1983; Miller et al., 1980; Najavits and
Weiss, 1994; Valle, 1981). The early sessions of CBI rely
heavily on motivational interviewing methods, and this clini-
cal style is to be maintained throughout treatment (Miller,
2004). Training in motivational interviewing itself can re-
quire a substantial time investment (Miller et al., 2004).
Given time constraints, we chose to require a prerequisite
level of proficiency in the client-centered skill of accurate
empathy before candidates were accepted for training. This
permitted us to focus CBI training on the broader range of
cognitive-behavioral treatment methods included in the
Candidates were therefore required to submit audiotapes
of two 1 0-minute practice sessions demonstrating their pro-
ficiency in accurate empathy, as defined by Carl Rogers
and his colleagues (Truax and Carkhuff, 1967). In the first
of these, the candidate interviewed a COMBINE staff mem-
ber and was given the following instructions. "Begin the
conversation with an open invitation: Tell me about some
experiences you have had that shaped the person you are
today. There is no need for you to fix or change anything.
Just listen empathically, relying as much as possible on
reflection and periodic open questions. After 10 minutes,
bring the conversation to a close and offer a summary of
what you have heard."
The second practice sample was a structured role-play
with an actor who portrayed a client presenting for treat-
ment of alcohol problems. The actor prepared the role from
a 2-page clinical scenario that was standard across sites.
The candidate, who knew that he or she was talking with
an actor but knew nothing of the role, was told: "Your task
is to learn about the person's situation with regard to alco-
hol and what led up to entering treatment. Using primarily
open questions and reflection, demonstrate how you would
begin the first counseling session."
These practice sample audiotapes were sent to the TQAC,
where they were coded using both global ratings and spe-
cific behavior counts pertinent to accurate empathy from
the Motivational Interviewing Skill Code (Miller and Mount,
2001; Moyers et al., 2003). Candidates who failed to meet
proficiency standards were given one further opportunity
to submit practice samples with new prescribed scenarios.
JOURNAL OF STUDIES ON ALCOHOL / SUPPLEMENT NO. 15, 2005
Of 68 candidates who submitted tapes, 47 (69%) met stan-
dards on the first round. Of the remaining 21, 10 were
approved for training after a second set of practice tapes.
Training and certification
Once clinicians had met qualifying standards, they could
begin training as MM practitioners or CBI therapists. No
clinician was permitted to deliver both treatments. Separate
initial training workshops were offered for MM practition-
ers and for CBI therapists. Before treating any client in the
COMBINE Study, clinicians had to be certified by the
TQAC. Certification procedures were developed for and
tested in a pilot feasibility study before implementation in
the main trial (COMBINE Study Research Group, 2003b).
As needed, additional MM and CBI clinicians completed
initial training at their local site with assistance from certi-
MM. MM was designed as an intervention compatible
with the management of alcohol dependence in primary
health care. The optimal MM practitioner role was defined
as providing accurate information and direction within a
clinical style that is both warm and authoritative. MM fo-
cused on increasing adherence with prescribed medication,
ensuring client safety, promoting abstinence from alcohol
and encouraging involvement in Alcoholics Anonymous or
other mutual-help groups.
A 1-day workshop was organized to train MM supervi-
sors who would be responsible for maintaining supervision
at their sites and was offered twice prior to the pilot study.
For use during and after these workshops, a detailed clini-
cian manual was assembled by the MM subcommittee of
the Behavioral Interventions Committee. After explanations
of the two study medications, initial and follow-up MM
sessions were first demonstrated, and then practiced by par-
ticipants. All project coordinators attended, and sites had
the option of sending additional MM staff to this workshop
for training. The project coordinator or designated MM su-
pervisor was then responsible for training and on-site moni-
toring of MM therapists throughout the main trial. The
TQAC prepared three training videotapes to illustrate opti-
mal clinician style and procedures in delivering MM, and
these were provided to sites for use in ongoing training. Of
44 MM practitioners who were certified in the COMBINE
trial, 17 (39%) participated in the original training, and 27
were subsequently trained at local sites.
Two standardized simulated client roles (one male, one
female) were developed for purposes of MM certification.
All MM practitioners completed one initial session and at
least four follow-up sessions, with actors portraying each
of these clients. The client roles were specifically designed
to provide the clinician with experience in handling both
abstinent and nonabstinent clients during treatment and in
working with medication-compliant and noncompliant cli-
ents. Audiotapes of these sessions (at least 10 per practition-
er) were sent to the TQAC and coded on nine dimensions
derived from the treatment protocol (see Pettinati et al.,
2004). Six of these dimensions pertained to practitioner per-
formance: giving accurate information, offering sufficient
direction, authoritative tone, warmth, adherence to protocol
and avoidance of nonprotocol procedures. Client engage-
ment and cooperation were also rated. Each dimension was
rated on a 7-point scale, with 5 defined as the minimum
acceptable score. In addition to scores for each session,
candidates and their supervisors were given detailed feed-
back on strengths and areas for improvement. Practitioners
were certified when their average performance scores ex-
ceeded 5.0. Those who failed to meet certification stan-
dards based on initial tapes were permitted to complete
additional practice cases or sessions.
As the flow of clients and pressure for service increased,
and as sites lost a certified practitioner, a mechanism was
developed for provisional approval of new MM practition-
ers. Provisional approval was granted by the TQAC when
a practitioner consistently met trial standards for one type
of session (e.g., initial MM visits), but had not yet met
criteria for all types of sessions. In this case, until they
received full certification based on additional practice tapes,
practitioners were approved to deliver only those MM ses-
sions for which they had met trial standards. In a few in-
stances of time urgency, provisional approval was also
granted by the TQAC when at least two initial and four
follow-up practice sessions (rather than eight) had all met
performance criteria. In these cases, the practitioner's ses-
sions with trial clients were continuously monitored until
full certification was achieved.
A total of 44 practitioners were certified to provide MM,
of whom 3 were only provisionally certified to conduct
follow-up MM sessions. An additional 6 practitioners be-
gan but did not complete the certification process and never
delivered MM sessions. An average of 12 session tapes
were monitored before MM practitioners were certified
A total of 62 MM sessions (0.5% of all 12,526 MM
sessions) were delivered by uncertified practitioners when
patients needed to be seen and no certified MM provider
was available. Only three uncertified practitioners deliv-
ered 10 or more MM sessions over the course of the trial.
CBI. In advance of the initial training workshop, candi-
date CBI therapists studied the CBI therapist manual (Miller,
2004) and a text on motivational interviewing (Miller and
Rollnick, 1991) and viewed a series of six training video-
tapes on motivational interviewing (Miller et al., 1998) as
well as three CBI-specific training videotapes prepared by
the TQAC. Two separate trial-wide training workshops (3
days and 4 days) were offered for CBI therapists.
As with MM, clinicians who entered the study after ini-
tial trial-wide training of CBI therapists were trained at
MILLER ET AL.
their local sites using these same materials and working
with certified CBI therapists. A drawback of this later en-
try, of course, is that they began without immersion train-
ing with the developers of CBI. This was reflected in
somewhat lower initial skill ratings on practice tapes and
longer times to certification.
After training, clinicians were required to demonstrate
competence in each of the phases of the CBJ intervention
(see Longabaugh et al., this supplement; Miller, 2004) by
submitting audiotaped recordings of an entire course of
therapy with at least two actual (not role-played) clients.
These could be clients in the pilot feasibility study (COM-
BINE Study Research Group, 2003b) or clients signing a
separate consent form to receive free CBI treatment with-
out participating in the overall study. As with MM, a stan-
dardized rating system was developed to address both the
overall clinical style (motivational interviewing) and the
specific protocol components to be covered in each phase
of CBI. Beyond Phases 1 and 2, which are completed with
all clients (normally completed within four sessions), at
least three required Phase 3 modules had to be demon-
strated: job finding, mood management and coping with
craving. When the clinicians' cases did not require these
specific Phase 3 modules, they were allowed to submit ses-
sions with actor-clients to complete certification requirements.
As with MM, time and service pressures prompted the
creation of a provisional approval process for CBL clini-
cians entering the trial after randomization had begun. Role-
played practice cases were accepted, with subsequent close
monitoring of initial trial cases.
Again paralleling MM, all CBI session ratings were com-
pleted on 7-point rating scales. To be certified, clinicians
had to achieve an average rating of 5.0 or higher for Phase
1 and 2 with actual clients and for at least the three speci-
fied Phase 3 modules. Certification focused in particular on
the five therapist ratings for interpersonal skills, direction,
empathy, motivational interviewing style and adherence to
protocol content. An average of these five ratings was com-
puted for each coded session before and after certification.
In the course of the COMBINE trial, a total of 44 CBI
therapists were certified for the 12 participating clinical re-
search units. Of these, 34 (77.3%) participated in the initial
CBI training workshops. The mean number of tapes rated
per therapist prior to certification was 19, with a range of
12-34. Time to achieve certification after TQAC receipt of
the first session tape ranged from 4 weeks to 12 months.
On-site supervision and coordination of behavioral
In addition to central trial-wide monitoring at the TQAC,
each of the 12 clinical research units was responsible for
providing on-site supervision of MM and CBI practitioners
to enhance protocol adherence. Procedures were also needed
whereby MM and CBI clinicians could communicate with
each other and coordinate on-site care without compromis-
ing the integrity of the two treatments.
MM. Each local site was responsible for MM practition-
er supervision. Weekly meetings were held with the pre-
scribing professionals, the MM clinicians and the project
coordinator. These meetings also served to review and
clarify different components of the MM approach, to iden-
tity problems clinicians were having in delivering MM (e.g.,
medication noncompliance, obtaining a commitment to ab-
stinence) and to monitor and improve fidelity to the treat-
ment protocol. The project coordinator or designated MM
supervisor was also expected to monitor at least one audio-
tape per month for each MM clinician.
CBL. For CBI, a lead therapist/supervisor was designated
for each site. This person was responsible for ensuring the
integrity of the intervention and for the training of new
therapists. The site supervisor was required to convene
weekly CBI supervision sessions and to review at least one
session tape per CBI therapist per week. The TQAC pro-
vided regular feedback of C1I therapists' performance
(scores and narrative comments from coding of session
tapes) to the CBI supervisor at each site, for use in weekly
supervision. There were also monthly trial-wide conference
calls for CBI supervisors with the TQAC to discuss ongo-
ing issues and problems in the implementation of CBI.
Coordination of MM and CBI. Care was taken to main-
tain the independence of the two interventions. Communi-
cation between MM and CBI providers on each case was
carefully constrained, being allowed only for purposes of
scheduling and conveying important medical information
pertinent to client safety. When an MM client was also
receiving CBI, the MM practitioner completed a brief ses-
sion report form for the CBI practitioner, conveying the
client's self-reports of drinking and of medication adher-
ence. The CBI practitioner reviewed the report, indicated
whether the information was consistent with the client's
self-reports in CBI sessions and returned the form through
the project coordinator to the MM practitioner. There were
no regular meetings of CBI and MM clinicians to discuss
cases, and any other important information was conveyed
through the project coordinator. In this way, safety informa-
tion was shared without contaminating the two interventions.
Central monitoring and coding of treatment sessions
Beyond initial certification, procedures were needed to
minimize clinician drift and ensure continued adherence to
treatment protocols. The usual clinical trial procedure for
behavioral interventions is tape recording of all treatment
sessions, with ongoing monitoring. In the COMBINE Study,
this monitoring was performed centrally at the TQAC.
Our initial plan was to monitor 20% of all sessions for
each clinician's first 10 cases after certification, and 10%
JOURNAL OF STUDIES ON ALCOHOL / SUPPLEMENT NO. 15, 2005
of all sessions thereafter. Due to budgetary limitations, how-
ever, the COMBINE Study Research Group reduced moni-
toring to 5% of all postcertification sessions. With up to 20
sessions permitted in CBI, this reduced in practice to moni-
toring one session per CBI case. For MM, in which the
protocol prescribed nine sessions, we adopted the practice
of monitoring one session for 50% of cases after certifica-
tion. Consistent with fidelity maintenance procedures (see
below), we increased the frequency of CBI or MM session
monitoring whenever performance concerns arose for par-
If all treatment sessions in a trial are not monitored,
decision rules are needed for selecting the particular sessions
to be coded. In light of resource limitations, the challenge
was to select 5% of tapes that would span the broad range
of clinical procedures delivered to prevent drift and docu-
ment intervention fidelity. To this end, a nonrandom sam-
pling strategy was used. For each clinician, tapes were selected
to represent all phases of CBI or MM. In CBI, priority was
given to sessions containing new modules or clinical pro-
cedures (i.e., that had not previously been monitored for
that therapist). The same coding systems were used before
and after certification. All below-threshold (<5) ratings were
reviewed for accuracy by the coding supervisor at the TQAC
and called to the attention of the intervention supervisor at
the clinician's site. At least 10% of coded CBI and MM
sessions were randomly selected for re-coding by a separate
rater to assess and maintain reliability of the rating process.
All CBI and MM sessions were logged routinely on a
Session Record Form (SRF) kept in each case file. The
SRF contained the key information needed to select ses-
sion tapes for monitoring. Clinicians were not informed of
procedures used to select sessions for monitoring. All ses-
sions were audiotape recorded, and the TQAC requested
specific session tapes directly from the project coordinator
and/or supervising clinician at each site.
The TQAC coded 939 postcertification CBI session tapes,
representing 12.2% of the 7,674 CBI sessions completed.
In MM, 749 postcertification session tapes were monitored,
representing 8.1% of all 9,245 MM sessions. Of these, 107
(11.4%) CBI tapes and 106 (14.2%) MM tapes were double-
coded by independent raters.
Because of the restricted range of observed values and
the relatively small number of double-coded tapes distrib-
uted across a number of pairs of coders, it was not possible
to compute coding reliability coefficients in the usual man-
ner (e.g., by intraclass correlations). Instead, we defined
coding reliability by the degree of discrepancy in ratings
on our 7-point scales. For 107 double-coded CBI tapes,
there was exact agreement on 55.9% of ratings, with de-
creasing frequency of discrepancies of 1 point (36.6%), 2
points (6.7%) and 3 points (0.7%). For 106 double-coded
MM tapes, intercoder convergence was still higher, with
85% exact agreement on ratings, 14.3% discrepancies of 1
point and 0.7% discrepancies of 2 points. The mean coding
discrepancy was one-half point (0.523) across five CBI
scales (range: 0.495-0.559) and one-quarter point (0.277)
across seven MM scales (range: 0.047-0.369). Although the
range of mean coding discrepancies was small, the proto-
col adherence rating scale yielded the highest coder point
disagreement for both CBI (0.559) and MM (0.369) scales.
Maintaining fidelity to behavioral interventions
The recording and coding of treatment sessions does not,
in itself, guarantee practitioner adherence to prescribed pro-
tocols. To ensure the maintenance of fidelity, coding of
session tapes must be completed in a timely manner, and
clinician performance compared to pre-established quality
standards. A system is also needed for providing feedback
to practitioners and their supervisors and to correct devia-
tions from established clinical protocols.
Redlining is a common practice in behavioral treatment
trials, whereby clinicians who fall below performance cri-
teria for protocol adherence are prohibited from taking on
new cases. In both CBI and MM, when a clinician had a
pattern of scores below 5 (on the 7-point rating scales),
either overall or in a particular area, a redline warning was
issued to the site's principal investigator and treatment su-
pervisor. The practitioner was allowed to continue treating
and accepting new clients, but monitoring was intensified.
Thorough feedback was provided as to reasons for the
redline warning, and a specific performance improvement
plan was negotiated. After a brief period to allow for inte-
gration of feedback and implementation of the performance
improvement plan, the TQAC requested new session tapes.
Depending on the pattern of deficiencies, the TQAC in some
cases requested a few new session tapes containing the spe-
cific modules or clinical procedures where deficits had been
noted. In other cases, all new session tapes were moni-
tored. Extensive feedback was sent to the site regarding
these sessions, including scores on all dimensions, specific
strengths and areas for improvement and clinician status
after the coding of the recent sessions. If sufficient im-
provement was not observed, the clinician was formally
redlined, meaning that current cases could be completed
but no new cases could be assigned unless and until the
redline was lifted. Clinicians could be recertified by the
TQAC based on improved performance in new session tapes
with current clients or role-played with actors.
MM. Detailed session checklists (up to four pages per
session) were developed to help practitioners ensure proper
delivery of MM procedures. Once proficiency was estab-
lished, practitioners shifted to a more streamlined one-page
checklist with simple reminders for each key component of
the protocol. There were separate checklists for initial and
follow-up sessions. The same checklists were used by coders
at the TQAC to enumerate omissions or commissions in
MILLER ET AL.
protocol that would result in a change of score on each
rating dimension. Specific errors that resulted in a score
reduction were noted and were summarized in a sentence
that listed the dimension, the amount subtracted and a brief
justification for the change in score (e.g., "-1 Protocol (42/
43): therapist did not ask about client's menstrual calendar
and birth control"). This information was shared with MM
practitioners and their supervisors so that the feedback would
be clear and understandable.
When similar protocol errors were noted across sites and
practitioners, the TQAC prepared a composite transcript of
a model MM session. The transcript highlighted how infor-
mation should be presented and gave MM clinicians spe-
cific guidelines for the depth and length of a solid "6"
session. The TQAC also provided feedback on common
problem areas to the MM oversight subcommittee, for use
in discussion on conference calls with MM practitioners.
Across the six dimensions on which MM therapists were
rated, observed mean ratings (N = 749) ranged from 5.4 to
6.2. Significant site differences were observed on two of
six dimensions: the ratings of information giving (p < .005)
and warmth (p < .001).
During the treatment phase of the COMBINE Study,
four MM redline warnings were issued. In all four cases,
practitioners succeeded in having their probationary status
lifted without formal redlining.
CBI. As for MM, detailed session checklists were devel-
oped for CBI and are included in the therapist manual
(Miller, 2004). These checklists served three purposes. First
and foremost, the CBI therapists used them during and af-
ter sessions to document and prompt adherence to the treat-
ment protocol. The on-site CBI supervisor at each site was
also instructed to use these checklists in the required moni-
toring of one session per CBI therapist per week. Finally,
coders at the TQAC used these checklists to detect omis-
sions and commissions in protocol adherence.
Across six dimensions on which CBI therapists were
rated on 1-7 scales, observed mean ratings (N = 722) ranged
from 5.65 to 6.04. On all six dimensions, significant differ-
ences were observed by analysis of variance in therapist
ratings across the 12 sites (all p's < .001).
During the course of the COMBINE Study, redline warn-
ings were issued to six CBI therapists (14% of the 44 certi-
fied). Of these, three had difficulty improving their
performance and were moved to redline status. Two of the
redlined therapists were recertified after implementation of
a performance improvement plan, and one chose to with-
draw from the study.
Discriminability of CBI and MM
The tape-rating systems for MM and CBI were not de-
signed specifically to evaluate the discriminability of these
two behavioral treatments, but rather to ensure fidelity in
the delivery of each. Each treatment had a distinct tape-
rating system, and there were no directly comparable scales
between the two systems.
A principal reason for this is that each of the two behav-
ioral treatment protocols contained many prescribed inter-
vention elements that were unique. For example, CBI had
no standard counterparts for these MM elements: educating
patients about the medications, monitoring physical health
issues, inquiring about medication dosing, monitoring and
managing side effects or counseling all patients to attend
Alcoholics Anonymous. Similarly, CBI contained many spe-
cific modules for which there was no MM counterpart, in-
cluding Motivational Enhancement Therapy, assessing client
strengths and teaching skills for communication and for
coping with negative moods. CBI therapists received spe-
cific training in the counseling method of motivational in-
terviewing, and adherence to this counseling style was rated
for every coded session. No such training was given to the
MM practitioners, who were instead rated and given feed-
back on their authoritative style, giving direction and inter-
personal warmth. Adherence to protocol was rated in both
CBI and MM, but of course the protocol to which each
was expected to adhere differed.
Our experience in this study underlines the importance
of devoting adequate forethought to training and quality
monitoring for behavioral treatments. The initial request
for proposals had anticipated that pre-existing behavioral
interventions would be used, for which manuals, training
and monitoring procedures were already available. Propos-
als therefore required little attention to details of training
and quality monitoring. To some extent, problems also arose
from the merger of two quite divergent research traditions
(Miller et al., this supplement). In pharmacotherapy research,
it has been common (with significant exceptions) to devote
minimal attention and resources to quality control of be-
havioral treatment. Rather, the behavioral intervention rep-
resents a platform on which to evaluate the efficacy of
medications. In behavioral treatment research, however, ex-
tensive effort is typically devoted to ensuring intervention
fidelity. Within Project MATCH, for example, a separate
coordinating center was funded at Yale University, with
the primary task of providing training and quality assur-
ance for the behavioral interventions being tested (Carroll
et al., 1994). In contrast, the funding and coordinating cen-
ter for COMBINE were conceptualized primarily as for a
pharmacotherapy trial, with minimal provision for the train-
ing and monitoring of behavioral interventions.
From the outset, then, balancing the costs of monitoring
against assurance of treatment fidelity proved to be a
JOURNAL OF STUDIES ON ALCOHOL / SUPPLEMENT NO. 15, 2005
challenging task in the COMBINE Study. With no budget
specifically earmarked for behavioral treatment monitoring,
cost containment was a substantial concern. This presented
the challenge of deciding the frequency of session monitor-
ing that would be just enough to ensure treatment fidelity.
Within these constraints, we obtained reasonably good evi-
dence that the behavioral interventions were delivered with
A lesson learned here is the importance of advance plan-
ning and resource allocation for the considerable tasks of
quality assurance in behavioral intervention research. Cer-
tainly if analyses of therapeutic process data are to be in-
cluded (e.g., Longabaugh and Wirtz, 2001), substantial
resources need to be planned for treatment session coding
of most or all treated cases. On the other hand, if fidelity
documentation is the primary aim, more modest resources
may suffice. However, we recommend that, particularly in
smaller trials, 10-20% of treatment sessions should be moni-
tored for fidelity assurance purposes.
Priority to process data and adherence
Behavioral intervention research requires a level of ad-
vance planning and priority for treatment process data that
has rarely been required in pharmacotherapy trials. This
again illustrates unique challenges in the convergence of
clinical trial traditions (Miller et al., this supplement). For
example, SRFs were constructed as an afterthought in de-
signing the two behavioral interventions. The SRFs con-
tained information including the session number, clinician,
specific treatment modules delivered, length of the session
and whether the client's significant other attended. Such
information, largely irrelevant in pharmacotherapy research,
was needed to select representative sessions to be moni-
tored and is useful in describing behavioral treatment de-
livery and processes. However, the SRFs were not regarded
as part of the core trial data set, and an electronic data
entry system for SRFs was not developed until well after
the treatment phase of the trial was underway. Consequently,
entry of SRF data (at first by hard copies faxed to the
TQAC, and later electronically) became a relatively low
priority among the many demands of a multisite trial. The
entry of SRF data in turn was a rate-limiting prerequisite
for the selection of treatment sessions to be requested and
monitored by the TQAC. Furthermore, being apart from
the core data set, SRF information was sent to the TQAC
with minimal central data checking and cleaning, resulting
in significant quantities of initially missing or inaccurate
data and further delays in identifying practitioners and ses-
sions to be monitored. As a result, performance feedback
to therapists was normally delayed by several months, and
sometimes arrived as long as a year after completion of a
monitored session. This obviously slows the process of de-
tecting and correcting clinician deviations from protocol.
Similarly, accountability was minimal for local site ad-
herence to treatment supervision and monitoring procedures.
Although the protocol required each site's two supervisors
(for MM and CBI) to convene weekly supervision sessions
and to regularly monitor session tapes for each clinician,
no data were collected to ensure that this occurred, nor
entered to document levels of adherence. This may be re-
flected in the observed site differences in therapist ratings,
which in turn may contribute to site differences in behav-
ioral intervention outcomes.
Our recommendation here is that treatment process and
adherence data should be given the same status as outcome
data in behavioral intervention research. The integrity of
the independent variable is at stake. We recommend that,
in future studies, a separate SRF be completed for each
session and entered promptly (e.g., within 1 week of ses-
sion completion) into the core database with the same pri-
ority to data quality control. This further permits early
identification of system-wide problems in treatment imple-
mentation. For example, we found that, whereas the proto-
col specified MM sessions as being between 15 and 30
minutes in length, TQAC coders who timed the audiotapes
were reporting some very long (>60 minutes) and very short
(<5 minutes) sessions. Examination of the available SRF
data revealed systematic variability by site and practitioner.
Feedback of mean session length was given to each site
investigator and practitioner, and session length was added
to TQAC tracking of deviations from protocol, with the
result of quickly reducing the degree of variability in MM
session length. At this point, routine tracking of session
length was added to the TQAC monitoring procedures.
To our knowledge, COMBINE is the first study to use
systematic behavioral performance prescreening of practi-
tioners' clinical skillfulness as a prerequisite for entry to
training. Given the amount of material to be learned by
CBI therapists, a prescreen for accurate empathy was in-
cluded prior to acceptance for training and employment.
We were quite pleased with the result. Compared with our
experience in training unscreened counselors (e.g., Miller
and Mount, 2001), CBI therapists showed relatively few
problems in manifesting the clinical style of motivational
interviewing. As might be expected, difficulties more often
emerged at the transition to Phase 3 of CBI, where thera-
pists had to shift to the more directive and prescriptive
tasks typical of Cognitive Behavioral Therapy.
The prescreening of study clinicians, of course, moves
the trial a step away from normal community practice.
Within the Clinical Trials Network of the National Institute
on Drug Abuse (http://www.nida.nih.gov/CTN/Index.htm),
study interventions are, by design, delivered by randomly
chosen, regular frontline staff of ongoing community treatment
MILLER ET AL.
programs. This addresses the real-world challenge of dis-
seminating treatment innovations to clinicians who may not
be particularly well disposed toward or prepared for them
(Rogers, 2003). In efficacy trials, however, this advantage
in external validity must be weighed against the protection
of internal validity through quality control of the indepen-
dent variable. The use of a performance prescreen can give
trial therapists a head start on acquiring proficiency in the
behavioral interventions being tested. This can be particu-
larly helpful when the trial treatment is complex.
Given the strength of evidence that clinician empathy
improves client treatment outcome, there is good reason to
consider behavioral prescreening as a prerequisite for em-
ployment of clinicians in community treatment programs.
While operating a large public outpatient treatment facility,
the Center on Alcoholism, Substance Abuse, and Addic-
tions in Albuquerque, New Mexico, included a performance
test of reflective listening as part of the hiring process for
new substance abuse counselors. Our positive clinical ex-
perience with this process led to the inclusion of a similar
prescreen in the COMBINE Study. In this regard,
prescreening of counselors for requisite behavioral profi-
ciency might not be regarded as a departure from good
clinical practice, but rather a process that should be more
widely and routinely included in community treatment sys-
tems. Indeed, it could be argued that the usual standard in
clinical trials-that clinicians demonstrate behavioral profi-
ciency and be certified before they deliver a specific interven-
tion-would be good practice in community settings as well.
The training and quality control of behavioral interven-
tions in clinical trials are not simple matters. Well-planned,
timely and carefully implemented systems are required not
only to document but also to maintain treatment fidelity
during the course of a trial. Well-conceived treatment moni-
toring can achieve research aims well beyond the assur-
ance of fidelity in delivery. Optimal processes for helping
clinicians to acquire and maintain proficiency in treatment
innovations are themselves a topic in need of study (Miller
et al., 2004; Rogers, 2003). If client outcomes are a func-
tion not just of an aggregate intervention, but of subtle
processes occurring within behavioral treatment, then clini-
cal trials miss vital information unless treatment process
analyses are included (Longabaugh and Wirtz, 2001). Such
analyses can also contribute to better understanding of the
mechanisms underlying behavioral interventions and to fur-
ther improvement of current treatment methods.
BABOR, T.F. AND DEL BOCA, F.K. (Eds.) Treatment Matching in Alcohol-
ism, New York: Cambridge Univ. Press, 2003.
CARROLL, K.M., KADDEN, R.M., DONOVAN, D.M., ZWEBEN, A. AND
ROUNSAVILLE, B.J. Implementing treatment and protecting the validity
of the independent variable in treatment matching studies. J. Stud.
Alcohol, Supplement No. 12, pp. 149-155, 1994.
COMBINE STuDY RESEARCH GROUP. Testing combined pharmacotherapies
and behavioral interventions in alcohol dependence: Rationale and
methods. Alcsm Clin. Exp. Res. 27: 1107-1122, 2003a.
COMBINE STUDY RESEARCH GROUP. Testing combined pharmacotherapies
and behavioral interventions for alcohol dependence (The COMBINE
Study): A pilot feasibility study. Alcsm Clin. Exp. Res. 27: 1123-
KADDEN, R., CARROLL, K., DONOVAN, D., COONEY, N., MONTI, P., ABRAMS,
D., LITT, M.
Therapy Manual: A Clinical Research Guide for Therapists Treating
Individuals with Alcohol Abuse and Dependence. NIAAA Project
MATCH Monograph Series, Vol. 3, DHHS Publication No. (ADM)
92-1895, Washington: Government Printing Office, 1992.
LONGABAUGH, R. AND WIRTZ, P.W. (Eds.) Project MATCH Hypotheses:
Results and Causal Chain Analyses. NIAAA Project MATCH Mono-
graph Series, Vol. 8, NIH Publication No. 01-4238, Washington: Gov-
ernment Printing Office, 2001.
MILLER, W.R. (Ed.) Combined Behavioral Intervention: A Clinical Re-
search Guide for Therapists Treating People with Alcohol Abuse and
Dependence. NIAAA COMBINE Monograph Series, Vol. 1, DHHS
Publication No. 04-5288, Bethesda, MD: Department of Health and
Human Services, 2004.
MILLER, W.R. AND BACA, L.M. Two-year follow-up of bibliotherapy and
therapist-directed controlled drinking training for problem drinkers.
Behav. Ther. 14: 441-448, 1983.
MILLER, W.R. AND MOUNT, K.A. A small study of training in motivational
interviewing: Does one workshop change clinician and client behav-
ior? Behav. Cog. Psychother. 29: 457-471, 2001.
MILLER, W.R. AND ROLLNICK, S. Motivational Interviewing: Preparing People
to Change Addictive Behavior, New York: Guilford Press, 1991.
MILLER, W.R., ROLLNICK, S. AND MOvERS, T.B. Motivational Interviewing,
6-Videotape Series, Albuquerque, NM: Center on Alcoholism, Sub-
stance Abuse, and Addictions (CASAA), University of New Mexico,
MILLER, W.R., TAYLOR, C.A. AND WEST, J.C. Focused versus broad-spec-
trum behavior therapy for problem drinkers. J. Cons. Clin. Psychol.
48: 590-601, 1980.
MILLER, W.R., YAHNE, C.E., MOvERS, T.B., MARTINEZ, J. AND PIRRITANO,
M. A randomized trial of methods to help clinicians learn motiva-
tional interviewing. J. Cons. Clin. Psychol. 72: 1050-1062, 2004.
MOYERS, T., MARTIN, T., CATLEY, D., HARRIS, K.J. AND AHLUWALIA, J.S.
Assessing the integrity of motivational interviewing interventions: Re-
liability of the Motivational Interviewing Skills Code. Behav. Cog.
Psychother. 31: 177-184, 2003.
NAJAVITS, L.M. AND WEISS, R.D. Variations in therapist effectiveness in
the treatment of patients with substance use disorders: An empirical
review. Addiction 89: 679-688, 1994.
PETTINATI, H.M., WEISS, R.D., MILLER, W.R., DONOVAN, D., ERNST, D.B.
AND ROUNSAVILLE, B.J. (Eds.) Medical Management Treatment Manual:
A Clinical Research Guide for Medically Trained Clinicians Provid-
ing Pharmacotherapy as Part of the Treatment for Alcohol Depen-
dence. NIAAA COMBINE Monograph Series, Vol. 2, DHHS
Publication No. 04-5289, Bethesda, MD: Department of Health and
Human Services, 2004.
ROGERS, E.M. Diffusion of Innovations, 5th Edition, New York: Free Press,
TRUAX, C.B. AND CARKHUFF, R.R. Toward Effective Counseling and Psy-
chotherapy: Training and Practice, Chicago, IL: Aldine, 1967.
VALLE, S.K. Interpersonal functioning of alcoholism counselors and treat-
ment outcome. J. Stud. Alcohol 42: 783-790, 1981.
AND HESTER, R. Cognitive-Behavioral Coping Skills
TITLE: Training, Supervision and Quality Monitoring of the
COMBINE Study Behavioral Interventions
SOURCE: J Stud Alcohol supp no15 Jl 2005
The magazine publisher is the copyright holder of this article and it
is reproduced with permission. Further reproduction of this article in
violation of the copyright is prohibited.
Copyright 1982-2005 The H.W. Wilson Company. All rights reserved.