Hindawi Publishing Corporation
Depression Research and Treatment
Volume 2012, Article ID 391084, 11 pages
PerspectivesonCognitive Therapy Trainingwithin
CommunityMental Health Settings: ImplicationsforClinician
ShannonWiltsey Stirman,1Christopher J. Miller,2KatherineToder,3Amber Calloway,1
AaronT. Beck,3ArthurC.Evans,4and PaulCrits-Christoph3
1VA National Center for PTSD, VA Boston Healthcare System, Boston University, Washington, DC 20420, USA
2Center for Organization, Leadership, and Management Research, VA Boston Healthcare System, Washington, DC 20420, USA
3Department of Psychiatry, University of Pennsylvania, Philadelphia, PA 19104, USA
4Department of Behavioral Health and Intellectual disAbilities Services, Philadelphia, PA 19107, USA
Correspondence should be addressed to Shannon Wiltsey Stirman, firstname.lastname@example.org
Received 30 March 2012; Accepted 31 July 2012
Academic Editor: Mark Williams
Copyright © 2012 Shannon Wiltsey Stirman et al. This is an open access article distributed under the Creative Commons
Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is
Despite the mounting evidence of the benefits of cognitive therapy for depression and suicidal behaviors over usual care, like other
evidence-based psychosocial treatments (EBTs), it has not been widely adopted in clinical practice. Studies have shown that
session treatment packages such as cognitive therapy. Given the critical role of training in EBT implementation, more information
on factors associated with the success and challenges of training programs is needed. To identify potential reasons for variation in
training outcomes across ten agencies in a large, urban community mental health system, we explored program evaluation data
cognitive therapy, contextual factors, and reactions to feedback on audio recordings emerged as broad categories of themes iden-
tified from interviews. These factors may interact and impact clinician efforts to learn cognitive therapy and deliver it skillfully
in their practice. The findings highlight experiences and stakeholder perspectives that may contribute to more or less successful
The public health impact and high rates of depression and
suicideincommunity populations arewellestablished[1–5].
evidence-based treatments (EBTs) such as cognitive therapy
(CT; ) results in substantial benefits to physical and men-
tal health symptoms, quality of life outcomes, and reduction
of health disparities and suicide attempts [7–9]. Training
clinicians in community mental health agencies to provide
CT for depression can result in improved treatment out-
comes . Like other EBTs, however, CT has not been widely
adopted in clinical practice. Until recently, few systematic
implementation efforts or training opportunities existed in
the public sector [10, 11]. In response to the shortage of ade-
quately trained providers , policymakers have devoted
substantial resources in recent years to train providers in
EBTs in the context of large-scale implementation programs
In 2007, the Beck Initiative was formed as a collaborative
partnership between the Philadelphia Department of Behav-
ioral Health and Intellectual disAbilities Services (DBHIDS),
and the University of Pennsylvania (Penn) to implement CT
within the city’s behavioral health provider agencies. The
mental health system administrators became interested in
implementing CT because it has a strong evidence base for
multiple disorders and client populations, it can prevent
relapse and reduce costs associated with treatment, and
2Depression Research and Treatment
it has been shown to improve quality of life [13–15].
In determining the disorders to target in the first phase
of the training program, the partnership members chose
depression and suicidal behavior for three reasons. First, a
high proportion of consumers in the system have a diagnosis
of depression. Second, they believed that training in CT
basics as applied to depression would serve as a good foun-
dation for more specialized trainings. Third, because suicide
prevention is a high priority, after the results of a recent
above usual care for highly comorbid suicidal individuals in
the community , partnership members were interested
in rapidly making the treatment available throughout the
system. Because psychosocial treatments for depression tend
to be complex multisession treatment packages [16, 17],
research has demonstrated that one-time trainings without
support in the form of expert consultation are insufficient
to promote skill development and behavior change [18–24].
workshops with an intensive consultation component that
included weekly discussion of cases combined with expert
review of, and feedback on, clinician’s efforts to provide
cognitive therapy to their clientele.
Beyond the growing evidence regarding the importance
of consultation in EBT implementation, limited information
is available about determinants of success, or shortcomings
of training and consultation strategies. Research to date has
focused on comparisons of front-end training strategies and
examinations of the impact of adding consultation. Little
research has been conducted to facilitate understanding of
the key processes in training and consultation. Qualitative
research with stakeholders can complement the existing
of training and consultation programs that may impact the
success of implementation efforts.
The purpose of this study is to identify aspects of the
To do so, we explored program evaluation data and provider,
consultant, and training program administrator perspectives
community-academic partnership to implement CT. In con-
trast with published randomized controlled trials of training
and consultation, which have included individually recruited
clinicians, this study examined training that occurred with
cohorts within ten agencies in an urban community mental
2.1. Training Program
2.1.1. Setting. DBHIDS is a large mental health system with
over 300 provider agencies that are heterogeneous in size,
structure, populations served, and availability of resources.
These agencies provide care to serve the mental health
and substance abuse needs of the city’s 420,000 Medicaid
recipients. Recent utilization data indicate that approxi-
mately 38% of the clients in the mental health system have
a diagnosis of major depressive disorder (28%) or depression
NOS (10%), and 87% of these clients receive psychotherapy
2.1.2. Treatment. CT is a psychosocial treatment that identi-
and emotional responses by helping individuals develop
skills for modifying beliefs, relating to others in different
ways, and changing behaviors through the use of a variety of
interventions . The structure of the session is intended
to reflect the active, goal-oriented, collaborative nature of
the treatment. Sessions are structured to include a brief
summary or bridge between the previous and current ses-
sion; efforts to set and follow an agenda; review of practice,
or “homework” completed between sessions; summaries of
important points; feedback from clients on their experience
efforts to identify a way to practice new skills between ses-
sions . Competence at structuring sessions as measured
by the structure subscale of the Cognitive Therapy Rating
Scale (CTRS; [27, 28]), and the assignment and completion
of homework [29, 30], have been shown to predict greater
symptom reduction in CT.
2.1.3. Training and Consultation. The training program was
designed to teach CT basics, with a particular emphasis
on depression and suicidal behaviors. It included extensive
guidance on using CT to address commonly cooccurring
problems among individuals who present with symptoms
of depression. Training consisted of 24 hours of workshops
gram was administered by a project director at Penn and by
a full-time operations specialist within DBHIDS. To address
financial constraints to participation in training, agencies
were reimbursed for time spent in training and consulta-
tion activities. A clear expectation of the program was that
agencies continue to support the ongoing use of CT after
training: clinicians were trained in cohorts within their
agency so they could build a community of practice, and
throughout the program, the partnership worked with
to training was based on information obtained through a
needs assessment  and a review of literature from a vari-
ety of disciplines . Given the evidence that a multilevel
approach is critical to implementation, the training program
within the system [31, 32]. External facilitation, a process of
interactive problem solving and support  that has been
shown to be effective in engaging stakeholders, addressing
potential barriers to implementation, and increasing the use
of EBTs [36–39], was integrated into the training program.
The training model that was developed, the ACCESS
training and consultation model, included the following
components : Assess needs and barriers (engage stake-
holders and assess through preliminary meetings, surveys,
work samples, interviews) and Adapt training content and
Depression Research and Treatment3
materials as required, Convey the basics through initial didac-
tics, Consult on case material and on strategies to overcome
barriers during consultation and through periodic meetings
with key personnel, Evaluate work samples to provide feed-
backand refineskills, Study outcomesin waysthatarefeasible
and acceptable to the agency, and Sustain by anticipating
and addressing future barriers, maintaining communica-
tion, and making a plan for training future staff. The evalua-
tion of work samples occurred in two formats. Initially, Penn
consultants reviewed audio recordings of sessions and pro-
ual clinicians, and a group consultation meeting was held to
discuss more general CT skills. Later, in response to concerns
about the time, costs, and scalability of this approach, indi-
vidual consultation was replaced with review and discussion
mat. Additionally, as the goal of the program was to promote
ongoing CT skill development and fidelity after training,
the group feedback format was viewed as preparation for
the groups to transition to ongoing peer-led group session
review. During the consultation phase, consultants also
provided individualized feedback on one to two full sessions
for each participant in the group consultation model. Two
clinicians who struggled to master CT in this model were
given additional feedback and consultation.
2.1.4. Selection of Participating Agencies and Clinicians.
Agencies were nominated for participation in the training
program by mental health system administrators, who
attempted to make the training available to a variety of pro-
grams serving diverse populations throughout the city. The
training director at the university made a final selection for
each round of training after reviewing information about
the agency and the populations that it served and taking into
consideration the areas of expertise of available consultants.
Key personnel meetings were held with agency administra-
tors to discuss the program, assess fit, and build collabora-
tion. Agencies had latitude to select clinicians for participa-
tion internally, although criteria for inclusion set forth by the
training program were that participants should have an
interest in learning CT and be willing and able to participate
fully in the program (e.g., record sessions with training cases
and attend training workshops and consultation meetings).
Consultation was limited to eight or fewer participants at
each agency, regardless of agency size, although some agen-
cies sent additional employees or supervisors to the initial
workshop. Participants completed an application that
included information about their prior training and reasons
for wanting to learn CT. Applications were used primarily to
understand clinicians’ backgrounds prior to training, and no
clinicians who were selected by their agencies were denied
participation in the training program. To assess and facilitate
engagement, prior to the workshops, preliminary meetings
were held with the clinicians (with no management present)
to describe the training program, emphasize the voluntary
that clinicians expressed. Training generally occurred in
groups of providers from a single agency, but on two
occasions, trainings included groups of providers from two
agencies. A total of 99 providers of adult outpatient services
at ten agencies enrolled in the basic training program.
2.1.5. Criteria for Successful Completion of Training. Provid-
ers were eligible for designation as skilled providers of CT
(hereafter described as “passing” for brevity’s sake) if they
6 months, submitted at least 20 recordings of sessions with
clients who agreed to serve as training cases, and achieved
a rating of at least 40 on the CTRS, a validated measure of
competence in CT . A score of 40 was chosen as the
passing score because it is the standard minimum required
competence score for clinicians who serve as therapists in
clinical trials of CT for depression .
2.2. Study Procedures. The study was approved by the Penn
and City of Philadelphia Institutional Review Boards. All
of the clinicians who participated in one of twelve training
programs at ten agencies were invited to participate in
interviews regarding their use of CT in adult outpatient
service settings. Forty clinicians expressed interest in partici-
able to be scheduled for posttraining interviews. Clinicians
who participated in interviews consented to allow their
CTRS scores to be linked to their interview data. Training
consultants (n = 6) and program administrators from the
university (n = 3) who were involved in the program were
also interviewed. To ensure that our sample would reflect a
broad range of perspectives, we also evaluated anonymous
feedback forms (free response and numeric ratings) from
training and consultation that were collected as program
Participants were scheduled for interviews at times that
were convenient for them and that would not interfere with
their clinical and work-related responsibilities. Semistruc-
tured interviews were conducted over the telephone or in
person and interviews were digitally recorded. An interview
guide included open-ended, scripted questions that were
based on the Promoting Action on Research Implementation
in Health Services (PARIHS) Framework . Participants
were given the opportunity to elaborate on issues that
they considered particularly relevant or about which they
expressed strong sentiments. Interviews typically lasted 45
sation or a gift card for their participation. Interviews were
transcribed and transcripts were reviewed and checked for
accuracy by at least one of the authors.
2.3. Coding Procedures. Clinician interviews included ques-
tions about a variety of factors, but for the current paper, we
will focus on responses that address training and consulta-
tion. Transcripts of interviews and program evaluation data
were subjected to qualitative analysis rooted in grounded
theory , using the ATLAS.ti software package . First,
study investigators developed a list of a priori themes and
emergent themes were added after reading the interview
script and several transcripts. After saturation was achieved
4 Depression Research and Treatment
(i.e., the coding of additional transcripts did not identify any
further themes to be coded), a preliminary codebook was
developed based on the identified themes. This preliminary
two additional transcripts, resulting in discussion and revi-
sions to the codebook based on group consensus. Using this
revised codebook, two study researchers rated the remaining
transcripts. A subsample of transcripts was coded by both
raters, and disagreements regarding coding were resolved
via consensus of the study team in weekly coding meetings.
This process also identified several additional subcategories
of codes. A process of axial coding suggested ways in
which some codes could be combined or collapsed into one
another, and arranged all of the codes into a treelike, hier-
archical structure. The result of this process was a finalized
codebook, with links specified between key concepts. The
two primary raters reviewed transcripts and updated their
ratings to reflect the themes from this finalized codebook.
Program evaluation data was integrated with interview and
free response data, and through a process of triangulation
and constant comparison, the categories were organized
around the major emergent themes.
ipated in interviews were 66% female and had an average of
5.7 years of experience working in mental health treatment
settings (sd = 1.3). Eighty percent had a Master’s degree,
7% had completed some graduate work (e.g., towards a
Master’s Degree), and 13% had a Bachelor’s degree. Partic-
ipants were 70% Caucasian, 19% Black, 4% Asian, and 7%
were multiracial or endorsed a different race or ethnicity.
Seven percent of the participants were Latino. Participating
agencies varied in size, with anywhere from 5 to over 50 clin-
icians serving their adult outpatient clientele. Most agencies
provided general outpatient mental health services, but four
agencies, or subprograms within the agencies, that served
specific populations participated in the training. These
populations included individuals with severe mental illness,
specific minority populations, and individuals in recovery
from substance use disorders. Additionally, one agency
included clinicians from their specialized program for severe
mental illness as well as from their general outpatient mental
3.2. Training Program Outcomes. Table 1 provides an over-
view of the proportion of clinicians who passed at each
agency, the feedback and consultation format used, informa-
tion about whether the agency specialized in treatment for
specific client populations or diagnoses, and information on
retention in the program. Overall, approximately 80% of the
clinicians who completed consultation passed. At four agen-
cies, one or two clinicians did not complete consultation.
Program evaluation data revealed that attrition was due to
relocation, layoffs due to budget constraints, medical leave,
or taking a new position. At two agencies, approximately half
higher pass rates. One of the agencies with lower pass rates
provided specialized services and the other provided general
mental health services. Of the 26 clinicians who participated
in interviews, seven clinicians from five different agencies
did not pass. Due to the sensitive nature of some of the
information that participants disclosed, and to ensure that
agencies could not be identified, we have not revealed exact
training, or linked quotations to agency IDs or more specific
information about the agencies.
Program evaluation data indicated that overall satisfac-
tion with the program was generally high. The mean overall
quality rating for the individual feedback model was 4.88
(SD = 1.4) on a 6-point Likert scale, and the group feedback
model was rated 5.63 (SD = .52). After training, clinicians
who participated in the individual model rated their comfort
in applying CT to be a mean of 3.81 (SD = 1.5) and
participants in the group format expressed a mean comfort
score of 4.88 (SD = .64).
3.3. Themes Related to Training Success. As described above,
an axial coding process was used to identify major themes
from the qualitative interview data. Five themes relevant to
training success emerged during the coding process, which
were grouped into three broad categories: perceptions of CT
(relevance to agency clientele and perceptions of CT struc-
ture); contextual factors (agency involvement and impact of
clinician selection process); and experience of consultation
and feedback on recordings.
3.3.1. Perceptions of Cognitive Therapy
Relevance of CT to Agency Clientele. In light of pre-training
findings that some clinicians within the system had doubts
that CT could meet the needs of some of their clients ,
efforts were made to match consultants who had clinical
experience with similar populations and to identify appro-
efforts to use relevant case material and demonstrate ways in
which case conceptualization could facilitate the appropriate
application of CT strategies to common cooccurring pre-
senting problems. Interview data indicated that this strategy
to the use of CT with their clientele. At an agency that
therapists conducting sessions with clients that
were more similar to the ones we dealt with. Cer-
tainly I could see where that was helpful, where it
worked... I feel that it could work absolutely with
our population... but only if the therapists were as
invested in the technique.
Clinicians in some agencies also expressed appreciation
that consultants appeared to have experience in working
with clients like theirs. In contrast, at an agency where fewer
clinicians passed, a consultant for that agency described
Depression Research and Treatment5
Attrition from consultation?†
Proportion of completers
Offered specialized treatment for
a specific population
Note. Numbers of clinicians who received training and passed the program are not provided to decrease the likelihood that the identities of agencies with
clinicians who participated are discerned.
†At agencies with noncompleters, noncompletion rates ranged from 10–25%. Reasons for not completing include leaving the agency, personal circumstances,
moved to another division of the agency that was not participating in training (e.g., intake department). No clinicians reported reasons for dropping out that
were related to the training or consultation.
challenges in addressing concerns about the use of CT with
the clientele at that agency.
Consultant: Training was very difficult at that
agency because from the beginning we had low
the application of CT for their clients and they
really struggled with taping, with utilizing CT
techniques.... (The other consultant) and myself
tried to discuss clients that we had seen with very
similar levels of functioning.... it was kind of gen-
eral community mental health depression, kind of
the multi-disordered. And then also some people
with psychotic disorders.... I do not see that pop-
ulation being any different from the other agen-
However, a clinician at that agency expressed a very dif-
tants], [who] really never gave ample time or conversation to
my population and that might be because they never had any
experience in it.” At another agency where fewer clinicians
as applicable to a variety of presenting problems and felt that
consultants were not open to their efforts to discuss the
limitations of CT.
Interviews with three of the consultants and two train-
ing program administrators suggested that the strategy of
teaching CT basics as applied to depression and suicide may
have generated some impatience among clinicians. Many
expressed interest in guidance on addressing issues that
they perceived to be more difficult to address in treatment,
such as psychosis, Axis II diagnoses, and active substance
dependence. A consultant described this concern at an
agency that offered specialized services.
Consultant: Although they really were open and
interested and invested in learning, I think they
struggled with the fit and they were sort of assured
we will think about how to apply it to this very
challenging population.” But they struggled I
think with that, to the extent that the training
program rethought that approach to some degree.
Reactions to CT Structure during Training. Despite previ-
ously documented concerns within the system about the
potential for CT to be rigid , most clinicians did not
express strong concerns about its structure. One clinician
found that the training disabused her of stereotypes in this
domain: “... it’s really supportive, and it’s pretty flexible, so
I think the training was good to break that stereotype that
CT is going to be very structured and rigid [without] a lot of
room for other opportunities to connect with people.” Some
one indicating on the program evaluation survey, “Agenda
setting has helped me organize the sessions and helped my
which is very efficient and respectful.” Additionally, some
noted the positive impact of using the CT structure on with
clients who presented as disorganized or “scattered.”
Some negative reactions to CT structure were described,
however, particularly at agencies where fewer clinicians
feedback to clinicians who demonstrated ongoing discom-
fort with the structure of CT. A consultant explained that
structuring sessions was challenging for some clinicians.
Clinician: I do not think that she was as com-
fortable with the structure or with delivering the
therapy. It took a fair amount of work to get her to
really start delivering the intervention with the
sell for her to implement it.
Without the structure in place, consultants stated that
6 Depression Research and Treatment
were delivered sporadically or unsystematically. Consultants
described efforts to balance this feedback regarding structure
with their ongoing efforts to simultaneously emphasize case
Consultant: I did not want anything that I did to
turn them off to [CT] so I tried to hear out their
concerns.... but then [it was also important to]
make sure that they got what they needed to really
be able to deliver the treatment....
However, at the two agencies with lower pass rates, clini-
cians expressed concern that training overemphasized the
structure of CT, to the detriment of patient-centeredness; as
one clinician concluded, “it seemed that the structure was
more important than helping the client.”
CT structure was initially perceived by some clinicians
who ultimately passed to be intimidating or challenging to
Clinicians: I think when I started the training I
was kind of afraid of all of it because... it was so
much for me... in terms of me thinking about the
structure and knowing where I was, and pacing,
and summarizing, and mood check. But now that
I do it fairly regularly, I think it’s okay.
Perhaps due to the perception that the CT structure was
challenging to implement, some clinicians who maintained
structured sessions during the training indicated that they
“loosened up” the structure soon after they completed
Clinician: When I would have a consultation
no one’s going to be supervising me—listening,
and so forth—I do not feel that pressure. I feel like
I can relax a little more. And relaxed to me means
it’s okay if I do not hit all the points.
3.3.2. Contextual Factors
Administration and Supervisor Involvement. Although the
training model included initial and periodic meetings with
agency administration and supervisors to assess needs,
obtain feedback, and discuss progress, clinicians and con-
sultants perceived varying levels of day-to-day involvement
and enthusiasm. For example, at an agency at which most
clinicians passed, a consultant noted the following.
and excited, and the head of the agency was
very involved in preliminary meetings and was
definitely present. She went to all of the check-
in meetings and then they put a fairly high level
administrator and supervisor in the training... so
that [they] could support CT within the agency.
In contrast, at an agency at which fewer clinicians
passed, a consultant noted, “we were concerned about how
things were unfolding and [the administration] seemed not
particularly concerned... it was problematic for the train-
ing. Without agency buy in, we’re asking a lot of these
therapists....” Administration at agencies where more clini-
cians passed set aside protected time for CT training, invited
board members and other administrators to celebrations
to express appreciation to the participants in the program,
effectively engaged in problem solving, and continued to
allocate time for CT-trained clinicians to meet for ongoing
peer consultation after training was completed.
In contrast, at three agencies, two of which had lower
passing rates, upper level management voiced, and at times
demonstrated, support for the training program, but mid-
with low clinician engagement and low CTRS scores at the
midpoint of the training, consultants attempted to discuss
progress, get feedback, and problem solve with the clinicians.
When this was not effective, they attempted to work with
ed to backfire.
Consultant: [The supervisors] went back and said
[to the clinicians], “well the consultants said you
aren’t doing this right so fix it.” And the result of
us and really angry with us because they thought
that we went over their heads.
The consultant described the upper management’s sub-
sequent efforts to try to support the process, but “We did
not see much result from that. Maybe a little bit more
participation for a week or two but then it kind of died
Perceived Pressure or Mandate to Participate in Training. A
salient issue raised by clinicians and consultants at three
agencies was a mandate, or a perception of a requirement,
by their agency to participate in the training program. One
clinician felt uncomfortable with the feeling of pressure to
participate, but nonetheless found the training helpful.
Consultant: Had I felt free to decline, that would
have been more helpful because I would have
[declined the training], not because I did not
appreciate the training, but because I felt that it
could have been somebody else who might have
In contrast, the mandate to attend the training was par-
ticularly onerous for clinicians whose initial attitudes toward
CT were unfavorable.
Clinician: We were not told, “Hey, do you want to
do this thing? Do you want to learn about CBT?”
We were told, “You are going to this.” So, it was
mandatory, so I really wasn’t on board with that
The apparent mandate from some agencies to complete
CT trainings was also noted by the consultants. For example,
Depression Research and Treatment7
one consultant described it as a “major issue” for one agency
and then described a consultation meeting at which it was
Consultant: That was the top [concern] on their
list-was that they had been required, mandated by
their agency to do this and that that had sort of set
the wrong tone for them, and I had a lot of con-
versations with them about our intentions ... that
it was important that this be voluntary. And they
said that if we went back to their agency and said
that they had admitted that they had been man-
dated to do it that they were afraid that they’d lose
the tone in sort of a negative way to be forced to do
After this experience, individual interviews with clini-
cians were added to assess their interest in the program and
their training needs. Despite these steps, clinicians appear-
ed to perceive pressure to participate at an agency that sub-
sequently participated in the program. As a consultant stated
“We did do the interviews prior to starting ... And I remem-
ber that I personallydid the interview for the person that was
the hardest and she swore up and down how much she loved
CT.” Notably, at two of the three agencies at which clinicians
described a perception of pressure to participate had lower
3.3.3. Experience of Consultation and Feedback on Recordings.
Consistent with program evaluation data on satisfaction,
nearly every clinician stated that recording sessions were
ultimately beneficial. Participants expressed appreciation for
individual feedback, with one clinician stating that “The
consultation with [consultant] was probably one of the
best supervisions that I’ve ever experienced. I was able to
learn things from her and take it and put it into action.”
Many commented that they felt that the feedback, whether
in group or individual format, was supportive and non-
judgmental. Advantages to group feedback that were noted
by consultants and clinicians who participated in the group
format included hearing how their colleagues applied CT to
a variety of cases and supporting one another’s challenges
and success as they learned. Perhaps surprisingly, clinicians
expressed little resistance to group review of their session
recordings. Program evaluation data and interviews also
indicated that more time spent in role plays during
workshops and consultation would have been appreciated,
uncomfortable training experiences if it could improve their
skills. Despite apprehension about playing sessions for peers,
clinicians’ and consultants’ anxiety decreased over time, and
ultimately this supervision format increased their skills. As
one clinician stated that “Peer and consultant feedback is
crucial to my learning. While humbling and at times anxiety
provoking, the feedback is always supportive and gives me
what I am doing well.”
clinicians did not appear to perceive the group feedback
model to be sufficient or supportive.
Consultant: We were getting frustrated because we
would play [examples of CT] that were clearly
different from tapes [a clinician] would play. And
saying that that’s good and what I’m doing is not
good.” And even when the other clinicians would
try to point out the differences, she would just get
really shut down and really annoyed to the point
that I think the other clinicians were afraid to say
anything to her.
A therapist at the same agency stated that if more
individual feedback on his or her own cases had been pro-
vided, the recordings would have felt more useful, “If you’re
a whole session then you do not get the complete feedback
that you need.”
Some clinicians who did pass expressed a desire for
ongoing feedback after training was complete, suggesting
that this aspect of training was particularly valued. They
indicated that they believed that the quality of the treatment
they provided could be improved by ongoing recording and
feedback after training was completed. One clinician, for
example, stated that “I would say when I was getting regular
supervision from [the consultant] and my sessions where
being taped, they were working pretty well; now I think it’s
hit or miss at best to tell you the truth.”
This study reported on the experiences of clinicians, con-
sultants, and administrators during a CT training and
implementation program in an urban behavioral health
that occurred with cohorts within agencies rather than
individually recruited clinicians. This difference allowed us
to investigate the ways in which organizational context may
influence the experience of training. Our findings are illus-
trative of the PARIHS model, which suggests that successful
implementation is a function of perceptions of the interven-
tion and its effectiveness, context, and facilitation , and
they suggest ways in which these three elements may interact
and influence the process of training and consultation.
Feedback and opportunities to discuss session recordings
were identified as important facilitators of clinician skill
and comfort in delivering CT, and administrative support
appearedto play a rolein success.Atagencieswithlower pass
rates, however, pressures that may have been felt by some
administrators (e.g., to encourage attendance at trainings),
consultants (e.g., to facilitate CT skill development), and
clinicians (e.g., to attend trainings despite misgivings about
CT) may have led to patterns of interactions that reduced the
likelihood of successful training outcomes. Perceived man-
dates and agency climate, combined with initial concerns
about CT, may have set up a dynamic in which doubts
8 Depression Research and Treatment
about CT’s structure and its relevance to a given population
resulted in a greater struggle, or less desire, to utilize CT on
the part of the clinicians. The resulting feedback, focusing
on increasing the use of CT interventions and procedures,
may have simply confirmed initial, negative perception of
CT as structured and rigid, and set up a dynamic by which
consultants and clinicians became increasingly discouraged
with the training experience. Future study with a larger
sample is necessary to investigate this hypothesis more
These findings provide a richer explanation of processes
that may underlie previous findings that indicate that agency
context and administrative support can influence clinician
attitudes [45, 46] and the implementation of EBTs . Sup-
protected time for trainings and public acknowledgments of
On the other hand, administrative mandates or pressure to
enroll in training did not appear to be successful, and more
research on the impact of mandates on clinician attitudes
and experience of training is warranted . However, our
results do not suggest that training programs should rule
out potential participants solely because they have initial
doubts about the potential effectiveness of CT. Such doubts
experienced a change of opinion, and ultimately passed
the training. Future research should investigate whether
the addition of an assessment of psychological safety or
organizational social context (c.f., ) prior to the selection
of participating agencies can decrease the likelihood of
pressure or compulsory participation. Addressing organi-
zational context can be a sensitive and challenging issue,
and allowing or encouraging uninterested clinicians to opt
out of consultation requires sensitivity to the organizational
climate, the potential impact on partnerships and relation-
ships with agencies, and clinician morale. For agencies with
more challenging organizational contexts, facilitation may
be insufficient and more intensive organization-level inter-
ventions might be necessary to promote successful outcomes
. In other contexts, particularly after a training program
matures and publicizes success, clinicians may be more likely
to volunteer for training. Since the initial twelve trainings
that were the focus of this report have occurred, more
clinicians in the system have been approaching their agency
administration with requests to participate in the training
Some of our findings may suggest the possibility that a
less structured EBT or a more modular approach would be
better received by some clinicians [50, 51]. However, some
clinicians ultimately viewed session structure to be helpful
for clients. Achieving a balance between emphasis on struc-
ture as a key element of CT and guidance on how to person-
alize treatment and increase clinician comfort appears to
be crucial, yet challenging, particularly in light of concerns
about the treatment that may be present prior to training
[33, 52, 53]. The tendency of some clinicians to “loosen”
the structure of the sessions after training was completed
also warrants attention, as doing so may decrease the
intervention’s effectiveness and ultimately contribute to or
reinforce perceptions that CT is not effective in community
settings. Followup strategies such as ongoing consultation or
fidelity monitoring may be effective methods of supporting
sustained implementation [32, 54].
behavioral health settings and systems. A significant pro-
portion of individuals enrolled in publicly funded mental
health systems are diagnosed with depression . Yet the
strategy of providing a general depression- and suicide-
focused training as a foundation for future, more specialized
trainings appeared to prove unsatisfying to some clinicians.
Interviews with consultants and clinicians revealed that they
perceived a need for training to address challenges related to
severe mental illness, comorbidities, and diagnostically com-
plicated clients. In response to stakeholder feedback on the
training program, considerable effort was devoted to find-
ing video clips of individuals who resembled the popula-
tions served by the agencies. Professionally produced video
materials did not meet all of the needs expressed by training
participants, but some videos of clients from clinical trials
who agreed to allow their case material to be shared for
training purposes were identified. However, to meet stake-
holder needs, the training program has also expanded to
include more specialized training programs for psychosis,
substance dependence, and personality disorders.
for depression and suicide. Most viewed the review of
session recordings as a critical component of their training.
This finding is particularly important because some large-
scale EBT training programs do not include the use of
session recordings . Further investigation is warranted
to determine whether this component improves training
outcomes over and above consultation without review of
work samples. Clinicians reported appreciation for feed-
back, including group-based feedback on recorded sessions,
despite some initial discomfort with sharing their work in a
group setting. While the mean rating for confidence in using
CT appeared to be higher among clinicians who participated
in group consultation sessions than for those in individual
consultation, the sample size and nesting of clinicians within
cohorts precluded an appropriate statistical comparison
between groups. It is possible that there are some benefits
to a group feedback model, such as cost efficiency, exposure
to examples of the use of CT with a wider variety of
cases, and supportive feedback that may increase confidence.
Additionally, after expert consultation is complete, ongoing
peer group consultation may help clinicians maintain their
adherence to CT. However, as some clinicians noted, the
back. Future comparisons of the two consultation models in
terms of costs, fidelity, and client outcomes would be useful
in determining the most advantageous models.
Findings from this study should be interpreted in light
of several limitations. Our interview sample was relatively
small, and individuals with stronger or more extreme
opinions may have been more likely to participate. We
attempted to mitigate this limitation by coding anonymously
Depression Research and Treatment9
provided program evaluation data as well. This study was
not designed to test hypotheses, but rather to inform further
investigation of potential determinants of successful training
outcomes, and to reveal potential reasons for the variations
in the numbers of clinicians who passed at different agencies.
Despite these limitations, we identified several directions for
future research. The relative benefits and disadvantages of
providing training in evidence-based interventions to clin-
icians within their organizational setting should be further
explored. It will be important to test strategies to address
organizational climates that are less favorable for training
and implementation. Additional research is needed to deter-
mine whether feedback on work samples enhances clinician
skills over and above consultation alone.
The authors would like to acknowledge the assistance of
Dr. Matthew Hurford, Dr. J. Bryce McLaulin, and Regina
Xhezo, M. S. in facilitating the training program, and they
extend their thanks to the clinicians and consumers who
participated in the program. The authors also wish to thank
Andrea DeVito and Sam Meisel for their assistance with
this project. Dr. Arthur C. Evans did not have access to the
study participant identities or to the original, nonaggregated
data discussed in this article. The preparation of this article
was supported through funding from the National Institute
of Mental Health (R00 MH 080100). At the time that this
research occurred, Dr. Stirman was a Fellow at the Imple-
mentation Research Institute (IRI), at the George Warren
Brown School of Social Work, Washington University in St.
Louis, which is funded through an award from the National
Institute of Mental Health (R25 MH080916-01A2) and the
Department of Veterans Affairs, Health Services Research
& Development Service, Quality Enhancement Research
Initiative (QUERI). The content is solely the responsibility
of the authors and does not necessarily represent the official
Institutes of Health, or the Department of Veterans Affairs.
 I. M. Blackburn, K. M. Eunson, and S. Bishop, “A two-year
naturalistic follow-up of depressed patients treated with cog-
nitive therapy, pharmacotherapy and a combination of both,”
Journal of Affective Disorders, vol. 10, no. 1, pp. 67–75, 1986.
 J. Johnson, M. M. Weissman, and G. L. Klerman, “Service
utilization and social morbidity associated with depressive
Association, vol. 267, no. 11, pp. 1478–1483, 1992.
 R. Sturm and K. B. Wells, “How can care for depression be-
come more cost-effective?” Journal of the American Medical
Association, vol. 273, no. 1, pp. 51–58, 1995.
ity during the long-term course of unipolar major depressive
disorder,” Archives of General Psychiatry, vol. 57, no. 4, pp.
sus a diagnosis of depression: differences in psychosocial func-
tioning,” Journal of Consulting and Clinical Psychology, vol.
63, no. 1, pp. 90–100, 1995.
 A. T. Beck, Cognitive Therapy and the Emotional Disorders,
Penguin Books, New York, NY, USA, 1979.
 C. D. Sherbourne, K. B. Wells, N. Duan et al., “Long-term ef-
fectiveness of disseminating quality improvement for depres-
sion in primary care,” Archives of General Psychiatry, vol. 58,
no. 7, pp. 696–703, 2001.
 A. D. Simons, C. A. Padesky, J. Montemarano et al., “Training
and dissemination of cognitive behavior therapy for depres-
sion in adults: a preliminary examination of therapist compe-
tence and client outcomes,” Journal of Consulting and Clinical
Psychology, vol. 78, no. 5, pp. 751–756, 2010.
 G. K. Brown, T. Ten Have, G. R. Henriques, S. X. Xie, J. E.
Hollander, and A. T. Beck, “Cognitive therapy for the preven-
tion of suicide attempts: a randomized controlled trial,” Jour-
nal of the American Medical Association, vol. 294, no. 5, pp.
 M. M. Weissman, H. Verdeli, M. J. Gameroff et al., “National
survey of psychotherapy training in psychiatry, psychology,
and social work,” Archives of General Psychiatry, vol. 63, no.
8, pp. 925–934, 2006.
 R. K. McHugh and D. H. Barlow, “The dissemination and
implementation of evidence-based psychological treatments.
A review of current efforts,” American Psychologist, vol. 65, no.
2, pp. 73–84, 2010.
 B. E. Karlin, J. I. Ruzek, K. M. Chard et al., “Dissemination
of evidence-based psychological treatments for posttraumatic
of Traumatic Stress, vol. 23, no. 6, pp. 663–673, 2010.
 S. D. Hollon, R. J. DeRubeis, R. C. Shelton et al., “Prevention
of relapse following cognitive therapy vs medications in
moderate to severe depression,” Archives of General Psychiatry,
vol. 62, no. 4, pp. 417–422, 2005.
 A. T. Beck, “The current state of cognitive therapy: a 40-year
retrospective,” Archives of General Psychiatry, vol. 62, no. 9, pp.
 M. Schoenbaum, J. Un¨ utzer, C. Sherbourne et al., “Cost-
effectiveness of practice-initiated quality improvement for
the American Medical Association, vol. 286, no. 11, pp. 1325–
health treatment procedures: maximizing the return on a
significant investment,” Behaviour Research and Therapy, vol.
47, no. 11, pp. 990–993, 2009.
 K. M. Carroll, S. Martino, and B. J. Rounsaville, “No train, no
gain?” Clinical Psychology, vol. 17, no. 1, pp. 36–40, 2010.
 J. S.Baer, D. B.Rosengren, C.W. Dunn, E.A.Wells, R.L. Ogle,
and B. Hartzler, “An evaluation of workshop training in
motivational interviewing for addiction and mental health
clinicians,” Drug and Alcohol Dependence, vol. 73, no. 1, pp.
 D. E. Sholomskas, G. Syracuse-Siewert, B. J. Rounsaville, S. A.
Ball, K. F. Nuro, and K. M. Carroll, “We don’t train in vain:
a dissemination trial of three strategies of training clinicians
in cognitive-behavioral therapy,” Journal of Consulting and
Clinical Psychology, vol. 73, no. 1, pp. 106–115, 2005.
 W. R. Miller, C. E. Yahne, T. B. Moyers, J. Martinez, and M.
Pirritano, “A randomized trial of methods to help clinicians
10 Depression Research and Treatment
learn motivational interviewing,” Journal of Consulting and
Clinical Psychology, vol. 72, no. 6, pp. 1050–1062, 2004.
 R. S. Beidas, K. Koerner, K. R. Weingardt, and P. C. Kendall,
“Training research: practical recommendations for maximum
impact,” Administration and Policy in Mental Health and Men-
tal Health Services Research, vol. 38, no. 4, pp. 223–237, 2011.
 R. S. Beidas and P. C. Kendall, “Training therapists in evi-
dence-based practice: a critical review of studies from a sys-
tems-contextual perspective,” Clinical Psychology, vol. 17, no.
1, pp. 1–30, 2010.
 A. D. Herschell, D. J. Kolko, B. L. Baumann, and A. C.
psychosocial treatments: a review and critique with recom-
mendations,” Clinical Psychology Review, vol. 30, no. 4, pp.
 R. Beidas, “An RCT of training and consultation as imple-
mentation strategies for an empirically supported treatment,”
Psychiatric Services, vol. 63, no. 7, pp. 660–665, 2012.
 M. B. Connolly Gibbons, A. Rothbard, K. D. Farris et al.,
“Changes in psychotherapy utilization among consumers of
services for major depressive disorder in the community men-
tal health system,” Administration and Policy in Mental Health
and Mental Health Services Research, vol. 38, no. 6, pp. 495–
 A. T. Beck, A. J. Rush, B. F. Shaw, and G. Emery, Cognitive
Therapy for Depression, Guilford Press, New York, NY, USA,
“Therapist competence in cognitive therapy for depression:
and Clinical Psychology, vol. 78, no. 3, pp. 429–437, 2010.
 B. F. Shaw, I. Elkin, J. Yamaguchi et al., “Therapist competence
ratings in relation to clinical outcome in cognitive therapy of
depression,” Journal of Consulting and Clinical Psychology, vol.
67, no. 6, pp. 837–846, 1999.
rationale and homework compliance in cognitive-behavioral
therapy for depression,” Cognitive Therapy and Research, vol.
24, no. 3, pp. 313–326, 2000.
 M. E. Thase and J. A. Callan, “The role of homework in cogni-
tive behavior therapy of depression,” Journal of Psychotherapy
Integration, vol. 16, no. 2, pp. 162–177, 2006.
 S. W. Stirman, R. Buchhofer, J. B. McLaulin, A. C. Evans, and
A. T. Beck, “The Beck Initiative: a partnership to implement
cognitive therapy in a community behavioral health system,”
Psychiatric Services, vol. 60, no. 10, pp. 1302–1304, 2009.
 S. W. Stirman, S. S. Bhar, M. Spokas et al., “Training and con-
sultation in evidence-based psychosocial treatments in public
mental health settings: the access model,” Professional Psychol-
ogy, vol. 41, no. 1, pp. 48–56, 2010.
 S.W.Stirman,A.Guti´ errez-Colina,K.Toderetal.,“Clinicians’
settings: implications for training and implementation,”
Administration and Policy in Mental Health and Mental Health
Services Research. In press.
 A. R. Lyon, S. W. Stirman, S. E. U. Kerns, and E. J. Bruns,
“Developing the mental health workforce: review and applica-
istration and Policy in Mental Health and Mental Health
Services Research, vol. 38, no. 4, pp. 238–253, 2011.
 C. B. Stetler, M. W. Legro, J. Rycroft-Malone et al., “Role of
“external facilitation” in implementation of research findings:
a qualitative evaluation of facilitation experiences in the
Veterans Health Administration,” Implementation Science, vol.
1, no. 1, article 23, 2006.
facilitation to implement cognitive behavioral therapy in VA
clinics: a pilot study,” Implementation Science, vol. 5, no. 1,
article 75, 2010.
 C. D. Helfrich, Y. F. Li, N. D. Sharp, and A. E. Sales, “Orga-
nizational readiness to change assessment (ORCA): develop-
ment of an instrument based on the promoting action on
research in health services (PARIHS) framework,” Implemen-
tation Science, vol. 4, no. 1, article 38, 2009.
 C. B. Stetler, M. W. Legro, C. M. Wallace et al., “The role
of formative evaluation in implementation research and the
QUERI experience,” Journal of General Internal Medicine, vol.
21, supplement 2, pp. S1–S8, 2006.
 L. Jones and K. Wells, “Strategies for academic and clinician
engagement in community-participatory partnered research,”
 J. Young and A. T. Beck, Cognitive Therapy Rating Scale
Manual, University of Pennsylvania, 1980.
 B. F. Shaw and K. S. Dobson, “Competency judgments in the
training and evaluation of psychotherapists,” Journal of Con-
sulting and Clinical Psychology, vol. 56, no. 5, pp. 666–672,
 C. B. Stetler, L. J. Damschroder, C. D. Helfrich, and H. J.
Hagedorn, “A Guide for applying a revised version of the PAR-
IHS framework for implementation,” Implementation Science,
vol. 6, no. 1, article 99, 2011.
 B. G. Glauser and A. L. Strauss, The Discovery of Grounded
Theory: Strategies for Qualitative Research, Aldine Publishing,
Chicago, Ill, USA, 1967.
 T. Muhr, User’s Manual for ATLAS.ti 5.0, ATLAS.ti Scientific
Software Development, Berlin, Germany, 2004.
 G. A. Aarons, “Transformational and transactional leadership:
association with attitudes toward evidence-based practice,”
Psychiatric Services, vol. 57, no. 8, pp. 1162–1169, 2006.
 G. A. Aarons and A. C. Sawitzky, “Organizational culture and
climate and mental health provider attitudes toward evidence-
based practice,” Psychological Services, vol. 3, no. 1, pp. 61–72,
 C. Glisson, J. Landsverk, S. Schoenwald et al., “Assessing the
Organizational Social Context (OSC) of mental health ser-
vices: implications for research and practice,” Administration
and Policy in Mental Health and Mental Health Services Re-
search, vol. 35, no. 1-2, pp. 98–113, 2008.
 A. Jensen-Doss, “Practice involves more than treatment: how
can evidence-based assessment catch up to evidence-based
treatment?” Clinical Psychology, vol. 18, no. 2, pp. 173–177,
 C. Glisson and S. K. Schoenwald, “The ARC organizational
and community intervention strategy for implementing evi-
dence-based children’s mental health treatments,” Mental
Health Services Research, vol. 7, no. 4, pp. 243–259, 2005.
 C. F. Borntrager, B. F. Chorpita, C. Higa-McMillan, and J. R.
Weisz, “Provider attitudes toward evidence-based practices:
are the concerns with the evidence or with the manuals?”
Psychiatric Services, vol. 60, no. 5, pp. 677–681, 2009.
 J. R. Weisz, B. F. Chorpita, L. A. Palinkas et al., “Testing stan-
dard and modular designs for psychotherapy treating depres-
sion, anxiety, and conduct problems in youth: a randomized
Depression Research and Treatment11 Download full-text
effectiveness trial,” Archives of General Psychiatry, vol. 69, no.
3, pp. 274–282, 2012.
 E. W. L. Smith, “A passionate, rational response to the “man-
ualization” of psychotherapy,” Psychotherapy Bulletin, vol. 33,
no. 2, pp. 36–40, 1995.
 W. H. Silverman, “Cookbooks, manuals, and paint-by-num-
bers: psychotherapy in the 90’s,” Psychotherapy, vol. 33, no. 2,
pp. 207–215, 1996.
 G. A. Aarons, D. H. Sommerfeld, D. B. Hecht, J. F. Silovsky,
and M. J. Chaffin, “The impact of evidence-based practice
implementation and fidelity monitoring on staff turnover:
evidence for a protective effect,” Journal of Consulting and
Clinical Psychology, vol. 77, no. 2, pp. 270–280, 2009.