Supervisor variance in psychotherapy outcome in routine practice

Abstract

Objective:
Although supervision has long been considered as a means for helping trainees develop competencies in their clinical work, little empirical research has been conducted examining the influence of supervision on client treatment outcomes. Specifically, one might ask whether differences in supervisors can predict/explain whether clients will make a positive or negative change through psychotherapy.

Method:
In this naturalistic study, we used a large (6521 clients seen by 175 trainee therapists who were supervised by 23 supervisors) 5-year archival data-set of psychotherapy outcomes from a private nonprofit mental health center to test whether client treatment outcomes (as measured by the OQ-45.2) differed depending on who was providing the supervision. Hierarchical linear modeling was used with clients (Level 1) nested within therapists (Level 2) who were nested within supervisors (Level 3).

Results:
In the main analysis, supervisors explained less than 1% of the variance in client psychotherapy outcomes.

Conclusions:
Possible reasons for the lack of variability between supervisors are discussed.

Full text

This article was downloaded by: [University of Alaska Fairbanks], [Jason Whipple]
On: 06 October 2014, At: 10:18
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK
Psychotherapy Research
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/tpsr20
Supervisor variance in psychotherapy outcome in
routine practice
Tony G. Rousmanierea, Joshua K. Swiftb, Robbie Babins-Wagnerc, Jason L. Whippled & Sandy
Berzinsc
a Student Health and Counseling, University of Alaska Fairbanks, Fairbanks, AK, USA
b Department of Psychology, University of Alaska Anchorage, Anchorage, AK, USA
c Calgary Counseling Center, Calgary, AB, Canada
d Department of Psychology, University of Alaska Fairbanks, Fairbanks, AK, USA
Published online: 02 Oct 2014.
To cite this article: Tony G. Rousmaniere, Joshua K. Swift, Robbie Babins-Wagner, Jason L. Whipple & Sandy
Berzins (2014): Supervisor variance in psychotherapy outcome in routine practice, Psychotherapy Research, DOI:
10.1080/10503307.2014.963730
To link to this article: http://dx.doi.org/10.1080/10503307.2014.963730
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions

EMPIRICAL PAPER
Supervisor variance in psychotherapy outcome in routine practice
TONY G. ROUSMANIERE
1
, JOSHUA K. SWIFT
2
, ROBBIE BABINS-WAGNER
3
,
JASON L. WHIPPLE
4
, & SANDY BERZINS
3
1
Student Health and Counseling, University of Alaska Fairbanks, Fairbanks, AK, USA;
2
Department of Psychology,
University of Alaska Anchorage, Anchorage, AK, USA;
3
Calgary Counseling Center, Calgary, AB, Canada &
4
Department
of Psychology, University of Alaska Fairbanks, Fairbanks, AK, USA
(Received 19 December 2013; revised 22 August 2014; accepted 1 September 2014)
Abstract
Objective: Although supervision has long been considered as a means for helping trainees develop competencies in their
clinical work, little empirical research has been conducted examining the influence of supervision on client treatment
outcomes. Specifically, one might ask whether differences in supervisors can predict/explain whether clients will make a
positive or negative change through psychotherapy. Method: In this naturalistic study, we used a large (6521 clients seen by
175 trainee therapists who were supervised by 23 supervisors) 5-year archival data-set of psychotherapy outcomes from a
private nonprofit mental health center to test whether client treatment outcomes (as measured by the OQ-45.2) differed
depending on who was providing the supervision. Hierarchical linear modeling was used with clients (Level 1) nested within
therapists (Level 2) who were nested within supervisors (Level 3). Results: In the main analysis, supervisors explained less
than 1% of the variance in client psychotherapy outcomes. Conclusions: Possible reasons for the lack of variability between
supervisors are discussed.
Keywords: supervision; training; psychotherapy outcome
Clinical supervision is widely regarded as an essential
component of psychotherapy training (Bernard &
Goodyear, 2014). The primary goals of supervision
cover two broad domains: facilitating the professional
development of supervisees and enhancing/protecting
the welfare of psychotherapy clients (Falender &
Shafranske, 2004). The impact of supervision on the
first domain (i.e., supervisees) has been well docu-
mented. For example, effective supervision has
been associated with increased supervisee self-efficacy
(Gibson, Grey, & Hastings, 2009), decreased super-
visee anxiety (Inman et al., 2014), skill acquisition
(e.g., Lambert & Arnold, 1987), encouraged super-
visee autonomy and increased supervisee openness
(Ladany, Mori, & Mehr, 2012), and reduced confu-
sion about professional roles (Ladany & Friedlander,
1995). Likewise, the effects of inadequate or harmful
supervision include increased supervisee anxiety
(Gray, Ladany, Walker, & Ancis, 2001), decreased
supervisee self-disclosure (Mehr, Ladany, & Caskie,
2010), increased multicultural misunderstandings
(Ladany & Inman, 2012), and even causing super-
visees to drop out of the field entirely (Ellis, 2001;see
also Ellis et al., 2013). Thus, it can be said with some
confidence that supervisors can have both positive
and negative effects on supervisees.
The impact supervisors have on clients is less
clear and has been the subject of some debate (e.g.,
Holloway & Neufeldt, 1995; Ladany & Inman, 2012;
Watkins, 2011). In their examination of the theoret-
ical connection between supervisors and client out-
come, Wampold and Holloway (1997) suggested that
the wide range of therapist characteristics that can
influence psychotherapy outcomes may also apply to
supervisors (e.g., therapeutic style and emotional
well-being; see Beutler, Machado, & Neufeldt,
1994). Additionally, characteristics specific to super-
visors may affect client outcome, such as experience
Correspondence concerning this article should be addressed to Tony G. Rousmaniere, Student Health and Counseling, University of Alaska
Fairbanks, 612 N Chandelar Dr, Fairbanks, AK 99775, USA. Email: trousmaniere@gmail.com
Psychotherapy Research, 2014
http://dx.doi.org/10.1080/10503307.2014.963730
© 2014 Society for Psychotherapy Research
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

level as a supervisor and supervisory competence.
However, Wampold and Holloway (1997) also raised
significant doubts about the impact supervisors may
have on client outcome due to the many potentially
mediating variables at the supervisor, therapist, and
client levels that separate supervisors from client
outcome, “Detection of a relation between supervi-
sion process and the patient’s rating of patient change
(the most distal outcome) would be expected to be
extremely small”(Wampold & Holloway, 1997,
p. 23).
Despite this debate, improving client welfare is
commonly viewed as the raison d’etre for clinical
training, and thus, there have been numerous calls
for examining this topic (e.g., Holloway & Carroll,
1996; Lambert & Arnold, 1987). Indeed, the impact
of supervision on client psychotherapy outcome has
been called the “acid test”of good supervision
(Ellis & Ladany, 1997, p. 485). Three reviews of
the supervision literature have examined this topic,
with mixed results. In a review of 10 studies, Freitas
(2002) found some relationship between supervi-
sion and client treatment outcome. However, those
studies were all also found to have methodological
problems significant enough to raise questions
about the validity of their findings (e.g., use of
outcome measures with poor psychometric proper-
ties and nonrandom assignment of participants). In
a review of 11 studies, Milne, Sheikh, Pattison, and
Wilkinson (2011) found that “the blend of train‐
ing and supervisory methods …were effective in
facilitating supervisor and supervisee (therapist)
development, which …was associated with patient
benefits”(pp. 61–62). However, Milne et al. (2011)
noted that only 2 of the 11 studies in the re-
view examined the direct effects of supervision on
client psychotherapy outcome, so the “clinical
outcome estimate should be treated with great
caution”(p. 62). Likewise, in his review of the
literature, Watkins (2011) noted that most studies
that explored the impact of supervision on client
psychotherapy outcome relied on supervisors’or
supervisees’perceptions, rather than client self-
reports and psychometrically sound measures.
Regarding the supervisor-to-client treatment out-
come connection, Watkins concludes, “After a
century of psychotherapy supervision and over half
a century of supervision research, we still cannot
empirically answer that question”(p. 252).
Perhaps the most frequently cited study on the
direct effects of supervision on treatment outcome is
Bambling, King, Raue, Schweitzer, and Lambert
(2006). This study had a randomized that of experi-
mental design, wherein 127 clients with a primary
diagnosis of major depressive disorder were ran-
domly assigned to 127 licensed therapists for eight
sessions of problem-solving treatment. Half of the
therapists were split into two groups who received
weekly supervision with either a process-focus or
working alliance skill-focus, based on a supervision
manual developed for this study; the other half of the
therapists received no supervision. After eight weeks
of treatment, clients receiving therapy from thera-
pists in the supervision group had significantly higher
scores on both the Working Alliance Inventory
(WAI; Horvath & Greenberg, 1989) and Beck
Depression Inventory (BDI; Beck, Steer, & Garbin,
1987), than clients in the no-supervision group.
Notably, clients in the two supervision groups had
much lower rates of noncompletion at eight sessions
(3.0% and 6.1%) than clients in the no-supervision
group (35%). Data from this study thus suggest that
supervision may contribute to client outcome, at
least in controlled experimental conditions, when
compared to licensed clinicians not receiving super-
vision. However, trainees cannot practice without
supervision, so the implications of these findings to
psychotherapy training (at least at the prelicensure
stage) are unclear. Additionally, the authors acknow-
ledge the possibility that therapist allegiance effects
or working alliance effects (WAI scores were tightly
correlated with BDI scores) may account for some of
the findings (Bambling et al., 2006).
Reese et al. (2009) performed a controlled study
in which the outcomes of trainees receiving supervi-
sion that included regular outcome feedback (n=9)
were compared to the outcomes of trainees receiving
supervision without regular outcome feedback
(n= 10). Data included 115 psychotherapy cases
collected over the course of year. Trainees in the
supervision-with-feedback condition had signifi-
cantly better outcomes than trainees receiving super-
vision without feedback. Notably, no significant
differences were found between supervisors within
the treatment conditions (Reese et al., 2009).
Another approach to this question is to examine
whether clients vary in their psychotherapy out-
comes, depending on who is assigned to supervise
the case. Analogous to how psychotherapy research
has been able to identify “supershrinks”(Okiishi,
Lambert, Nielsen, & Ogles, 2003; Miller, Hubble, &
Duncan, 2008), it may be possible to identify
“super-supervisors,”based on their supervisees’out-
comes. Callahan, Almstrom, Swift, Borja, and Heath
(2009) took this approach to examining the effects of
supervision on client outcome using naturalistic
archival data from a psychology department training
clinic that tracked client outcome as part of routine
practice with two measures: the Symptom Checklist-
90, Revised (Derogatis, 1992) and the Beck Depres-
sion Inventory-II (BDI-II; Beck, Steer, & Brown,
1996). Participants in their study were 76 adult
2T. G. Rousmaniere et al.
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

psychotherapy clients who were randomly assigned
to 40 trainee therapists who were in supervision with
nine supervisors. Supervisors were found to have
a significant effect on client treatment outcome,
accounting for 16% of the variance. However, the
study’s small sample size (the mean number of cases
per supervisor was eight, and six supervisors had five
or fewer cases) makes these findings difficult to
reliably generalize.
Thus, there is preliminary evidence from both
experimental and naturalistic data that supervisors
may influence or at least predict client outcome.
However, the existing research is limited by consist-
ing of only a few studies with relatively small sample
sizes. Thus, this topic warrants further study. The
purpose of our current study was to further explore
the variance of impact of supervisors on client
psychotherapy outcome in a naturalistic training
setting by using a large data-set and more sensitive
statistical procedures (hierarchical linear modeling,
HLM) than have been used in previous studies.
Specifically, our goal was to examine the amount of
variance in client psychotherapy outcome accounted
for by supervisors, in a manner similar to recent
meta-analytic research examining the factors con-
tributing to psychotherapy outcome (e.g., Wampold,
2010). If supervisors were found to explain a signi-
ficant amount of variance, we then planned examin-
ing whether a set of supervisor level variables
(supervisor demographics, experience, and type)
could predict the amount of change that clients
made through psychotherapy.
Method
Participants
Archival data from a large private nonprofit,
community-based counseling center in Western
Canada were used for this study. The counseling
center tracks the clinical outcome of all clients, on a
session-by-session basis. A total of 7929 adult clients
were seen in the counseling center by supervised
trainee therapists over a 5-year period. However, 886
of those clients had no OQ data for any of their
sessions and 392 were missing an intake OQ score,
and thus were removed from the data file. For those
who were missing a last session OQ score (n= 759),
we used a last observation carried forward method
(the last OQ score that was recorded was used as
their end score). In order to ensure that therapist/
supervisor case averages were not based on a single
client/supervisee, only therapists who saw at least
two clients and supervisors who provided supervi-
sion for at least two therapists were retained in our
data-set. These minimum criteria resulted in the
removal of 20 of the 195 therapists and 3 of the 26
supervisors. The final sample included 23 super-
visors, 175 therapists, and 6562 clients.
Clients. Over half (57.0%) of the client partici-
pants were females, 42.5% were males, and 0.5%
did not declare their gender. Of the client partici-
pants, 33% were married or in a common-law
relationship, 15.9% were separated or divorced,
43.4% were single, and 7.6% were other or did not
declare their relationship status. For education,
32.8% of the client participants had some high
school or less, 34.2% had a college or technical
school degree, 30.4% had a university degree, and
2.5% had other or did not declare their level of
education. For employment, 70.6% of client partici-
pants who indicated an employment status worked
full-time, 11.6% worked part-time, 6.8% were stu-
dents, 1.6% were retired, 3.7% were on disability
leave, and 5.8% were other. The average age for
participants was 37.72, SD = 11.24, ranging from
18 to 92 years. Clients presented with a range of
concerns typical for community mental health cen-
ters, including family/marital problems (45.9%) and
personal functioning problems, including stress,
depression, eating disorders, and anxiety disorders
(51.7%), occupational/vocational issues, including
job satisfaction, work conflict, and career path
choices (1.1%), and social/community concerns,
including social isolation, lifestyle choices, financial,
or legal problems (1.3%) as their primary concern.
Most (83.0%) of the clients were provided individual
counseling, and 17% were provided couples’coun-
seling. The average intake OQ score for these clients
was 71.33, SD = 26.32, and they attended an
average of 4.81 sessions (SD = 5.36), ranging from
1 to 92.
Therapists. There were two categories of super-
visees (n= 175) in this data: practicum students and
residents. Practicum supervisees were students who
were still completing their master’s level courses in
social work, psychology, marriage and family therapy,
and pastoral counseling. Resident supervisees had
graduated with a master’s degree in psychology, social
work, or marriage and family therapy and were
obtaining hours toward their registration as a licensed
practitioner. Supervisees were ages 23–mid-50s. Prac-
ticum supervisees were randomly matched with a
supervisor for a single 8-month assignment. Resident
supervisees were randomly matched with a supervisor
for a single 1-year assignment. Supervisor degree level
was not a factor in trainee assignment: both practicum
and resident supervisees could be assigned to masters-
or doctoral-level supervisors. A few (<5%) residents
continued working at the center for additional years.
Psychotherapy Research 3
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

Both practicum and resident supervisees received one
hour of individual and two hours of group supervision
per week, from the same supervisor. All therapists had
only one supervisor at a time. The included therapists
treated an average of 37.50 (SD = 43.48) clients.
Attempts were made to obtain additional demographic
data on therapists. However, the data were drawn
from archival records, and the supervisees in the data
have since left the center, so additional demographic
data were not available.
Supervisors. The included supervisors (n= 23)
supervised an average of 11.96, SD = 9.18, therapists
and an average of 285.30, SD = 349.10, clients. Most
(69.57%) of the supervisors were female, and 30.43%
were male. About half (47.48%) of thesupervisors had
MS Psych degrees, 30.43% had MSW degrees, and
21.74% had Ph.D. degrees. Supervisors were of ages
28–64 years and 60.87% had 1–5 years of supervisory
experience, 17.39% had 6–10 years, 8.70% had 11–15
years, 8.70% had 16–20 years, and 4.35% had over 20
years of supervisory experience. Supervisors at the
center are required to work from a therapeutic mod-
ality, but the choice of modality is left to the super-
visor. The supervisors in this data-set practiced a
wide range of modalities (e.g., cognitive-behavioral,
psychodynamic, solution-focused, family systems,
Ericksonian, and strategic). Therapeutic modalities
were roughly evenly distributed among supervisors,
with no single modality used by more than 20% of
supervisors. All supervisors had taken at least one
academic course in supervision, and a few had taken
multiple courses. Attempts were made to obtain
additional demographic data on supervisors. How-
ever, the data were drawn from archival records, and
approximately 65% of the supervisors have since left
the center, so additional demographic data were not
available.
Measure
The self-report OQ–45.2 (Lambert et al., 1996) was
used as the outcome measure in this study. The OQ-
45.2 was designed to help therapists monitor clients’
progress in therapy via weekly administrations over
the course of treatment. The OQ-45.2 is comprised
of 45 items. Each item is rated on a 5-point Likert
scale; high scores indicate more disturbance. The
items on the OQ-45.2 can also be divided into three
subscale scores: Subjective Discomfort (intrapsychic
functioning, e.g., “I feel blue”), Interpersonal Rela-
tionships (e.g., “I feel lonely”), and Social Role
Performance (e.g., “I feel stressed at work/school”).
The score of all items can be combined for a total
score of 0–180. For this study, only the OQ-45.2
total score was used. The OQ-45.2 is regarded to
have adequate psychometric properties: The manual
reports an internal consistency of .93, a test–retest
reliability of .87, and high concurrent validity with a
number of other measures, including the Symptom
Checklist-90-Revised, Beck Depression Inventory,
State-Trait Anxiety Inventory, Inventory of Interper-
sonal Problems, and Social Adjustment Scale
(Lambert et al., 2004). The OQ-45.2 has demon-
strated sensitivity to change for clients in treatment,
while remaining stable for people not in treatment
(Vermeersch, Lambert, & Burlingame, 2000), and
the manual reports no differences based on gender
(Lambert et al., 2004).
Procedure
Therapists and clients conducted therapy as usual
throughout the period that archival data were col-
lected. Most (>95%) of clients were randomly
assigned to therapists (supervisees). Less than 5%
of clients were assigned to specific therapists due to
specific requests by the client (e.g., pastoral counsel-
ing). Clients paid $1 to over $160 per session for
psychotherapy. Some (<15%) clients paid for coun-
seling with private insurance. All adult clients at the
center complete the OQ-45.2 prior to every psycho-
therapy session, as part of routine treatment. Thera-
pists and supervisors were able to review clients’
OQ-45.2 scores throughout the course of therapy.
All clients and their resultant data were treated in
accordance with the Ethical Principles of Psycholo-
gists and Code of Conduct.
Data Analysis Plan
In this study, we were interested in testing whether
or not client outcomes differed between the included
supervisors. Given that clients were nested within
therapists, and therapists were nested within super-
visors, HLM with maximum likelihood estimation
was utilized to test for variance at both of these
levels. The baseline model predicting client OQ
change scores (defined as the difference between
start OQ and end OQ scores) included therapists
at Level 2, and supervisors at Level 3, with the
intercepts modeled as random effects. The Wald
statistic was used to test the variance at the therapist
and supervisor levels. Additionally, interclass corre-
lations (ICC) were calculated in order to identify the
amount of variance in client OQ change scores that
was explained by therapists and supervisors. One
might hypothesize that the impact of the supervisor
partially depends on the experience level of the
supervisee (i.e., supervisors may play a bigger
role with less-experienced supervisees and thus
more supervisor variability would be seen with a
4T. G. Rousmaniere et al.
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

sample of less-experienced therapists). Thus, we
next tested this interaction hypothesis by entering
the therapist type (practicum students vs. residents)
at Level 2 and comparing a model where supervisor
variance was held constant across therapist type (the
original baseline model) to a model where supervisor
variance was allowed to vary depending on the
therapist type. Last, we were interested in examining
whether supervisor characteristics could predict cli-
ent outcomes. Supervisor field (social work vs.
psychology), level of education (MS vs. Ph.D.),
and years of experience providing supervision were
added to the baseline model as fixed effects. Com-
parison of the −2LL values between models was
used to determine if this model was a better fit for
the data compared to the baseline model. Although
other variables could have been added at Levels 1
and 2 that may have explained a significant amount
of variance in client OQ change scores, given the
focus of this manuscript, we chose to only test Level
3 (supervisor) predictors. In running these analyses,
we removed 41 cases (14 who demonstrated extreme
negative change and 27 who demonstrated extreme
positive change) that were determined to be outliers
(zvalue >3.5) on the main outcome variable (OQ
change scores).
Results
On average, the 6521 client participants improved by
M= 8.81, SD = 17.30, points on the OQ-45.2 over
the course of their treatment, ranging from an
improvement of 73 points to a deterioration of 55
points. At the therapist level, therapists on average
saw a M= 8.26, SD = 5.40, point change for their
clients, ranging from an average improvement of
27.5 points for one therapist to an average deteri-
oration of 11.33 points for another therapist. At the
supervisor level, supervisors on average saw a
M= 8.72, SD = 2.52, point change for their clients,
ranging from an average improvement of 14.00 for
one supervisor to an average improvement of 2.21
points for another supervisor. Figure 1 plots the
supervisors’means for their therapists’and clients’
average change. Overall, 4.9% of the sample dete-
riorated (OQ score showed a 14 point or greater
increase from treatment from start to end), 64.9%
displayed no change on the OQ (less than 14 point
increase or decrease from start to end), 8.4% made a
reliable improvement (14 point or greater decrease
from start to end) but were still in the clinical range
(64 or greater) upon terminating therapy, and 21.9%
made a clinically significant change by the end of
treatment (14 point or greater decrease and an end
OQ score in the nonclinical range). Percentages of
client deterioration, no change, reliable improvement,
and clinically significant change are reported sepa-
rately for each supervisor in Table I.
Parameter estimates and −2LL values for the
different models that we tested can be found in
Table II. The first baseline model was used to
determine whether therapists (Level 2) and super-
visors (Level 3) differ among themselves in average
client outcomes. In this baseline model, the esti-
mated variance between therapists was significant,
estimated variance = 2.51, 95% CI [1.13, 5.57], Wald
Z= 2.46, p< .05; however, the estimated variance
between supervisors was not, estimated variance =
0.12, 95% CI [0.00, 19.64], Wald Z = 0.38, p> .05.
Calculation of ICCs indicated that only a small
amount of variance (0.84%) in client OQ change
scores was accounted for by differences between
therapists, and virtually no variance (0.04%) in client
OQ change scores was accounted for by differences
between supervisors.
In order to test for an interaction between super-
visor variance and therapist type (residents vs.
interns), in this second model, we allowed the
supervisor variance to vary between the two therapist
types. A comparison of the −2LL values between the
models where supervisor variance was held constant
across therapist type (−2LL = 55,670.90) and the
model where supervisor variance was allowed to
differ depending on the therapist type (−2LL =
55,671.09) indicated that the second model did not
fit the data any better than the first, χ
2
= 0.19, p>
.05. The lack of significant findings indicates that in
this sample client outcomes did not vary between
supervisors any differently depending on whether the
treating clinician was an intern or a resident.
Although supervisors were found not to differ in the
amount of change that was made by the clients who
were seen by the therapists that they supervised, it is
still possible that some supervisor characteristics could
–10
–5
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 1011121314151617181920212223
Average client OQ change
Supervisor
Figure 1. Average client change on the OQ-45.2 with 95% CI
error bars plotted by supervisors.
Psychotherapy Research 5
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

predict the change that clients made. In our final
model, we added supervisor experience, supervisor
field (social work vs. psychology), and supervisor level
(MS vs. Ph.D.) to the original baseline model with
both types of trainee therapists (practicum interns and
residents). A comparison of the −2LL values indicated
that this model was not significantly better than the
baseline model, χ
2
(3) = 2.43, p> .05, thus indicating
that taken together these supervisor variables did not
explain a significant amount of variance in client
outcomes above what was explained by the baseline
model. Additionally, given the full model, there was
no evidence that client outcomes differed depending
on the supervisor’s level of experience, t(553.19) =
0.77, p> .05, whether the supervisor’s field was social
work or psychology, t(701.13) = 0.48, p>.05,and
whether the supervisor had obtained a MS or a Ph.D.,
t(277.76) = 1.63, p> .05.
In addition to these main tests, we also conducted a
series of five analyses to test the sensitivity of the results
to the sample assumptions. First, we conducted all
analyses without inputting a last observation carried
Table II. Parameter estimates for the models examining client OQ change scores with therapists modeled at Level 2 and supervisors at
Level 3.
Baseline model with
both types of
therapists
Model where
supervisor variance
is allowed to vary by
therapist type
Full model with both
types of therapists
Fixed effects
Intercept 8.67* 8.70* 7.70*
Supervisor experience 0.22
Supervisor field (social work vs. psychology) –0.28
Supervisor level (MS vs. Ph.D.) 1.14
Random effects
Therapists 2.51* 2.63* 2.40*
Supervisors 0.12 0.00 0.00
Residual 296.63* 296.47* 296.67*
Deviance (−2LL) 55,670.90 55,671.09 55,668.47
*p<.01.
Table I. Percentages of client change reported separately for each supervisor.
Supervisor
No. of
cases
Avg. OQ-45
change
Percentage of
deterioration
Percentage of no
change
Percentage of reliable
improvement
Percentage of clinically
significant change
1 24 2.21 12.5 75.0 12.5 0.0
2 42 4.95 4.8 78.6 4.8 11.9
3 166 6.20 2.4 75.9 7.8 13.9
4 454 7.01 8.4 62.8 7.0 21.8
5 51 7.65 5.9 74.5 3.9 15.7
6 111 7.89 5.4 65.8 10.8 18.0
7 821 7.91 5.6 65.9 9.0 19.5
8 221 7.94 7.2 63.8 9.0 19.9
9 452 8.02 4.0 67.9 7.5 20.6
10 52 8.02 3.8 65.4 21.2 9.6
11 265 8.31 2.3 68.3 9.8 19.6
12 259 8.51 5.4 64.5 8.9 21.2
13 365 8.97 4.4 64.4 7.7 23.6
14 177 9.05 4.0 66.7 8.5 20.9
15 59 9.19 5.1 64.4 8.5 22.0
16 163 9.39 4.9 62.6 9.8 22.7
17 1583 9.64 4.4 63.7 7.4 24.5
18 453 9.96 4.2 64.0 9.1 22.7
19 369 9.98 4.1 61.2 6.5 28.2
20 362 10.02 5.0 62.2 10.5 22.4
21 10 12.60 10.0 70.0 0.0 20.0
22 46 13.24 6.5 56.5 15.2 21.7
23 16 14.00 0.0 56.3 12.5 31.3
6T. G. Rousmaniere et al.
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

forwardscoreforthoseclientswhoweremissingalast
sessionOQscore.Theonlydifferenceinresults
between this sample and the original sample reported
above was that in the baseline model, with both types
of trainees (residents and interns) the variance between
therapists was not significant. Second, as a more
inclusive approach, we conducted all analyses without
deleting any therapist or supervisors (even if they only
worked with one client/supervisee). No differences
were found between the results using this sample and
the original sample. Third, as a more conservative
approach, we conducted all analyses after removing
any therapist if they treated less than five clients (27
additional therapists were removed) and any supervisor
if they supervised less five therapists (five additional
supervisorswereremoved).Theonlydifferencein
results between this sample and the original sample
was that in the final model cases supervised by MS-
level supervisors on average displayed a significantly
greater change (1.47 points greater) than cases super-
vised by Ph.D.-level supervisors, t(235.32) = 2.02, p=
.05. However, it should be noted that although this
supervisor variable was significant, the overall model
with this and other supervisor variables entered did not
explain variance in client outcomes any better than the
empty model, χ
2
(3) = 4.04, p> .05. Fourth, we
conducted all analyses without removing any outliers.
Two differences with the original sample were
observed: (i) with both types of trainees (residents
and interns) the variance between therapists was not
significant, and (ii) cases supervised by MS-level
supervisors on average displayed a significantly greater
change (1.62 points greater) than cases supervised by
Ph.D.-level supervisors, t(266.5) = 2.19, p= .03. For
the last sensitivity analysis, we examined whether the
presence of couples counseling cases in the data
affected the results. Given the nature of couples
counseling, data from these clients violate the assump-
tion of independence of observations. Rather than
including another level of nesting for these clients, we
also conducted all analyses with the couples cases
removed. No differences between the results using this
sample and the original sample were found.
Discussion
The purpose of this study was to explore the amount
of variance in client psychotherapy outcome
accounted for by supervisors, in a naturalistic train-
ing setting. All available outcome data from thera-
pists and supervisees working at a community-based
counseling center over a five-year period were
included. The data thus present a comprehensive
picture of how supervision is routinely being prac-
ticed at a large counseling center.
The main finding of this study was that, for a five-
year period at this mental health center, super-
visors accounted for .04% of the variance in client
psychotherapy outcome. The lack of variability
between supervisors in client outcomes was demon-
strated across variables at the supervisor level—
supervisor experience level, field (social work vs.
psychology), and degree (MS vs. Ph.D.)—as well
as the trainee level (residents vs. interns). The reason
for the difference in findings is not immediately clear.
One possibility may be that this study included a
much larger sample size at all three levels (supervisor,
therapist, and client), and utilized a different statist-
ical test (HLM) than either of the previous studies.
Alternately, the difference in results may be due to
the different samples in each study: all of the super-
visors in this sample may have achieved a similar level
of competence in supervision, and thus had similarly
positive outcomes, in contrast to the no-supervision
condition in Bambling et al. (2006), and the super-
visors in Callahan et al. (2009), who may have had
different levels of supervision competency, and thus
better or worse outcomes. Another possible explana-
tion for the difference in results is that all supervisors
and trainees in this study were receiving session-
by-session outcome feedback, unlike the participants
in Bambling et al. (2006) and Callahan et al. (2009).
Given that the use of regular feedback in supervision
has been shown to significantly improve outcomes
(e.g., Lambert & Shimokawa, 2011; Reese et al.,
2009), it is possible that the ubiquitous use of
outcome feedback itself may have been powerful
enough to obscure the outcome variance due to
other supervision variables, and thus made the effects
of all supervisors similar.
One possible reason for the lack of difference in
client outcomes between supervisors may be con-
founding variables within the chain of supervision
which we were unable to control for in this study.
Supervision theorists have proposed that many vari-
ables at the supervisor, therapist, and client levels may
moderate supervisor effects on client outcome, and
thus reduce the amount of variance in outcome that
may be attributed to supervisors (e.g., Wampold &
Holloway, 1997). To affect client outcome, super-
visors’interventions have to, in effect, travel through
three layers of mediating variables: client variables,
therapist variables, and supervisor variables. Recent
research suggests that the mediating effects of vari-
ables at all three levels may be quite strong. At the
client level, multiple variables have been shown
to greatly moderate the effects of psychotherapy
(e.g., the therapeutic working alliance, Norcross &
Lambert, 2011; client expectations for change, Swift,
Greenberg, Whipple, & Kominiak, 2012). Indeed, it
has been suggested that client variables themselves
Psychotherapy Research 7
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

may account for most of the variance in psychother-
apy outcome (e.g., symptom severity and treatment
history; Bohart & Tallman, 2010). If this is accurate,
then these variables may serve to minimize variance in
supervisor effects on outcome.
Likewise, research has identified multiple variables
at the supervisee level that may moderate super-
visors’effects on client outcome. The most promin-
ent variable is the supervisory working alliance,
which has been demonstrated to vary widely across
supervisory dyads (e.g., Inman et al., 2014). Other
potentially mediating supervisee variables include the
extent of supervisee nondisclosure, which has been
shown to frequently be quite high (Mehr et al., 2010),
and supervisees’perceptions of the extent of collab-
oration in supervision, which may often be low
(Rousmaniere & Ellis, 2013). Additionally, variables
at the supervisor level may moderate supervisors’
effects on outcome, such as the method or model
for supervision (Bernard & Goodyear, 2014). For
example, live one-way-mirror supervision may
address the possibility that supervisees may forget
lessons from supervision (e.g., Rousmaniere & Fre-
derickson, 2013).
A significant limitation of this study was the lack of
a control group of trainees not receiving supervision
(due to legal and ethical concerns), which limited the
study to exploring the variation of treatment out-
come associated with supervisors, rather than if
supervisors had an effect on treatment outcome at
all (unlike Bambling et al., 2006). Thus, the findings
do not imply supervisors in this study did not
enhance or protect client welfare. Rather, the finding
suggests that client psychotherapy outcome was
extremely similar when aggregated across super-
visors. Given that the mean change for all clients
was 8.81 points on the OQ-45.2, the findings may
suggest that, on average, competent client treatment
was provided, under the care of these supervisors.
Another limitation is the small sample of super-
visors, which could have contributed to the mixed
findings. Additionally, all data came from one mental
health center, and it is unclear if the findings of this
study generalize to other locations. For example, it is
possible that this counseling center may just have very
similar supervisors (e.g., most are females), thus
explaining the findings of minimal difference between
supervisors. However, this is unlikely, given that the
23 supervisors who participated practice a range of
clinical modalities. Thus, this study should be repli-
cated at other locations. Another limitation is that the
two hours per week of group supervision that practi-
cum and resident supervisees received (from the same
supervisor) may have influenced the results. For
example, good advice from peers may have moder-
ated the impact of bad supervision, and vice-versa.
Another limitation is that all supervisors, therapists,
and patients were provided with OQ-45.2 feedback,
so it was impossible to assess whether feedback in
supervision had a moderating effect. Other limitations
were that data were only obtained via client self-report
from one outcome measure (the OQ-45.2), and that
limited demographic information was obtained on
supervisors and supervisees (due to confidentiality
concerns and availability of records).
Implications for Theory and Research
Supervision literature has widely conceptualized
supervisors as one of, if not the, primary gatekeepers
tasked with ensuring and protecting client welfare
(Bernard & Goodyear, 2014). However, there has
been theoretical debate about the extent to which
supervisors can affect client outcome (e.g., Ladany &
Inman, 2012; Wampold & Holloway, 1997), and
this debate has been informed by little empirical data
(e.g., Watkins, 2011). The findings from this study
allow us to explore one perspective on this question:
the degree of variance in supervisor effects on client
outcome, from the actual practice of 23 supervisors
and 175 supervisees, with 6521 psychotherapy cases.
One possible positive interpretation of the minimal
variance is that the overall average welfare of 6521
clients was ensured and protected because the mean
client outcome improved by 8.81 points. This could
be taken as an indication that the supervisors in the
study were overall successful at protecting client
welfare. Future research on this topic would benefit
from controlling for the many variables that may
moderate supervisors’effects on client outcome at
the supervisor, supervisee, and client levels (e.g., the
supervisory working alliance, and the influence of
peers in group supervision). It is our hope that future
research will further explore these important
questions.
Acknowledgment
The authors would like to thank Michael Ellis for
help with this manuscript.
References
Bambling, M., King, R., Raue, P., Schweitzer, R., & Lambert, W.
(2006). Clinical supervision: Its influence on client-rated
working alliance and client symptom reduction in the brief
treatment of major depression. Psychotherapy Research,16, 317–
331. doi:10.1080/10503300500268524
Beck, A. T., Steer, R. A., & Brown, G. K. (1996). The BDI-II
manual. London: Harcourt Brace.
Beck, A. T., Steer, R. A., & Carbin, M. G. (1987). Psychometric
properties of the beck depression inventory: Twenty-five years of
8T. G. Rousmaniere et al.
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

evaluation. Clinical Psychology Review, 8(1), 77–100. doi:10.1016/
0272-7358(88)90050-5
Bernard, J. M., & Goodyear, R. K. (2014). Fundamentals of clinical
supervision (5th ed.). Needham Heights, MA: Allyn & Bacon.
Beutler, L. E., Machado, P. P., & Neufeldt, S. (1994). Therapist
variables. In A. E. Bergin & S. Garfield (Eds.), Handbook of
psychotherapy and behavior change (4th ed. pp. 229–269).
Oxford: John Wiley & Sons.
Bohart, A. C., & Tallman, K. (2010). Clients: The neglected
commonfactorinpsychotherapy.InB.L.Duncan,S.D.Miller,
B. E. Wampold, & M. A. Hubble (Eds.), The heart and soul of
change: Delivering what works in therapy (2nd ed. pp. 83–111).
Washington, DC: American Psychological Association.
Callahan, J. L., Almstrom, C. M., Swift, J. K., Borja, S. E., &
Heath, C. J. (2009). Exploring the contribution of supervisors
to intervention outcomes. Training and Education in Professional
Psychology, 3(2), 72–77. doi:10.1037/a0014294
Derogatis, L. R. (1992). SCL-90-R: Administration, scoring &
procedures manual-II (2nd ed.). Towson, MD: Clinical Psycho-
metric Research, Inc.
Ellis, M. V. (2001). Harmful supervision, a cause for alarm:
Comment on. Journal of Counseling Psychology,48, 401–406.
doi:10.1037/0022-0167.48.4.401
Ellis, M. V., Berger, L., Hanus, A. E., Ayala, E. E., Swords, B. A.,
& Siembor, M. (2013). Inadequate and harmful clinical
supervision: Testing a revised framework and assessing occur-
rence. The Counseling Psychologist, 42(4), 434–472.
doi:10.1177/0011000013508656
Ellis, M. V., & Ladany, N. (1997). Inferences concerning super-
visees and clients in clinical supervision: An integrative review.
In C. E. Watkins (Ed.), Handbook of psychotherapy supervision
(pp. 447–507). New York, NY: Wiley.
Falender, C. A., & Shafranske, E. P. (2004). Clinical supervision: A
competency-based approach. Washington, DC: American Psy-
chological Association.
Freitas, G. J. (2002). The impact of psychotherapy supervision on
client outcome: A critical examination of 2 decades of research.
Psychotherapy: Theory, Research, Practice,Training,39, 354–367.
doi:10.1037/0033-3204.39.4.354
Gibson, J. A., Grey, I. M., & Hastings, R. P. (2009). Supervisor
support as a predictor of burnout and therapeutic self-efficacy
in therapists working in ABA schools. Journal of Autism and
Developmental Disorders, 39, 1024–1030. doi:10.1007/s10803-
009-0709-4
Gray, L. A., Ladany, N., Walker, J. A., & Ancis, J. R. (2001).
Psychotherapy trainees’experience of counterproductive events
in supervision. Journal of Counseling Psychology, 48, 371–383–
383. doi:10.1037/0022-0167.48.4.371
Holloway, E., & Carroll, M. (1996). Reaction to the special
section on supervision research: Comment on Ellis et al.
(1996), Ladany et al. (1996), Neufeldt et al. (1996), and
Worthen and McNeill (1996). Journal of Counseling Psychology,
43(1), 51–55. doi:10.1037/0022-0167.43.1.51-55
Holloway, E. L., & Neufeldt, S. A. (1995). Supervision: Its
contributions to treatment efficacy. Journal of Consulting and
Clinical Psychology, 63, 207–213. doi:10.1037/0022-006X.63.
2.207
Horvath, A. O., & Greenberg, L. S. (1989). Development and
validation of the working alliance inventory. Journal of Counsel-
ing Psychology, 36, 223–233. doi:10.1037/0022-0167.36.2.223
Inman, A. G., Hutman, H., Pendse, A., Devdas, L., Luu, L., &
Ellis, M. V. (2014). Current trends concerning supervisors,
supervisees, and clients in clinical supervision. In C. E.
Watkins & D. Milne (Eds.), The international handbook of
clinical supervision (pp. 61–102). Oxford: Wiley Press.
Ladany, N., & Friedlander, M. L. (1995). The relationship
between the supervisory working alliance and trainees' experi-
ence of role conflict and role ambiguity. Counselor Education
and Supervision, 34, 220–231. doi:10.1002/j.1556-6978.1995.
tb00244.x
Ladany, N., & Inman, A. G. (2012). Training and supervision. In
E. M. Altmaier & J. C. Hansen (Eds.), The Oxford handbook of
counseling psychology (pp. 179–207). New York, NY: Oxford
University Press.
Ladany, N., Mori, Y., & Mehr, K. E. (2012). Effective and
ineffective supervision. The Counseling Psychologist,41(1), 28–
47. doi:10.1177/0011000012442648
Lambert, M. J., & Arnold, R. C. (1987). Research and the
supervisory process. Professional Psychology: Research and Prac-
tice, 18, 217–224. doi:10.1037/0735-7028.18.3.217
Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K.,
Okiishi, J., Burlingame, G. M., & Reisinger, C. W. (1996).
Administration and scoring manual for the OQ-45.2. Stevenson,
MD: American Professional Credentialing Services.
Lambert,M.J.,Morton,J.J.,Hatfield,D.,Harmon,C.,Hamilton,
S., Reid, R. C., …Burlingame, G. M. (2004). Administration and
scoring manual for the outcome questionnaire, 45.2. Salt Lake City,
UT: OQ Measures.
Lambert, M. J., & Shimokawa, K. (2011). Collecting client
feedback. Psychotherapy, 48(1), 72–79. doi:10.1037/a0022238
Mehr, K. E., Ladany, N., & Caskie, G. I. L. (2010). Trainee
nondisclosure in supervision: What are they not telling you?
Counselling and Psychotherapy Research,10(2), 103–113.
doi:10.1080/14733141003712301
Miller, S., Hubble, M., & Duncan, B. (2008). Supershrinks.
Therapy Today,19,4–9.
Milne, D., Sheikh, A., Pattison, S., & Wilkinson, A. (2011).
Evidence-based training for clinical supervisors: A systematic
review of 11 controlled studies. The Clinical Supervisor,30(1),
53–71. doi:10.1080/07325223.2011.564955
Norcross, J. C., & Lambert, M. J. (2011). Evidence-based therapy
relationships. In J. C. Norcross (Ed.), Psychotherapy relation-
ships that work (2nd ed., pp. 3–21). New York, NY: Oxford
University Press.
Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003).
Waiting for supershrink: An empirical analysis of therapist
effects. Clinical Psychology & Psychotherapy,10, 361–373.
doi:10.1002/cpp.383
Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A.,
Halstead, J. L., Rowlands, S. R., & Chisholm, R. R. (2009).
Using client feedback in psychotherapy training: An analysis of
its influence on supervision and counselor self-efficacy. Train-
ing and Education in Professional Psychology, 3, 157–168.
doi:10.1037/a0015673
Rousmaniere, T. G., & Ellis, M. V. (2013). Developing the
construct and measure of collaborative clinical supervision:
The supervisee’s perspective. Training and Education in Profes-
sional Psychology, 7, 300–308. doi:10.137/a0033796
Rousmaniere, T., & Frederickson, J. (2013). Internet-based
one-way-mirror supervision for advanced psychotherapy
training. The Clinical Supervisor,32(1), 40–55. doi:10.1080/
07325223.20130.778683
Swift, J. K., Greenberg, R. P., Whipple, J. L., & Kominiak, N.
(2012). Practice recommendations for reducing premature
termination in therapy. Professional Psychology: Research and
Practice,43, 379–387. doi:10.1037/a0028291
Vermeersch,D.A.,Lambert,M.J.,&Burlingame,G.M.
(2000). Outcome questionnaire: Item sensitivity to change.
Journal of Personality Assessment, 74, 242–261. doi:10.1207/
S15327752JPA7402_6
Psychotherapy Research 9
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014

Wampold, B. E. (2010). The research evidence for common
factors models: A historically situated perspective. In B. L.
Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble (Eds.),
The heart and soul of change: Delivering what works in therapy
(2nd ed. pp. 49–81). Washington, DC: American Psychological
Association.
Wampold, B. E., & Holloway, E. L. (1997). Methodology, design,
and evaluation in psychotherapy supervision research. In C. E.
Watkins (Ed.), Handbook of psychotherapy supervision (pp. 11–27).
New York, NY: Wiley.
Watkins, C. E. (2011). Does psychotherapy supervision contribute
to patient outcomes? Considering thirty years of research. The
Clinical Supervisor,30, 235–256. doi:10.1080/07325223.2011.
619417
10 T. G. Rousmaniere et al.
Downloaded by [University of Alaska Fairbanks], [Jason Whipple] at 10:18 06 October 2014