ArticlePDF Available

Abstract and Figures

Clinical supervision is deemed an essential element in the development of therapist competence and provision of psychotherapy to clients. However, the association between supervision and psychotherapy process and outcome has been mixed, unclear, and presumed to vary widely given the idiosyncratic features of the supervision and therapy process. Thus, to provide an up-to-date (articles published until May, 2019) quantitative summary, we conducted a meta-analytic review to examine the associations between supervision variables and psychotherapy process and outcome variables including: therapeutic relationship, client satisfaction, and treatment outcomes. Using a random effects model, the pooled Pearson’s correlation between supervision and psychotherapy process and outcome variables was .21 across 12 studies (32 effects) that were included. Thus, supervision accounted for 4% of the variance in client outcomes. Approximately 54% of the total variance between studies was due to heterogeneity and not to chance. An additional meta-analysis without the 4 studies that assessed client outcomes using supervisor/therapist ratings yielded a slightly higher correlation (r .24), accounting for 6% of the variance in client outcomes. Effect sizes regarding the therapeutic relationship and client satisfaction varied widely while effect sizes for treatment outcomes were less varied with consistently small positive effects. Supervisory working alliance was most frequently examined in assessing supervision and accounted for wider variance in effect sizes. There seemed to be less variance among specific supervision factors (e.g., style, satisfaction, structure) with consistent small to medium positive effects. Implications for future research are discussed.
Content may be subject to copyright.
Translational Issues in Psychological
Science
Supervision and Psychotherapy Process and Outcome: A
Meta-analytic Review
Brian TaeHyuk Keum and Lei Wang
Online First Publication, September 10, 2020. http://dx.doi.org/10.1037/tps0000272
CITATION
Keum, B. T., & Wang, L. (2020, September 10). Supervision and Psychotherapy Process and
Outcome: A Meta-analytic Review. Translational Issues in Psychological Science. Advance
online publication. http://dx.doi.org/10.1037/tps0000272
Supervision and Psychotherapy Process and Outcome:
A Meta-analytic Review
Brian TaeHyuk Keum
University of California, Los Angeles
Lei Wang
Chatham University
Clinical supervision is deemed an essential element in the development of therapist
competence and provision of psychotherapy to clients. However, the association between
supervision and psychotherapy process and outcome has been mixed, unclear, and pre-
sumed to vary widely given the idiosyncratic features of the supervision and therapy
process. Thus, to provide an up-to-date (articles published until May, 2019) quantitative
summary, we conducted a meta-analytic review to examine the associations between
supervision variables and psychotherapy process and outcome variables including: thera-
peutic relationship, client satisfaction, and treatment outcomes. Using a random effects
model, the pooled Pearson’s correlation between supervision and psychotherapy process
and outcome variables was .21 across 12 studies (32 effects) that were included. Thus,
supervision accounted for 4% of the variance in client outcomes. Approximately 54% of the
total variance between studies was due to heterogeneity and not to chance. An additional
meta-analysis without the 4 studies that assessed client outcomes using supervisor/therapist
ratings yielded a slightly higher correlation (r.24), accounting for 6% of the variance in
client outcomes. Effect sizes regarding the therapeutic relationship and client satisfaction
varied widely while effect sizes for treatment outcomes were less varied with consistently
small positive effects. Supervisory working alliance was most frequently examined in
assessing supervision and accounted for wider variance in effect sizes. There seemed to be
less variance among specific supervision factors (e.g., style, satisfaction, structure) with
consistent small to medium positive effects. Implications for future research are discussed.
What is the significance of this article for the general public?
This meta-analysis suggests that clinical supervision is associated with improvements in
psychotherapy process and outcome. However, the effects are generally small and vary
across treatment outcomes, client satisfaction, and therapeutic relationship.
Keywords: clinical supervision, treatment outcomes, client satisfaction, therapeutic
relationship, meta-analysis
Clinical supervision is deemed an essential
element in the provision of therapy (Bernard &
Goodyear, 2014). Under the guidance of a su-
pervisor, supervision allows therapists to en-
gage in continual development of their clinical
competence. Supervision also acts as a check
for safeguarding and protecting the welfare of
mental health clients (Falender & Shafranske,
XBrian TaeHyuk Keum, Department of Social Wel-
fare, Luskin School of Public Affairs, University of Cali-
fornia, Los Angeles; Lei Wang, Department of Graduate
Psychology, School of Health Sciences, Chatham Univer-
sity.
Correspondence concerning this article should be ad-
dressed to Brian TaeHyuk Keum, Department of Social
Welfare, Luskin School of Public Affairs, University of
California, Los Angeles, 5244 Public Affairs Building, 337
Charles E. Young Drive East, Los Angeles, CA 90095.
E-mail: briankeum@luskin.ucla.edu
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Translational Issues in Psychological Science
© 2020 American Psychological Association 2020, Vol. 2, No. 999, 000
ISSN: 2332-2136 http://dx.doi.org/10.1037/tps0000272
1
2004). Effective supervision has been linked to
multitude of therapist benefits (Wheeler &
Richards, 2007), such as increased supervisee
counselor self-efficacy (Gibson, Grey, & Hast-
ings, 2009), and skill acquisition (Lambert &
Arnold, 1987). Thus, many accept the idea that
supervision is a critical component in improv-
ing the overall quality of counseling services
that therapists provide, which in turn enhances
the level of psychological care clients receive.
However, despite the monolithic beliefs in
the importance of supervision in therapy, little
empirical evidence exists on the efficaciousness
of supervision on client outcomes and the topic
has been a subject of debate among psychother-
apy researchers (Holloway & Neufeldt, 1995;
Ladany & Inman, 2012; Watkins, 2011; Wat-
kins & Callahan, 2019). Findings from studies
on supervision effects on client outcomes re-
mains less clear and mixed. Freitas (2002) con-
ducted a review of studies from 1987 to 1997
and suggested that no conclusive summary
could be drawn due to methodological and con-
ceptual limitations. Milne, Sheikh, Pattison, and
Wilkinson (2011) systematically reviewed 11
studies and suggested that a combination of
training and supervisory methods that enhance
therapist development could be effective in im-
proving client outcomes but noted that only two
of the 11 studies directly examined the super-
vision– client outcome association. More re-
cently, Watkins (2011) conducted a review of
studies from 2006 to 2011 and concluded that
the supervision effects on client outcomes still
remained inconclusive compared to 30 years
ago. Based on the studies to date, the three
review papers suggested discrepancies in the
effects and questioned whether supervision in-
deed had a significant positive effect on client
outcomes.
However, a major caveat among these re-
views was that their conclusions could not be
substantiated due to the lack of studies and
appropriate methodologies to examine the su-
pervision– client outcome associations. Further-
more, the reviews did not consider the quanti-
tative implications of studies reviewed and did
not include recent quantitative studies. In the
past decade, an increasing number of quantita-
tive studies have been conducted regarding su-
pervision on client outcomes, including more
relevant approaches such as multilevel analysis.
For example, supervision variables such as su-
pervisory working alliance, supervision quality,
and supervisor characteristics have been studied
as direct predictors of client outcomes (e.g.,
Lainas, 2014; McCune, 2016; Reese et al.,
2009). With greater number of supervision–
client outcome quantitative studies, an impor-
tant question is whether supervision processes
correspond to variability in client outcomes and
the therapeutic process. Thus, the purpose of the
present study was to conduct a meta-analytic
review of supervision effects on client outcomes
and therapeutic process based on a comprehen-
sive literature search to provide an up-to-date,
quantitative summary on the significance of su-
pervision.
Defining Clinical Supervision
For the current study, we adopted the defini-
tion used by the American Psychological Asso-
ciation (2015) in the Guidelines for Clinical
Supervision in Health Services, which may also
be applicable to other mental health profes-
sions:
Supervision is a distinct professional practice employ-
ing a collaborative relationship that has both facilita-
tive and evaluative components, that extends over
time, which has the goals of enhancing the professional
competence and science-informed practice of the su-
pervisee, monitoring the quality of services provided,
protecting the public, and providing a gatekeeping
function for entry into the profession. (p. 5)
There are many models of supervision that
strive to embody this definition. Notably, the
three major categories of supervision organized
by Bernard and Goodyear (2014) are models
grounded in psychotherapy theory, develop-
mental models, and process models. With mod-
els grounded in psychotherapy theory, since su-
pervisors were therapists first, supervisors tend
to adopt the worldview from their therapy to the
clinical supervision they provide. In terms of
developmental models, there is a focus on as-
sessing and meeting supervisees’ needs, expe-
riences, and contexts. Lastly, the process mod-
els address the supervision process and the roles
that supervisors undertake.
Rather than operating from a single model,
Bernard and Goodyear (2014) posits that super-
visors tend to adopt an integration of above
models such as combined models, target issue
models, common-factors models, and evidence-
based supervision. Combined models integrate
2 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
and blend supervision models, whereas target
issue models serve to address specific critical
issues (e.g., multicultural competence, supervi-
sory relationship). Among them, scholars have
called for a model of evidence-based supervi-
sion that reflects the field’s shift toward empir-
ically supported psychotherapy. However,
scholars (e.g., Wampold, Goodheart, & Levant,
2007) have argued that supervision is nuanced
and that common factors (e.g., supervision re-
lationship, instillation of hope; Lampropoulos,
2003) should be considered when working with
supervisees. Bernard and Goodyear (2014) con-
cluded that instead of conceptualizing evidence-
based supervision as a new model, it should be
considered as an overarching movement in that
any supervision model can be labeled as evi-
dence-based as long as there are enough re-
search and empirical support. Clearly, clinical
supervision can be conceptualized and em-
ployed in many different ways depending on the
therapeutic context and the needs, goals, and
preferences in the supervision process.
In terms of assessment of the supervision
process, the majority of existing literature
have focused on the supervisory working al-
liance (e.g., Bell, Hagedorn, & Robinson,
2016; Gnilka, Chang, & Dew, 2012; McCar-
thy, 2013), typically measured by the Super-
visory Working Alliance Inventory (SWAI;
Efstation, Patton, & Kardash, 1990) that as-
sesses rapport and client focus in supervision.
Some of the other factors that have been
operationalized include interpersonal affinity
to supervisor (e.g., Dodenhoff, 1981), super-
vision style-or the distinctive manner of ap-
proaching and responding to trainees (e.g.,
Van Dam, 2014), supervision satisfaction
(i.e., the supervisee’s understanding of their
interactions in supervision, their interpreta-
tion of their supervisor’s perceived perfor-
mance and personal characteristics, and the
level of ease in expressing their thoughts dur-
ing supervision; measured by Trainee Per-
sonal Reaction Scale; Holloway & Wampold,
1984), and multicultural supervision, or the
focus on cultural differences and multicul-
tural competence (e.g., Perez, 2018). Existing
measures have been developed in the 1980s
and 1990s and based on the assumption that
the nature and standards of the supervisory
relationship are similar across different set-
tings (e.g., university counseling centers vs.
community agencies). Scholars have also crit-
icized the limited availability of conceptually
and psychometrically valid measures (Wat-
kins & Callahan, 2019).
Relationship of Supervision With the
Therapeutic Process and Client Outcomes
Therapeutic Relationship
A handful of studies have tested the associa-
tion between supervision and working alliance
(WA) between therapists and clients, frequently
measured using the Working Alliance Inventory
(WAI; Horvath & Greenberg, 1989). Over the
years, WA has gained strong empirical support
in fostering positive client outcomes (e.g., Dun-
can & Moynihan, 1994; Horvath, Del Re,
Flückiger, & Symonds, 2011) via establishment
of affective bond, and agreement on task and
goals between the therapist and the client. WA
in therapy has been found to benefit from su-
pervision in number of studies. Bambling, King,
Raue, Schweitzer, and Lambert (2006) found
that clients receiving therapy from therapists in
the supervision group had significantly higher
scores on the WAI than clients in the no-
supervision group. Kivlighan, Angelone, and
Swafford (1991) found that clients working
with therapists who received live supervision
reported stronger WA than those working with
therapists receiving videotaped supervision.
Patton and Kivlighan (1997) found that the un-
biased correlation between therapist-rated su-
pervisory WA and the client-rated WA was
moderate at .66. On the other hand, Gnilka and
colleagues (2012) found that the correlation be-
tween therapist-rated supervisory and therapeu-
tic WA was small at .22. The small correlation
reported in Gnilka and colleagues (2012) study
compared to Patton and Kivlighan’s (1997)
study may be due to the fact that therapists rated
the WA in Gnilka and colleagues (2012) study
rather than clients. Literature suggests that ther-
apists tend to underestimate their therapy prog-
ress with clients (Atzil-Slonim et al., 2015).
However, it is also possible that the discrepancy
reflects the variances in supervision that can
differentially affect the therapeutic WA.
Beyond WA, very few studies have been
conducted regarding supervision effects on
other relational aspects of the therapeutic rela-
tionship. Two early studies (Crutchfield & Bor-
3META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ders, 1997; Karr & Geist, 1977) found that there
was no significant association between supervi-
sion and trainees’ level of empathy in therapy.
Interestingly, Bell and colleagues (2016) found
negative small to medium effects between the
interpersonal relationships in supervision and
therapy, suggesting that strong relationship in
supervision was associated with weaker thera-
peutic relationship. Overall, most of the super-
vision studies on therapeutic relationship were
focused on WA, while there was a lack of recent
quantitative studies examining other aspects of
the therapeutic relationship (e.g., real relation-
ship, rupture) in therapy using more relevant
approaches such as multilevel analysis.
Treatment Outcomes
Studies have also tested the association be-
tween supervision and various treatment out-
comes, ranging from comprehensive symptom
measures such as Outcome Questionnaire-45
(OQ-45) and Symptom Checklist-90 (SCL-90),
domain-specific changes (e.g., alcohol use;
Mastroleo, Magill, Barnett, & Borsari, 2014),
population-based symptom changes (e.g.,
CBCL; Schoenwald, Sheidow, & Chapman,
2009), to brief proprietary items such as one-
item assessment of change (Bartle-Haring, Sil-
verthorn, Meyer, & Toviessi, 2009; Lainas,
2014). Treatment outcomes have commonly
been assessed via client ratings, although ther-
apists and/or supervisors have rated the treat-
ment outcomes in some studies (e.g., Gnilka et
al., 2012; McCarthy, 2013; Perez, 2018). A
handful of studies have tested the association
between supervision and OQ-45 (e.g., Bell et
al., 2016; McCune, 2016; Rieck, Callahan, &
Watkins, 2015; Wrape, Callahan, Ruggero, &
Watkins, 2015) and found generally small ef-
fects. Small to moderate effects have been
found in studies that used the SCL-90 (Calla-
han, Almstrom, Swift, Borja, & Heath, 2009),
Vanderbilt Functioning Inventory (Schoenwald
et al., 2009), and the Beck Depression Inventory
(Callahan et al., 2009). Compared to indicators
of therapeutic relationship, supervision ap-
peared to have greater positive associations with
treatment outcomes.
Client Satisfaction
Client satisfaction is also anticipated to in-
crease as a result of effective supervision, which
may help therapists to better attend to clients’
expectations in therapy. Mcaleavey, Caston-
guay, and Xiao (2014) found that clients’ ses-
sion quality ratings were better when their ther-
apists’ orientations were in agreement with their
respective supervisors. Several studies have ex-
amined whether supervision translates to clients’
increased sense of fulfillment and whether their
expectations of sessions have been met. Early
studies have tested whether supervision improves
clients’ contentment with therapy. Harkness and
Hensley (1991) found that clients reported greater
satisfaction with therapists receiving client-
centered supervision compared to control condi-
tion (mixed focus on administration, training, and
consultation). Kivlighan et al. (1991) found that
compared to therapists receiving videotaped su-
pervision, clients working with therapists receiv-
ing live supervision reported stronger WA. Zarski,
Bubenzer, and Walter (1980) assessed the rela-
tionship between supervisor’s rating of the thera-
pist’s performance and client’s rating of the satis-
faction using the Counseling Evaluation Inventory
(CEI; Linden, Stone, & Shertzer, 1965) and found
a moderate association. Using the same scale,
Couchon and Bernard (1984) found that timing of
supervision was positively associated with clients’
scores on the CEI.
More recently, studies have reported smaller
effects between supervision and clients’ evalua-
tion of their session expectations. Reese and col-
leagues (2009) found a small effect between su-
pervisory WA and clients’ session rating scale
scores among 28 graduate trainees who were pro-
vided client feedback supervision (i.e., data gained
from clients was provided to their supervisor for
use within supervision). Van Dam (2014) exam-
ined the association between several supervision
factors and client goal attainment with data from
37 social work master’s-level field supervisory
triads. Interestingly, SWAI had a negative corre-
lation with client goal attainment. Supervisory
style and tools were positive but with small ef-
fects. Collectively, the findings seem more mixed,
with generally small effects on the role that super-
vision may play on improving clients’ satisfaction
in therapy.
The Present Study
In our review of the literature, several areas
emerged that may potentially contribute to the
variability in the association between supervi-
4 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
sion and psychotherapy and treatment out-
comes. First, there seems to be wide variability
in the relationships depending on the type of
outcomes. In particular, associations with ther-
apeutic relationship and client satisfaction vari-
ables varied from small to large effects. On the
other hand, relationships with treatment out-
comes varied less with consistently small ef-
fects. In addition to the magnitude, several stud-
ies have found negative relationships with
treatment outcomes and therapeutic relationship
(e.g., Bell et al., 2016) suggesting that the as-
sociation may not be uniformly positive. Sec-
ond, the relationships seem to vary depending
on who rated the client outcomes. As noted
previously, the differences in the correlations
with therapeutic WA (.22 vs. .66) depending on
therapist or client ratings suggests that rater bias
(e.g., underestimation, overestimation) may
contribute to the variability in the outcomes.
Furthermore, such bias may contribute to the
variability in the predictability of the supervi-
sion variables. Third, variability may stem due
to the different aspects of supervision that have
been assessed in the studies. Most studies in our
review examined the supervisory WA (e.g., Re-
ese et al., 2009; Rieck et al., 2015) but other
studies have assessed skill-based or didactic
approaches (e.g., Schoenwald et al., 2009), su-
pervisory style (e.g., Van Dam, 2014), and do-
main-specific supervision (e.g., multicultural
supervision; Perez, 2018). Fourth, measurement
error may also contribute to the variability.
Much of the early studies have employed mea-
sures with inadequate psychometric properties
(e.g., poor internal consistency, lack of validity
evidence, content validity issues) that may be
prone to greater measurement error compared to
recent studies. Additionally, while studies have
used treatment outcome and relationship mea-
sures with established psychometric properties
(e.g., OQ-45; Bell et al., 2016), some studies
have developed their own proprietary items
(e.g., Bartle-Haring et al., 2009). The lack of
adequate measurement in some of the studies
may introduce measurement error that may
complicate the variability in the effects.
Against this backdrop, to examine the quan-
titative relationship between supervision and
client outcome/therapy process, we conducted a
meta-analysis of studies that have examined
some aspect of the supervision process and its
relationship to the psychotherapy process and
client outcomes. Meta-analysis allows a test of
variability in effects between the studies and
whether certain associations explain greater ef-
fect in client outcomes and the therapeutic pro-
cess. In doing so, our aim was to focus on
studies that strived to model both the supervi-
sor-therapist and therapist-client perspectives
(not just one party rating both supervision and
outcome variables) and utilized psychometri-
cally sound measures. Furthermore, we focused
on studies that assessed the process of supervision
rather than supervisor effects (i.e., effects due to
having different supervisors) as the latter provides
no information on specific factors (e.g., supervi-
sory WA) that may explain the supervision vari-
ance in client outcomes. Furthermore, Rous-
maniere, Swift, Babins-Wagner, Whipple, and
Berzins (2016) examined supervisor variance in
client outcomes (OQ-45) using longitudinal three-
level multilevel analysis (6,521 clients, seen by
175 trainee therapists, supervised by 23 supervi-
sors) and did not find any significant supervisor
effects and supervisors accounted for only .04% of
the variance in client outcomes. The authors sug-
gested that the nonsignificant and underwhelming
finding may be due to confounding variables (e.g.,
supervisory WA), part of which the current study
sought to examine.
In reviewing the current body of quantitative
evidence on supervision, our overarching ques-
tion was to assess whether supervision is asso-
ciated with client outcomes, and perhaps asso-
ciated with improvements in outcomes. We
hypothesized that greater supervision quality
(e.g., stronger WA, satisfaction, etc.) would be
associated with a more positive therapeutic pro-
cess between the therapist and the client, and
better client outcomes. However, we anticipated
that the effect size of the relationship may be
small and moderate at best given that most
studies have reported small and even negative
associations between supervision and client out-
comes (e.g., Bell et al., 2016; McCarthy, 2013).
Method
Literature Search
We conducted an electronic search of published
and unpublished studies using clinical supervis
,
psychotherapy supervis
, counseling supervis
,
nursing supervi
, psychology supervis
, social
work supervi
, counselor train
, trainee therapist,
5META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
and practicum train
search terms in the following
databases: Academic Search Complete, CINAHL,
ERIC, Family and Society Studies, ISI, Medline,
Nursing/Academic Edition, ProQuest, PubMed,
PsycARTICLES, PsycINFO, and Social Services
Abstracts. We also searched for articles by exam-
ining reference lists of review studies, and manual
searches based on relevant authors doing research
on supervision. The search process included all
papers published up until May, 2019 (inclusive).
Initially, as shown in Figure 1, we identified
26,564 articles including duplicates, from which
108 articles were deduced by screening the titles.
We independently screened these articles by read-
ing abstracts and method sections to decide
whether to include them for further analysis. Qual-
itative, analogue, case studies, or review articles
were excluded. This resulted in 33 articles for
detailed evaluation. Final inclusion criteria for
analysis were: (a) the study included a quantitative
measure of supervision process (either rated by
trainee or supervisor) or a highly related construct
and client outcome (either rated by trainee, super-
visor, or client) and (b) the supervision process (or
related construct) was correlated with client out-
come. The process resulted in excluding 21 stud-
ies that did not meet the final criteria. Of the 12
studies included, 10 were published studies in
peer-reviewed journals and two were unpublished
dissertations.
Coding
We coded for the following information: (a)
sample size and client demographics, including
age, gender, percentage of racial/ethnic minor-
ities; (b) sample size and therapist/supervisee
demographics, including age, gender, percent-
age of racial/ethnic minorities, level of educa-
tion; (c) sample size and supervisor demograph-
ics, including age, gender, percentage of racial/
ethnic minorities, level of education; (d)
average numbers of psychotherapy sessions cli-
ents received and supervision sessions supervis-
ees received; (e) modality of supervision, theo-
retical orientation of supervisor and/or
Figure 1. Study inclusion flowchart.
6 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
therapist/supervisee, supervisors’ years of prac-
tice, setting of clinical context; (f) the types and
names of measures used to assess supervision
process and client outcomes as well as who
rated each measure. Information from all of the
included studies were retrieved directly from
the published articles and dissertations.
Supervision process measures included in-
ventories or self-report tools used to assess the
supervisory relationship, supervision WA, ther-
apist/supervisee’s affinity toward supervisor,
supervision outcome satisfaction, and supervi-
sion style. These measures were rated by either
the supervisor or the therapist/supervisee. Client
outcome measures were used to assess symp-
toms, functioning, satisfaction of treatment,
WA in therapy, and client progress. These mea-
sures were rated by either the supervisor, ther-
apist/supervisee, or client.
Statistical Analyses
Our effect size of interest was the Pearson
product–moment correlation (r). We calculated
the effect sizes for the studies that did not report
rs between our variables of interest (i.e., super-
vision process and client outcome). For inter-
pretation purposes, rs from treatment outcome
studies were reversed so that positive correla-
tions represent positive scores on supervision
process associating with better client outcomes.
To correct for non-normality of the correlation
effect size and stabilize variance estimation, rs
were converted to z scores for the meta-analysis
and then back-transformed to rs for interpreta-
tion (Fisher, 1924). Variability in effect sizes
across studies was examined using the I
2
statis-
tic, which quantifies the amount of “true” be-
tween-study heterogeneity in effect sizes not
due to chance (Huedo-Medina, Sánchez-Meca,
Marín-Martinez, & Botella, 2006). Magnitude
of effect sizes were based on Cohen’s guide-
lines (.20 for small, .50 for medium, .80 for
large; Cohen, 1988).
We conducted meta-analyses to provide an
overall estimate of supervision factors on ther-
apeutic process and client outcomes. Additional
meta-analyses on specific outcomes or supervi-
sion variables were only considered depending
on the sample size. To account for dependencies
between multiple observed correlations within a
single study (Schmidt & Hunter, 2015), we ag-
gregated associations within the study such that
there was one observed correlation between su-
pervision and outcome within the study. For
multiple observations of the same type of cor-
relations within a study (e.g., single study re-
porting two or more correlations between same
supervision variable and same client outcome/
therapeutic process, same correlations at differ-
ent time points, same correlations rated by both
supervisees and supervisors), these correlations
were aggregated. For different types of associ-
ations reported within a study due to multiple
variables on supervision or client outcomes
(e.g., supervision with client outcome vs. super-
vision with WA; supervisory WA with client
outcome vs. supervisory style vs. client out-
come), effects were not aggregated in order to
allow for different patterns of association across
outcomes.
We used a random-effects model which “as-
sumes that the average effect size in the popu-
lation varies randomly from study to study”
(Field & Gillett, 2010, p. 672). This model
allows for generalizability of the results that
extend beyond the studies included in the anal-
ysis (Field & Gillett, 2010). The meta-analysis
was conducted following the package provided
by Field and Gillett (2010). Outputs included an
estimate of the average effect size of all studies,
goodness of fit, and parameters of random-
effects model. Publication bias was examined
using R with the meta-analysis package “meta-
for” (Viechtbauer, 2014). Funnel and forest
plots were generated based on the results.
Results
Descriptive Characteristics
Twelve studies (21 effects; 32 before aggre-
gation) met the inclusion criteria of studies that
assessed supervision in relation to the therapeu-
tic process and client outcomes (see Tables 1, 2,
and 3). Of the 32 original effects (see Table 3),
17 (53%) examined the association between
supervision and treatment outcomes, 12 (38%)
examined the association between supervision
and client satisfaction, and three (9%) examined
the association between supervision and thera-
peutic relationship. Of the studies that provided
demographic data, the mean number of sessions
clients received was 15.97 sessions (median
5.08, range 4 –22.2); the mean number of clients
was 377 (median 96; range 37–1,979);
7META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Table 1
Major Demographics of Clients, Therapists, and Supervisors
Clients Therapists Supervisors
Study NAge (M,SD) % Fem % REM NAge (M,SD) % Fem % REM NAge (M,SD) % Fem % REM
Bell, Hagedorn, & Robinson, 2016 88 Modal 21
(range 18–59)
66 48 55 Modal 23
(range 22–59)
87 33 —
Dodenhoff, 1981 44 12 42
Gnilka, Chang, & Dew, 2012 232 32.8 (10.09) 86 45
McCarthy, 2013 166 46.50 (12.24) 74 10
Morrison & Lent, 2018 240 28.10 (4.31) 82 24
Patton & Kivlighan, 1997 75 20.12 (1.30) 79 11 75 27.71 (10.24) 71 15 25 32.30 (5.10) 72 100%White
Perez, 2018
32 40% 25 to 34;
35% 35 to 44
80 62 19 Demographics aggregated with
therapists’
Reese et al., 2009 110 19.54 (2.15) 71 20 28 25.14 (3.49) 64 21 9 44 100%White
Rieck, Callahan, & Watkins, 2015 256 10.75 (30.73) 49 18 32 26.41 (2.15) 66 28 13 49 (8.40) 46 15
Schoenwald, Sheidow, &
Chapman, 2009
1979 Y 14.0 (2.35),
C40.8 (8.48)
Y35
C88
Y48
C41
429 74 — 122 78
Van Dam, 2014
37 31.9 (14.5) 60 54 37 31.4 (8.8) 78 27 37 43.4 (10.2) 89 11
Zarski, Bubenzer, & Walter, 1980 96 32 33.4 (range22–45) 75 4 100%Male
Note. Fem female; REM racial and ethnic minority; Y youth; C caregivers.
Unpublished dissertation.
8 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Table 2
Descriptives of Supervisor and Supervision
Study Education/Degrees Amount of supervision Modality
Theoretical
orientation Clinical context
Bell et al., 2016 Practicum at a
University-based
clinic
Dodenhoff, 1981 8 Faculty, 4 students
(2 doctoral, 2
master’s)
16 sessions Two-way mirror observation;
direct and indirect
supervision
University counselor
training center
Gnilka et al., 2012 Multiple 1 (see
note)
McCarthy, 2013 M2.35 years (SD 2.51) with current
supervisor; M112.41 mins/week
(SD 300.00) on administration, M
69.61 mins/week (SD 203.76) on
individual therapy, M173.02 mins/
week (SD 512.59) on group therapy
Individual (15% supervision
contract, 55% non-regular
supervision)
5 State/Federal
vocational
rehabilitation
programs
Morrison & Lent, 2018 M8.26 months (SD 6.56) with
current supervisor
Individual Multiple 2 (see
note)
Patton & Kivlighan, 1997 Doctoral 4 sessions; each supervisor supervised 3
counselors
One-way mirror observation,
individual, group (for
supervisors)
TLDP Practicum
Perez, 2018
14 had supervision
licensure with
3 years of
experience
Multiple unspecified
Reese et al., 2009 Full-time/adjunct
faculty
Weekly each semester Individual, group CBT, S-F, S University-based
clinic serving
community
Rieck et al., 2015 Psychologists 1 hour weekly for individual, 2 hours
weekly for group
Individual, group Multiple Training clinic
Schoenwald et al., 2009 Bachelor’s,
Master’s, and
Doctoral students
Weekly 1–2 hour Group MST Home-based model
Van Dam, 2014
Licensed field
supervisors
Weekly Individual TLDP University based
clinical training
Zarski, Bubenzer, & Walter,
1980
Full-time faculty Minimum 60 hours over 15 weeks; 2-
hour weekly seminar
Individual, seminar Practicum at schools
and community
agencies
Note. TLDP Time-Limited Dynamic Psychotherapy; CBT Cognitive Behavioral Therapy; S-F Solution-Focused; S Systems; MST Multi-Systemic Theory; Multiple 1
community mental health agency, faith-based agency, hospital, private practice, school, university counseling center, some unspecified; Multiple2counseling center, community, training
clinics, hospital.
Unpublished dissertation.
9META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Table 3
Measures of Studies of Supervision Process and Client Outcomes
Study
Supervision
variable rater
Supervision
process variable
Outcome
variable rater
Client outcome
variable Original r
Reversed
rfor study
Bell et al., 2016 Supervisee BLRI Client BLRI .22 —
OQ-45 .15 .15
Dodenhoff, 1981 Supervisee Attraction (CRF) Supervisor Rating Scale for Outcome .30
Direct/Indirect Supervisory Style .35
Gnilka et al., 2012 Supervisee SWAI-Trainee Supervisee WAI-Short .22
McCarthy, 2013 Supervisee SWAI-Trainee Supervisee Status 26 .049
Morrison & Lent, 2018 (ag) Supervisee SWAI-Trainee Supervisee GAF (reversed) .08 .08
RISE .15 .15
RISE-Other Efficacy .06 .06
Patton & Kivlighan, 1997 Supervisee SWAI Client WAI .66
Perez, 2018
(ag) Supervisor SWAI-Supervisor Supervisor Perceived Client Outcomes (SOS) .33
Satisfaction (TPRS) .17
CCI-R .68 —
Supervisee SWAI-Supervisor Supervisee .21
Satisfaction (TPRS) .17
CCI-R .25 —
Reese et al., 2009 (ag) Supervisee Satisfaction (SOS) Client SRS .01 (T1), .72 (T2)
SWAI .37 (T1), .89 (T2)
Rieck et al., 2015 (ag) Supervisor WAI-Supervision Client OQ-45 .23 .23
Supervisee WAI-Supervision .18 .18
Schoenwald et al., 2009 (ag) Supervisee SAM Client CBCL (change) .20 .20
CBCL (early reductions) .17 .17
VFI .19 .19
Van Dam, 2014
(ag) Supervisor SWAI Client CGA .13 —
SSI .15 —
STI .22 —
Supervisee SWAI .16 —
SSI .01 —
STI .10 —
Zarski, Bubenzer, & Walter, 1980 Supervisor CERS Client CEI .40
Note. (ag) Studies with aggregated correlations; BLRI Barrett Lennard Relationship Inventory; CRF Counselor Rating Form; OQ-45 Outcome Questionnaire-45;
SWAI Supervisory Working Alliance Inventory-Trainee Version; WAI Working Alliance Inventory; GAF Global Assessment of Functioning; RISE Relation Inferred
Self-Efficacy; TPRS Trainee Personal Reaction Scale; SOS Supervision Outcome Scale; CCI-R Cross-Cultural Inventory-Revised; SRS Session Rating Scale; SAM
Supervisor Adherence Measure; CBCL Child Behavior Checklist; VFI Vanderbilt Functioning Inventory; SSI Supervisory Style Inventory; STI Supervisory Tools
Inventory; CGA Client Goal Attainment; CERS Counselor Evaluation Rating Scale CEI Counselor Evaluation Inventory; T1 Time 1; T2 Time 2.
Unpublished dissertation.
10 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
mean number of therapists was 117 (median
49.5; range 28 429); mean number of super-
visors was 30 (median 16; range 4 –122).
As seen in Table 1, across studies, the majority
of clients and therapists were female while the
gender seemed more balanced for supervisors;
majority of the sample were White among cli-
ents, therapists, and supervisors. Across studies,
the mean age of clients was 25.44 (median
21; range 14 41); the mean age of therapists
was 30.03 (median 27.91; range 23– 47);
and the mean age of supervisors was 37.65
(median 37.85, range 26 – 49). Six studies
(50%) were conducted within a training clinic,
four (33%) were multisite, one (8%) was voca-
tional rehabilitation programs, and one (8%)
home-based. See Table 2 for details of descrip-
tives of supervisor and supervision.
Overall Effect Between Supervision and
Psychotherapy Process/Outcome
The effect sizes for the various outcomes are
listed in Figure 2 and Figure 3. As hypothesized,
we found a significant overall pooled Pearson’s
correlation between supervision variables and cli-
ent outcomes/therapeutic process, r.191, k
21, CI [.118, .263]. The overall pooled effect was
small. The heterogeneity in effect size was mod-
erate (Higgins, Thompson, Deeks, & Altman,
2003), I
2
54.13%, Q(20) 43.601, p.002.
The pooled correlation suggested that supervision
accounted for 4% of the variance in client out-
comes.
Given that four of the studies reported on ther-
apist/supervisor-reported client outcomes or ther-
apeutic process (Gnilka et al., 2012; McCarthy,
2013; Morrison & Lent, 2018; Perez, 2018), we
ran an additional meta-analysis without these
studies to calculate an effect size based on only
client-rated outcomes. The pooled Pearson’s cor-
relation between supervision variables and client
outcomes/therapeutic process was, r.242, k
14, CI [.145, .334]. The effect size was small and
slightly larger than the effect size from the overall
analysis (.19). The heterogeneity in effect size was
moderate (Higgins et al., 2003), I
2
55.19%,
Q(13) 29.008, p.007. The pooled correlation
suggested that supervision accounted for 6% of
the variance in client outcomes.
Publication Bias
Although not as recommended due to depen-
dence on significance testing, Rosenthal’s fail-
safe Nwas 1,109, suggesting that there would
Figure 2. Funnel plot of precision by Fisher’s Z.
11META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
need to be 1,109 unpublished studies not in-
cluded in the meta-analysis to make the popu-
lation effect size nonsignificant. For visual ex-
amination of publication bias, we assessed the
funnel plot (see Figure 2) with effect sizes on
the X-axis and standard error (index of preci-
sion) on the Y-axis (Egger, Smith, Schneider, &
Minder, 1997). A funnel plot that looks roughly
symmetrical around the population effect size
suggests publication bias may not be an issue as
small studies are scattered near the bottom and
larger studies are clustered toward the top closer
to the mean effect in a symmetrical fashion.
Asymmetrical scattering suggests possible pub-
lication bias, in that greater clustering of studies
on either side of the mean effect indicates bias
to publish studies with either small or larger
effect, or even positive or negative effect. For
our funnel plot, we observed a symmetrical
scattering except toward the bottom right where
we observed a deficit of studies, suggesting
possible publication bias to publish studies with
small effect. Thus, to examine the robustness of
the meta-analytic finding, we tested for the pres-
ence of publication bias using Vevea and
Woods (2005) sensitivity analysis. The results
from adjusted parameter estimates showed that
under severe selection bias, the rchanges from
.191 to .174, while under moderate bias, it
changed to .190. Therefore, the supervision ef-
fect appeared robust even after correcting for
moderate to severe selection bias.
Discussion
About two decades ago, Wampold and Hol-
loway (1997) predicted that supervision likely
has “extremely small” effects on client out-
comes due to diverse intermediary variables
across the supervisor–therapist– client triad that
make it difficult to tease apart the true effects of
supervision on client outcomes. Since then, nu-
merous reviews have also noted the method-
ological limitations and lack of clarity on
Figure 3. Forest plot of random effects (r) between confidence and accuracy. Each reffects
are represented by squares. Lines represent the 95% confidence intervals (CIs) for each study.
The overall random effects rreflects the mean weighted overall confidence-accuracy effect.
BLRI Barret Lennard Relationship Inventory; CGA Client Goal Attainment; GAF
Global Assessment of Functioning; OQ45 Outcome Questionnaire-45; SPCO Supervi-
sor/Supervisee Perceived Client Outcomes; VFI Vanderbilt Functioning Inventory;
CBCL Child Behavior Check List; WAI Working Alliance Inventory; RSO Rating
Scale for Outcomes; SRS Session Rating Scale; CEI Counseling Evaluation Inventory.
12 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
whether supervision improves client outcomes
(Freitas, 2002; Milne et al., 2011; Rousmaniere
et al., 2016; Watkins & Callahan, 2019). Our
findings provide some quantitative evidence on
the relationship between supervision and thera-
peutic process and outcomes. Our meta-analytic
review of 12 studies with 32 effects suggested
that supervision has small positive effects on
client outcomes and therapeutic processes, r
.19 (rs ranged from .22 to .66). Supervision
accounted for approximately 4% of the variance
in client outcomes and therapeutic processes.
For context, this is larger than the .04% of
variance that supervisors account for in client
outcomes (Rousmaniere et al., 2016) but
smaller compared to therapist effects. Most
studies on therapist effects have consistently
reported that small proportion (5 to 10%) of the
variance in treatment outcomes were accounted
by therapist effects (Baldwin & Imel, 2013).
The overall positive associations between su-
pervision and client outcomes appeared to be
small in effect size and in general, likely smaller
than therapist effects.
The small overall effect must be considered
with context. Although we were not able to
conduct additional meta-analyses on specific
categories of outcomes (therapeutic relation-
ship, treatment outcomes, client satisfaction) or
test moderators due to limited sample size
(Bloch, 2014), we did observe some nuanced
differences in the effect sizes across these out-
comes. Regarding the effects on therapeutic re-
lationship, effect sizes on WA ranged from .22
(therapist rated; Gnilka et al., 2012) to .66 (cli-
ent rated; Patton & Kivlighan, 1997). Both of
these studies assessed supervision with the
SWAI, suggesting that the parallel WA associ-
ations between supervisory (supervisor–thera-
pist) and therapeutic (therapist– client) relation-
ships may be small or moderate. Although some
qualities (e.g., therapist factors) of the supervi-
sory and therapeutic relationships may be
shared, other qualities pertaining to supervisor
and client characteristics (Swift, Greenberg,
Whipple, & Kominiak, 2012) may uniquely dif-
ferentiate the strength of the WA in the respec-
tive dyads (Wampold & Holloway, 1997). In
line with the latter point, the weak relationship
(low correlation) found in Gnilka and col-
leagues (2012) suggests that the WA across
supervisory and therapeutic relationships may
be more distinct than related. Hence, the
strength of the WA in the therapeutic relation-
ship may be independent of the strength of the
supervisory relationship overseeing the respec-
tive therapy work. It should also be noted that
the therapeutic WA in this study was rated by
the therapists. Therapists in the Gnilka and col-
leagues (2012) study may have underestimated
or overestimated their WA with clients, a bias
that may have contributed to the low correlation
between WA across therapeutic and supervisory
relationships. On the other hand, the moderate
correlation found in Patton and Kivlighan
(1997) study suggests stronger positive parallel
associations among WA across therapeutic and
supervisory relationships. Thus, a strong WA in
the supervisory relationship may be related to a
strong WA in therapeutic relationship. It is pos-
sible that there are shared transferable elements
across the two relationships regarding the WA
(Mcaleavey et al., 2014). For example, a thera-
pist engaged in strong WA with her supervisor
may be able to model such relationship with her
clients, which may ultimately have benefits on
client outcomes (Bernard & Goodyear, 2014).
Beyond WA, Bell and colleagues (2016)
found a small negative effect between the inter-
personal relationship established in supervision
and therapy (e.g., strong relationship in super-
vision but weak relationship in therapy). It
would be difficult to generalize this negative
effect given that only one study examined the
real relationship qualities, but as with WA, the
findings suggest that stronger relational quali-
ties in supervision may not necessarily translate
to similar qualities in the therapeutic relation-
ship. In fact, Bell and colleagues’ (2016) study
demonstrated this case as therapists who had
strong interpersonal relationships with their su-
pervisors were associated with weaker interper-
sonal relationships with their clients; a reason-
able and common scenario of therapists who
maintain strong relationships with their super-
visors while working with difficult, resistant, or
interpersonally distant clients.
Whereas supervision consistently had small
effects on the therapeutic relationship, the ef-
fects on treatment outcomes were more varied
(rs ranged from .08 to .35). Effects from
studies that examined client-rated treatment
outcomes on psychometrically adequate symp-
tom measures (OQ-45, VFI, CBCL) were con-
sistently small (rs ranged from .03 to .19). The
effects varied even more for studies that exam-
13META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ined supervisor or therapist-rated treatment out-
comes (vs. client-rated); effects on the GAF
were very small and negative (Morrison & Lent,
2018), and effects on the perceived client out-
comes (SOS; Tsong & Goodyear, 2014) ranged
from small to medium (Perez, 2018). The vari-
ability in effect sizes are likely due to the dif-
ferent treatment outcome variables and mea-
sures used in the studies. However, the
deviation of effect sizes from the overall small
effects on treatment outcomes in both Morrison
and Lent (2018) and Perez (2018) studies sug-
gest a possibility that supervisor or therapist
biases can create complexities in understanding
the role of supervision on client outcomes. In-
deed, literature has documented therapists’ un-
derestimation (e.g., Atzil-Slonim et al., 2015;
Keum, Morales Dixon, Kivlighan, Hill, &
Gelso, 2020) and overestimation (e.g., Brosan,
Reynolds, & Moore, 2008) in predicting their
competence and perceived client outcomes.
Thus, evaluating supervision– client outcome
associations may need to prioritize client ratings
on the outcomes.
Regarding client satisfaction, the effects
found in our study were most varied (rs ranged
from .13 to .63). Specifically, small negative
or very small effects (.15 to .08) were found
among studies that assessed client goal attain-
ment (general and vocational; McCarthy, 2013;
Van Dam, 2014). On the other hand, effects on
clients’ evaluation of sessions were moderate
(.37 to .63; Reese et al., 2009; Zarski et al.,
1980). All of these studies assessed supervisory
WA and satisfaction with supervision, suggest-
ing that the variance may likely be due to dif-
ferences in the two outcomes. It is possible that
effects on clients’ evaluation of sessions may be
higher given that they provide assessment of
their immediate, process-oriented satisfaction of
their sessions compared to assessment of their
therapeutic goals that may require longer time
and more therapeutic engagements to gauge im-
provements toward achieving the said goals.
Perhaps, the impact of supervision on client
satisfaction may be more noticeable in session-
to-session changes rather than the longer-term
impact on their therapeutic goals. Although the
number of studies are limited, the two client
satisfaction studies (Reese et al., 2009; Zarski et
al., 1980) employed longitudinal designs with
multiple data points (e.g., after every session)
whereas the two goal attainment studies (Mc-
Carthy, 2013; Van Dam, 2014) respectively em-
ployed a cross-sectional correlation design and
a longitudinal design with only pre- and posttest
data collection. It is possible that the designs in
latter studies were not able to capture the ses-
sion-to-session changes in client outcomes due
to supervision and only reflect the retrospective
net gains at the end of the therapy work or at a
certain time point during the work.
Regarding how supervision quality was as-
sessed in the studies, supervisory relationship
was most frequently examined (k10) with
eight effects based on supervisory WA (rs rang-
ing from .03 to .66) and two effects based on
interpersonal relationship (rs ranging from
.22 to .15). Remaining effects were based on
more specific factors in supervision including,
attraction to supervisor (.30), supervisory style
(.08 to .35), supervisor estimation of therapist’s
self-efficacy (.05), multicultural supervision
(.47), therapists’ satisfaction of supervision (.17
to .37), supervisor adherence to supervision
structure and process (.19, .19), and supervisory
tools and activities (.06). Compared to the wider
variance in effects based on supervisory WA,
there seemed to be less variance among specific
supervision factors (e.g., style, satisfaction,
structure) with consistent small to medium pos-
itive effects. Supervisory relationships, in par-
ticular, supervisory WA, are dyadic in nature
and likely affected by random factors in the
relationship that may disperse the impact on
client outcomes. On the other hand, skills-based
(e.g., tools, activities), structural (e.g., style,
structure), or tailored supervision approaches
(e.g., client feedback-based supervision) may be
more modular and systematic, which may trans-
late into the therapy work with consistent effect.
Design-wise, all but four studies employed a
longitudinal design, with majority of studies
collecting data at multiple time points ranging
from couple weeks to several years. However,
most studies conducted correlational data anal-
yses and only two studies employed hierarchi-
cal linear modeling. Thus, we are in need of
more longitudinal studies that effectively ac-
counts for the nesting within the supervisor–
therapist– client triad in examining the associa-
tion between supervision and client outcomes.
Collectively, although the overall effect was
small, several factors emerged in our meta-
analysis that may be helpful in nuanced assess-
ment of the impact of supervision on client
14 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
outcomes. First, especially regarding the impact
on treatment outcomes, supervision effects on
client outcomes may be more interpretable or
recognizable in relation to client-rated out-
comes rather than therapists’ or supervisors’
ratings due to introduction of biases (e.g., un-
der- or overestimation). Second, based on the
effects on client satisfaction and longitudinal
designs, nuanced benefits due to supervision
may be demonstrable in the session-to-session
changes in client outcomes rather than out-
comes that assess longer-term impact or end of
therapy outcomes. Finally, examination of par-
allel associations between supervisory and ther-
apeutic relationships (e.g., WA) may not be as
informative given that many mutually exclusive
random factors such as supervisor and client
characteristics can dictate the trajectory of re-
spective relationships. Rather, specific, modu-
lar, systematic factors such as supervisor style,
supervision activities, and therapists’ satisfac-
tion of supervision may provide more consistent
positive effects and explanatory context.
Limitations and Implications for
Future Research
There are several noteworthy limitations re-
garding our results that call for future research.
First, our comprehensive search of articles re-
sulted in just 12 studies with 32 effects. While
this sample size can still be used to conduct a
meta-analysis (Turner, Bird, & Higgins, 2013),
we were not able to conduct additional specific
meta-analyses to tease apart the effects based on
different types of outcomes (e.g., treatment out-
comes, therapeutic relationship, client satisfac-
tion) and test potential moderators (e.g., clinical
setting, training approaches, theoretical orienta-
tion, supervisor credentials, number of super-
visees) that may have provided additional con-
text and sources of variance in how supervision
effects may be differentiated and affect thera-
pists’ work with clients. For example, it is pos-
sible that the types of therapy models may in-
teract with the supervision process and the
resulting therapeutic work given that different
theoretical orientations have been found to yield
differences in effect size in attending to specific
presenting problems (Wampold, 2013). Simi-
larly, differences in training frameworks may
moderate the supervision process, such as em-
phasis on clinical practice or research, selection
criteria of trainees, and the predominant therapy
models espoused by training programs. Future
research and critique of the role of supervision
on therapy outcomes should attend to these
moderators.
Part of this limitation also represents the
shortcomings in the field; there is still a shortage
of research that strives to examine the associa-
tion between supervision and client outcomes.
We feel that our inclusion criteria were not
stringent; for example, we did not consider any
perspective restrictions, such as excluding stud-
ies that did not report on client-rated outcomes.
In essence, more studies need to be conducted
regarding supervision and client outcomes with
consideration of the factors that we have iden-
tified in refining the effects. For example, it
would be important for future studies to ensure
client-rated outcomes rather than relying on the
therapist’s or supervisor’s ratings to reduce ob-
server bias. When we conducted a post hoc
meta-analysis using only studies that provided
client rated outcomes (eight studies, 14 effects),
the effect size increased to .24, accounting for
6% of the variance in client outcomes and ther-
apeutic processes. With more studies, future
meta-analyses can further tease apart the effects
across types of outcomes and examine moder-
ators that can explain the moderate heterogene-
ity across the studies and how supervision ef-
fects may be dispersed via mediating variables
at the supervisor, therapist, and client levels.
Second, almost all of the studies have re-
ported on supervision variables rated by super-
visees. Thus, we are limited in what we can say
about the perspective of supervisors. Although
supervisee perceptions are important, supervi-
sion is a relational process and not exclusive to
the therapist. For example, perceived supervi-
sory WA may vary between the supervisees and
supervisors, and may differentially associate
with client outcomes. Incorporating supervi-
sors’ ratings would be important to comprehen-
sively examine the supervision experience as
there may be agreement or discrepancies in
supervision between the supervisors and super-
visees that may impact the therapy work. Fur-
thermore, future studies can also test supervisor
specific variables such as supervisor orienta-
tions, models employed for supervision, provi-
sion of supervision structure, and level of em-
pathy.
15META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Third, although a variety of supervision mea-
sures were noted in the studies, our meta-
analysis highlights the lack of measures assess-
ing the content and process (e.g., coverage of
skills and theories, structure of supervision,
practice opportunities, etc.), and domain-
specific modes of supervision. For example,
only two studies examined structure in supervi-
sion (Schoenwald et al., 2009) and the tools and
activities employed in supervision (Van Dam,
2014). Only one study examined multicultural
supervision (Perez, 2018). Part of this limitation
may be due to the fact that supervision is not
standardized and takes on a variety of forms and
approaches. However, it would be important for
future studies to start addressing the effective-
ness of certain tasks, structures, modules, and
behavioral interventions conducted in supervi-
sion to examine aspects that can refine the im-
pact on client outcomes.
Fourth, the overall pooled Pearson’s correla-
tion found in our meta-analysis assumes that
supervision is uniformly associated with client
outcomes. Although we have provided some
context in interpreting the effect size differences
across the outcomes, it is likely that the associ-
ation between supervision and client outcomes
is a nonlinear process. For example, the positive
and negative supervision effects on WA (Gnilka
et al., 2012) and interpersonal relationships
(Bell et al., 2016) are pieces of evidence sug-
gesting that the process may fluctuate over time.
Additionally, supervision may affect therapist’s
growth, but that effect may not immediately
show up in the therapy work. Instead, there may
be a lagged effect over time on client outcomes
as therapists take time to work through their
own issues (e.g., countertransference), aware-
ness, and skill improvements in supervision.
These unique trajectories of supervision should
be explored in future studies.
Fifth, the overall effect found in our study
needs to be interpreted with caution given that
most studies did not account for the nesting of
the client–therapist–supervisor data structure.
For example, the Zarski et al. (1980) study
included data from 96 clients seen by 32 ther-
apists working with four supervisors. Pearson’s
rcomputed on these 96 observations ignores the
nested structure of the dataset, resulting in bias
in estimates of the standard error of r.
Finally, our results may overlook evidence-
based approaches that can be used to optimize
client outcomes via supervision. For instance,
scholars have recently called for incorporation
of client feedback systems into supervision so
that changes in client data can be actively and
routinely addressed in supervision (Swift et al.,
2015; Worthen & Lambert, 2007). In fact, one
study included in our meta-analysis tested client
feedback systems (Reese et al., 2009) and have
yielded larger effects (rs.37 and .63) on
client satisfaction compared to other studies.
Providing a tailored supervision according to
client’s progress and outcome data may be an
efficacious supervision intervention given that
specific presenting concerns can be localized
and targeted more effectively in therapy. On the
other hand, a qualitative study by McComb,
Diamond, Breunlin, Chambers, and Murray
(2019) found that additional time needed to
incorporate a client feedback system actually
led to decreased usage, and Grossl, Reese, Nor-
sworthy, and Hopkins (2014) found that there
were no significant differences in clients’ treat-
ment outcome between feedback supervision
and supervision-as-usual conditions. Despite
the potential, it seems that constraints of imple-
mentation and intervention refinements must be
addressed for practical impact. Regardless, fu-
ture studies need to examine evidence-based
supervision approaches such as client feedback
systems.
Conclusion
In summary, we conducted a meta-analytic
review (12 studies with 32 effects identified) to
assess whether supervision is associated with
improvements in client outcomes. We found
that supervision accounted for 4% of the vari-
ance in client outcomes. When the four studies
that assessed client outcomes using supervisor/
therapist ratings were excluded, we found that
supervision accounted for 6% of the variance in
client outcomes. The effect sizes were consid-
ered small (.19 and .24). In unpacking these
results, we observed context-dependent varia-
tions in effect sizes across the outcome do-
mains. Effect sizes regarding the therapeutic
relationship and client satisfaction varied
widely from small negative, to small and me-
dium positive effects, while effect sizes for
treatment outcomes were less varied with con-
sistently small positive effects. Based on our
review, we deduced several suggestions that
16 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
may be helpful for nuanced assessment of the
impact of supervision on client outcomes; re-
ducing rater bias by focusing on client-rated
outcomes, examining the role of supervision on
session-to-session changes in client outcomes,
and testing possible moderators (e.g., theoreti-
cal orientation, training models/programs) that
may impact the supervisory relationship and
therapy work.
References
Sampled studies in the meta-analysis.
American Psychological Association. (2015). Guide-
lines for clinical supervision in health service psy-
chology. American Psychologist, 70, 33– 46. http://
dx.doi.org/10.1037/a0038112
Atzil-Slonim, D., Bar-Kalifa, E., Rafaeli, E., Lutz,
W., Rubel, J., Schiefele, A. K., & Peri, T. (2015).
Therapeutic bond judgments: Congruence and in-
congruence. Journal of Consulting and Clinical
Psychology, 83, 773–784. http://dx.doi.org/10
.1037/ccp0000015
Baldwin, S. A., & Imel, Z. E. (2013). Therapist
effects: Findings and methods. Bergin and Gar-
field’s handbook of psychotherapy and behavior
change (6th ed., pp. 258–297). New York, NY:
Wiley.
Bambling, M., King, R., Raue, P., Schweitzer, R., &
Lambert, W. (2006). Clinical supervision: Its in-
fluence on client-rated working alliance and client
symptom reduction in the brief treatment of major
depression. Psychotherapy Research, 16, 317–331.
http://dx.doi.org/10.1080/10503300500268524
Bartle-Haring, S., Silverthorn, B. C., Meyer, K., &
Toviessi, P. (2009). Does live supervision make a
difference? A multilevel analysis. Journal of Mar-
ital and Family Therapy, 35, 406– 414. http://dx
.doi.org/10.1111/j.1752-0606.2009.00124.x
Bell, H., Hagedorn, W. B., & Robinson, E. M.
(2016). An exploration of supervisory and thera-
peutic relationships and client outcomes. Coun-
selor Education and Supervision, 55, 182–197.
http://dx.doi.org/10.1002/ceas.12044
Bernard, J. M., & Goodyear, R. K. (2014). Funda-
mentals of clinical supervision (5th ed.). Upper
Saddle River, NJ: Pearson Education.
Bloch, M. H. (2014). Meta-analysis and moderator
analysis: Can the field develop further? Journal of
the American Academy of Child & Adolescent Psy-
chiatry, 53, 135–137. http://dx.doi.org/10.1016/j
.jaac.2013.12.001
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-
evaluation of cognitive therapy performance: Do
therapists know how competent they are? Behav-
ioural and Cognitive Psychotherapy, 36, 581–587.
http://dx.doi.org/10.1017/S1352465808004438
Callahan, J. L., Almstrom, C. M., Swift, J. K., Borja,
S. E., & Heath, C. J. (2009). Exploring the contri-
bution of supervisors to intervention outcomes.
Training and Education in Professional Psychol-
ogy, 3, 72–77. http://dx.doi.org/10.1037/a0014294
Cohen, J. (1988). The effect size index: d. In J. Cohen
(Ed.), Statistical power analysis for the behavioral
sciences (2nd ed., pp. 20–27). Mahwah, NJ: Law-
rence Erlbaum Associates.
Couchon, W. D., & Bernard, J. M. (1984). Effects of
timing of supervision on supervisor and counselor
performance. The Clinical Supervisor, 2, 3–20.
http://dx.doi.org/10.1300/J001v02n03_02
Crutchfield, L. B., & Borders, L. D. (1997). Impact of
two clinical peer supervision models on practicing
school counselors. Journal of Counseling & De-
velopment, 75, 219–230. http://dx.doi.org/10.1002/
j.1556-6676.1997.tb02336.x
Dodenhoff, J. T. (1981). Interpersonal attraction and
direct–indirect supervisor influence as predictors
of counselor trainee effectiveness. Journal of
Counseling Psychology, 28, 47–52. http://dx.doi
.org/10.1037/0022-0167.28.1.47
Duncan, B. L., & Moynihan, D. W. (1994). Applying
outcome research: Intentional utilization of the cli-
ent’s frame of reference. Psychotherapy: Theory,
Research, Practice, Training, 31, 294–301. http://
dx.doi.org/10.1037/h0090215
Efstation, J. F., Patton, M. J., & Kardash, C. M.
(1990). Measuring the working alliance in coun-
selor supervision. Journal of Counseling Psychol-
ogy, 37, 322–329. http://dx.doi.org/10.1037/0022-
0167.37.3.322
Egger, M., Smith, G. D., Schneider, M., & Minder,
C. (1997). Bias in meta-analysis detected by a
simple, graphical test. BMJ, 315, 629– 634. http://
dx.doi.org/10.1136/bmj.315.7109.629
Falender, C. A., & Shafranske, E. P. (2004). Clinical
supervision: A competency-based approach.
Washington, DC: American Psychological Associ-
ation. http://dx.doi.org/10.1037/10806-000
Field, A. P., & Gillett, R. (2010). How to do a
meta-analysis. British Journal of Mathematical
and Statistical Psychology, 63, 665– 694. http://dx
.doi.org/10.1348/000711010X502733
Fisher, R. A. (1924). The distribution of the partial
correlation coefficient. Metron, 3, 329–332.
Freitas, G. J. (2002). The impact of psychotherapy
supervision on client outcome: A critical examina-
tion of 2 decades of research. Psychotherapy: The-
ory, Research, Practice, Training, 39, 354–367.
http://dx.doi.org/10.1037/0033-3204.39.4.354
Gibson, J. A., Grey, I. M., & Hastings, R. P. (2009).
Supervisor support as a predictor of burnout and
therapeutic self-efficacy in therapists working in
ABA schools. Journal of Autism and Developmen-
17META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
tal Disorders, 39, 1024–1030. http://dx.doi.org/10
.1007/s10803-009-0709-4
Gnilka, P. B., Chang, C. Y., & Dew, B. J. (2012).
The relationship between supervisee stress, coping
resources, the working alliance, and the supervi-
sory working alliance. Journal of Counseling &
Development, 90, 63–70. http://dx.doi.org/10
.1111/j.1556-6676.2012.00009.x
Grossl, A. B., Reese, R. J., Norsworthy, L. A., &
Hopkins, N. B. (2014). Client feedback data in
supervision: Effects on supervision and outcome.
Training and Education in Professional Psychol-
ogy, 8, 182–188. http://dx.doi.org/10.1037/
tep0000025
Harkness, D., & Hensley, H. (1991). Changing the
focus of social work supervision: Effects on client
satisfaction and generalized contentment. Social
Work, 36, 506–512. http://dx.doi.org/10.1093/sw/
36.6.506
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., &
Altman, D. G. (2003). Measuring inconsistency in
meta-analyses. British Medical Journal, 327, 557–
560. http://dx.doi.org/10.1136/bmj.327.7414.557
Holloway, E. L., & Neufeldt, S. A. (1995). Supervi-
sion: Its contributions to treatment efficacy. Jour-
nal of Consulting and Clinical Psychology, 63,
207–213. http://dx.doi.org/10.1037/0022-006X.63
.2.207
Holloway, E. L., & Wampold, B. E. (1984, August).
Dimensions of satisfaction in the supervision in-
terview. Paper presented at the 92nd Annual Con-
vention of the American Psychological Associa-
tion, Toronto, Ontario, Canada.
Horvath, A. O., Del Re, A. C., Flückiger, C., &
Symonds, D. (2011). Alliance in individual psy-
chotherapy. Psychotherapy, 48, 9–16. http://dx.doi
.org/10.1037/a0022186
Horvath, A. O., & Greenberg, L. S. (1989). Devel-
opment and validation of the Working Alliance
Inventory. Journal of Counseling Psychology, 36,
223–233. http://dx.doi.org/10.1037/0022-0167.36
.2.223
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-
Martínez, F., & Botella, J. (2006). Assessing het-
erogeneity in meta-analysis: Qstatistic or I
2
index?
Psychological Methods, 11, 193–206. http://dx.doi
.org/10.1037/1082-989X.11.2.193
Karr, J. T., & Geist, G. O. (1977). Facilitation in
supervision as related to facilitation in therapy.
Counselor Education and Supervision, 16, 263–
268. http://dx.doi.org/10.1002/j.1556-6978.1977
.tb01024.x
Keum, B. T., Morales Dixon, K., Kivlighan, D. M.,
Jr., Hill, C. E., & Gelso, C. J. (2020). Do therapists
improve in their ability to assess clients’ satisfac-
tion? A truth and bias model. Journal of Counsel-
ing Psychology. Advance online publication.
http://dx.doi.org/10.1037/cou0000525
Kivlighan, D. M., Jr., Angelone, E. O., & Swafford,
K. G. (1991). Live supervision in individual psy-
chotherapy: Effects on therapist’s intention use
and client’s evaluation of session effect and work-
ing alliance. Professional Psychology: Research
and Practice, 22, 489– 495. http://dx.doi.org/10
.1037/0735-7028.22.6.489
Ladany, N., & Inman, A. G. (2012). Training and
supervision. In E. M. Altmaier & J. C. Hansen
(Eds.), The Oxford handbook of counseling psy-
chology (pp. 179–207). New York, NY: Oxford
University Press.
Lainas, H. L. (2014). The relationship between su-
pervisory working alliance and supervisees’ client
outcomes (Unpublished doctoral dissertation). The
University of North Carolina at Charlotte, Char-
lotte, NC.
Lambert, M. J., & Arnold, R. C. (1987). Research
and the supervisory process. Professional Psychol-
ogy: Research and Practice, 18, 217–224. http://
dx.doi.org/10.1037/0735-7028.18.3.217
Lampropoulos, G. K. (2003). A common factors
view of counseling supervision process. The Clin-
ical Supervisor, 21, 77–95. http://dx.doi.org/10
.1300/J001v21n01_06
Linden, J. D., Stone, S. C., & Shertzer, B. (1965).
Development and evaluation of an inventory for
rating counseling. The Personnel and Guidance
Journal, 44, 267–276. http://dx.doi.org/10.1002/j
.2164-4918.1965.tb03516.x
Mastroleo, N. R., Magill, M., Barnett, N. P., & Bor-
sari, B. (2014). A pilot study of two supervision
approaches for peer-led alcohol interventions with
mandated college students. Journal of Studies on
Alcohol and Drugs, 75, 458– 466. http://dx.doi
.org/10.15288/jsad.2014.75.458
Mcaleavey, A. A., Castonguay, L. G., & Xiao, H.
(2014). Therapist orientation, supervisor match,
and therapeutic interventions: Implications for ses-
sion quality in a psychotherapy training PRN.
Counselling & Psychotherapy Research, 14, 192–
200. http://dx.doi.org/10.1080/14733145.2014
.929418
McCarthy, A. K. (2013). Relationship between su-
pervisory working alliance and client outcomes in
state vocational rehabilitation counseling. Rehabil-
itation Counseling Bulletin, 57, 23–30. http://dx
.doi.org/10.1177/0034355213484177
McComb, J. L., Diamond, R. M., Breunlin, D. C.,
Chambers, A. L., & Murray, K. S. (2019). Intro-
ducing client feedback into marriage and family
therapy supervision: A qualitative study examining
the transition to empirically informed supervision.
Journal of Family Therapy, 41, 214–231. http://dx
.doi.org/10.1111/1467-6427.12222
McCune, A. (2016). The impact of trainee self-
disclosure and openness to experience on clinical
supervision and client outcomes (Unpublished
18 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
doctoral dissertation). Adler University, Chicago,
IL.
Milne, D. L., Sheikh, A. I., Pattison, S., & Wilkinson,
A. (2011). Evidence-based training for clinical su-
pervisors: A systematic review of 11 controlled
studies. The Clinical Supervisor, 30, 53–71. http://
dx.doi.org/10.1080/07325223.2011.564955
Morrison, M. A., & Lent, R. W. (2018). The work-
ing alliance, beliefs about the supervisor, and
counseling self-efficacy: Applying the relational
efficacy model to counselor supervision. Journal
of Counseling Psychology, 65, 512–522. http://dx
.doi.org/10.1037/cou0000267
Patton, M. J., & Kivlighan, D. M., Jr. (1997). Rel-
evance of the supervisory alliance to the counsel-
ing alliance and to treatment adherence in coun-
selor training. Journal of Counseling Psychology,
44, 108–115. http://dx.doi.org/10.1037/0022-0167
.44.1.108
Perez, A. (2018). Multicultural counselor supervi-
sion and perceived differences on client outcome
(Unpublished doctoral dissertation). The Univer-
sity of Texas Rio Grande Valley, Brownsville, TX.
Reese, R. J., Usher, E. L., Bowman, D. C., Norswor-
thy, L. A., Halstead, J. L., Rowlands, S. R., &
Chisholm, R. R. (2009). Using client feedback in
psychotherapy training: An analysis of its influ-
ence on supervision and counselor self-efficacy.
Training and Education in Professional Psychol-
ogy, 3, 157–168. http://dx.doi.org/10.1037/
a0015673
Rieck, T., Callahan, J. L., & Watkins, C. E., Jr.
(2015). Clinical supervision: An exploration of
possible mechanisms of action. Training and Ed-
ucation in Professional Psychology, 9, 187–194.
http://dx.doi.org/10.1037/tep0000080
Rousmaniere, T. G., Swift, J. K., Babins-Wagner, R.,
Whipple, J. L., & Berzins, S. (2016). Supervisor
variance in psychotherapy outcome in routine
practice. Psychotherapy Research, 26, 196–205.
http://dx.doi.org/10.1080/10503307.2014.963730
Schmidt, F., & Hunter, J. (2015). Methods of meta-
analysis. London, UK: SAGE. http://dx.doi.org/10
.4135/9781483398105
Schoenwald, S. K., Sheidow, A. J., & Chapman,
J. E. (2009). Clinical supervision in treatment
transport: Effects on adherence and outcomes.
Journal of Consulting and Clinical Psychology,
77, 410– 421. http://dx.doi.org/10.1037/a0013788
Swift, J. K., Callahan, J. L., Rousmaniere, T. G.,
Whipple, J. L., Dexter, K., & Wrape, E. R. (2015).
Using client outcome monitoring as a tool for
supervision. Psychotherapy, 52, 180–184. http://dx
.doi.org/10.1037/a0037659
Swift, J. K., Greenberg, R. P., Whipple, J. L., &
Kominiak, N. (2012). Practice recommendations
for reducing premature termination in therapy.
Professional Psychology: Research and Practice,
43, 379–387. http://dx.doi.org/10.1037/a0028291
Tsong, Y., & Goodyear, R. K. (2014). Assessing
supervision’s clinical and multicultural impacts:
The Supervision Outcome Scale’s psychometric
properties. Training and Education in Professional
Psychology, 8, 189–195. http://dx.doi.org/10.1037/
tep0000049
Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013).
The impact of study size on meta-analyses: Exam-
ination of underpowered studies in Cochrane re-
views. PLoS ONE, 8, e59202. http://dx.doi.org/10
.1371/journal.pone.0059202
Van Dam, D. D. (2014). The relationship among
supervisory alliance, style, and supervisory tools,
and client goal attainment in social work field
supervision (Unpublished doctoral dissertation).
The Catholic University of America, Washington,
DC.
Vevea, J. L., & Woods, C. M. (2005). Publication
bias in research synthesis: Sensitivity analysis us-
ing a priori weight functions. Psychological Meth-
ods, 10, 428– 443. http://dx.doi.org/10.1037/1082-
989X.10.4.428
Viechtbauer, W. (2014). Metafor: Meta-analysis package
for R. (R package Version 1.9 –4) [Computer software].
Retrieved from http://cran.rproject.org/web/packages/
metafor/index.html
Wampold, B. E. (2013). The great psychotherapy
debate: Models, methods, and findings. Hillsdale,
NJ: Erlbaum. http://dx.doi.org/10.4324/97802
03893340
Wampold, B. E., Goodheart, C. D., & Levant, R. F.
(2007). Clarification and elaboration on evidence-
based practice in psychology. American Psychol-
ogist, 62, 616– 618. http://dx.doi.org/10.1037/
0003-066X62.6.616
Wampold, B. E., & Holloway, E. L. (1997). Meth-
odology, design, and evaluation in psychotherapy
supervision research. In C. E. Watkins, Jr., (Ed.),
Handbook of psychotherapy supervision (pp. 11–
27). Hoboken, NJ: Wiley.
Watkins, C. E., Jr. (2011). Does psychotherapy su-
pervision contribute to patient outcomes? Consid-
ering thirty years of research. The Clinical Super-
visor, 30, 235–256. http://dx.doi.org/10.1080/
07325223.2011.619417
Watkins, C. E., Jr., & Callahan, J. L. (2019). Psycho-
therapy supervision research—A status report and
proposed model. In S. G. De Golia & K. M.
Corcoran (Eds.), Supervision in psychiatric prac-
tice: Practical approaches across venues and pro-
viders (pp. 25–34). Washington, DC: American
Psychiatric Association.
Wheeler, S., & Richards, K. (2007). The impact of
clinical supervision on counsellors and therapists,
their practice and their clients. A systematic review
of the literature. Counselling & Psychotherapy Re-
19META-ANALYSIS OF SUPERVISION ON CLIENT OUTCOMES
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
search, 7, 54– 65. http://dx.doi.org/10.1080/
14733140601185274
Worthen, V. E., & Lambert, M. J. (2007). Outcome
oriented supervision: Advantages of adding sys-
tematic client tracking to supportive consultations.
Counselling & Psychotherapy Research, 7, 48–53.
http://dx.doi.org/10.1080/14733140601140873
Wrape, E. R., Callahan, J. L., Ruggero, C. J., &
Watkins, C. E., Jr. (2015). An exploration of fac-
ulty supervisor variables and their impact on client
outcomes. Training and Education in Professional
Psychology, 9, 35– 43. http://dx.doi.org/10.1037/
tep0000014
Zarski, J. J., Bubenzer, D. L., & Walter, D. A.
(1980). Counseling effectiveness and trainee help-
ing qualities: Another view. Counselor Education
and Supervision, 20, 15–21. http://dx.doi.org/10
.1002/j.1556-6978.1980.tb02027.x
Received March 24, 2020
Revision received June 11, 2020
Accepted July 4, 2020
20 KEUM AND WANG
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
... Additionally, an environmental climate including aspects such as university facilities, social support, supervision, and environmental factors was found to correlate highly and significantly with counseling self-efficacy (CSE) (Bagheri et al., 2012). Access to training programs may impact the supervisory working alliance and psychotherapy process and outcome (Keum and Wang, 2021). For SCs, their supervisor's role in providing equal resource access such as training is imperative in maintaining work effort, especially when it contributes to the therapeutic relationship, client satisfaction, and treatment outcomes (Keum and Wang, 2021). ...
... Access to training programs may impact the supervisory working alliance and psychotherapy process and outcome (Keum and Wang, 2021). For SCs, their supervisor's role in providing equal resource access such as training is imperative in maintaining work effort, especially when it contributes to the therapeutic relationship, client satisfaction, and treatment outcomes (Keum and Wang, 2021). The greater the support from their supervisor, the greater the sense of efficacy at work. ...
... A positive supervisory working relationship increased the counselors' willingness to disclose and perceive the evaluations and critiques received from supervisors as constructive feedback (Mehr et al., 2015). A collaborative supervisory alliance delivered higher clients' satisfaction and produced positive therapeutic outcomes (Keum and Wang, 2021). ...
Article
Full-text available
The concept of self-efficacy has been widely studied and shown to contribute to individuals’ job satisfaction. For counselors, the concept measures their belief in their ability to conduct counseling sessions. However, it is an understudied area. As Bandura states, self-efficacy and its sources should be investigated and measured within its domain, which in this case is school counseling. This study examined the impact on school counselors’ self-efficacy and job satisfaction of the personal and environmental factors: (a) mastery experience, (b) social persuasion, (c) vicarious learning, (d) physiological and affective state, (e) the access to training, and (f) perceived supervisor support of training. The cross-sectional study involved 541 Malaysian secondary school counselors nationwide via a random sampling-distributed questionnaire. Results which were analyzed using PLS-SEM, with importance-performance functionality embedded in it, indicated that mastery experience, access to training, and perceived supervisor support of training explained 45.6% variance in counseling self-efficacy and together with counseling self-efficacy, contributed 13.2% variance in job satisfaction among the school counselors. The importance-performance map analysis revealed supervisor support of training as of greatest importance in shaping counseling self-efficacy. Counseling self-efficacy partially mediated the relationship between mastery experience, access to training, supervisor support toward training, and job satisfaction Arising from this finding is a proposed theoretical framework in which efficacy information (i.e., mastery experience), environmental determinants (i.e., access to training and supervisor support of training) and cognitive determinant (i.e., counseling self-efficacy) corresponded together congruently and lead to higher job satisfaction. Suggestions are also made for training providers, content developers, and policymakers to include these factors in professional development training and continuous education, to sustain the wellbeing of school counselors.
... Practitioners generally find supervision highly important (Orlinsky & Ronnestad, 2005), but empirical findings on its effects are inconclusive and methodologically limited (Watkins, 2014). Accordingly, there has been a call for new, evidence-based approaches to supervision (Keum & Wang, 2021). In a significant randomised controlled trial (RCT) of supervision, Bambling et al. (2006) proposed that the effects of supervision on client outcomes may work through the mediating pathway of an improved working alliance. ...
... In addition, participants' descriptions of the CALL instructors indicate a strong supervisory alliance. According to meta-analyses, a strong supervisory bond contributes to the well-being and self-efficacy of the therapist (Keum & Wang, 2021). Thus, CALL and its instructors seemingly provided a positive and collaborative environment, which may have been sustained, and perhaps reenacted, by the psychologists in their subsequent work. ...
Article
A general lack of supervision outcome studies and new advancements within therapist training methods inspired us to develop a randomised controlled trial, Case formulation and ALLiance (CALL), testing the efficacy of a brief pre‐treatment training session. CALL combines generic case formulation and alliance principles with the supervision of a specific client. This qualitative study acts as a preliminary to the randomised controlled trial and aims to explore psychologists' experiences of participating in CALL. Semi‐structured interviews were conducted with seven psychologists in private practice, who participated in CALL between January and May 2020. The transcribed interviews were analysed using thematic analysis. The psychologists described CALL as a clinically useful and collegial experience but simultaneously conveyed a lack of recollection of the training session and doubts as to its enduring utilisation. CALL has the potential to positively influence the process and outcome of therapy through the fostering of case formulation and alliance management skills, as well as a strong supervisory alliance. However, the familiar content or brief format of the session, and participants' memory bias, may have limited the utilisation and recall of the intervention.
... Given research associated turnover with high caseloads and a lack of support, including supervision , adequate resources dedicated to both staffing and supervision are required to provide the best possible service to clients. While not definitive, research supports the link between supervision and client outcomes (Bambling et al., 2006;Keum & Wang, 2021;Schoenwald et al., 2009), further justifying this resource commitment. ...
Article
Supervision of staff in the helping professions is valued and considered important for effective service delivery, with expert consensus that the delivery of high-quality supervision strengthens the social service workforce. The demands of publicly funded settings provide compelling rationale for supervision to protect marginalized clients, while ensuring staff are supported in delivering quality services. As conceptual frameworks, models of supervision are one way to strengthen supervision through intentionally incorporating theory, best practices, and research. Theoretically grounded and integrative, the Stop Now and Plan (SNAP) clinical supervision model incorporates best practices, guidelines, and techniques informed by explanatory and practice theories, and research. This article presents a qualitative study exploring the experiences of supervisors and supervisees in children's mental health settings implementing this model of supervision for an evidence-based intervention. Reflexive thematic analysis was conducted using qualitative data collected from participant interviews (N = 20) directly following the completion of a 6-month implementation of the SNAP model of clinical supervision. Identified themes demonstrate the value of a structured model of supervision in supporting practitioner confidence, accountability, and professional development, while allowing supervisors to meet their responsibilities to their team and their organizations. The challenges associated with implementing a supervision model within publicly funded settings are highlighted, with suggestions for over coming these challenges provided. (PsycInfo Database Record (c) 2022 APA, all rights reserved).
... Thus, when supervisors perceived a stronger SWA, clients felt that their sessions were more effective, perhaps because trainees felt freer in sessions given that they felt supported by their supervisors. This finding is aligned with theory and empirical findings that "better" supervision is related to "better" client outcome (e.g., Bambling et al., 2006;Keum & Wang, 2021). The novel aspect of this result is that it occurs at the between-supervisor level of analysis, meaning that some supervisors were better than others at forming supervisory alliances and this superiority was linked with better client-rated session evaluations. ...
Article
Several theorists (Bandura, 1969; Hackney & Goodyear, 1984; Searles, 1955) suggest parallels between the relationship in supervision and the relationship in therapy. We examined supervisor and therapist trainee ratings of supervisory working alliance (SWA) in 1 week predicting client-rated therapeutic working alliance (TWA) and client-rated therapy session evaluation (TSE) in the following week as well as TWA and TSE ratings in 1 week predicting SWA ratings in the following week. Our data included 663 weeks of therapy nested within 28 trainees nested within 15 supervisors, disaggregated into differences between supervisors, differences within supervisors, and differences within trainees. At the between-supervisor level, when supervisors' trainees rated the SWA higher on average compared with other supervisors' trainees' average SWA ratings, their clients' average TWA rating was higher. In contrast, when supervisors rated the SWA higher on average compared with other supervisors' average SWA ratings, their trainees' clients' average TSE rating was higher but the average TWA rating was lower. At the within-supervisor level, when trainees rated a higher SWA on average compared with other trainees' average SWA ratings with the same supervisor, their clients' average TSE rating was higher. The theoretical prediction of parallel relationships in supervision and therapy was supported, but only for between-supervisor and within-supervisor differences in SWA. We found no evidence that week-to-week changes in SWA or client-rated TWA or TSE reflected parallel relationships. We provide suggestions for further research, including exploring the mechanisms through which supervision relates to the therapy process and outcome. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
Preprint
For over three decades in psychology, meta-analysis has been a popular methodological tool for summarizing effect sizes within a given research domain. Statistical meta-analytic summaries typically reflect a mean and associated variance (heterogeneity) estimate, and visual summaries of constituent effect sizes typically use forest and funnel plots. Although these plots are useful, they do not reveal the shape of the distribution of effect sizes directly. To remedy this gap, we offer a weighted histogram that is more interpretable and useful when depicting the distribution of effect sizes and their associated sampling error variance. In support of the weighted histogram, we reviewed a hundred of the most recent meta-analyses published in American Psychological Association (APA) journals and in Journal of Applied Psychology, many popular books in psychology on meta-analysis, and several software programs and packages for conducting meta-analysis—all of which suggest a strong need for a more succinct and effective way to visualize the distribution and accuracy of effects included in a meta-analysis (i.e., a weighted histogram). Importantly, we also offer a user-friendly interactive online app (developed using R Shiny) that allows meta-analytic researchers to easily create their own publication-ready weighted histograms.
Chapter
Im folgenden Kapitel beschreiben wir den Regelkreis sowie die Gestaltung und Evaluierung der Kompetenzentwicklung. Der Hauptteil dieses Kapitels beschäftigt sich mit verschiedenen Formen der Kompetenzentwicklung, gegliedert in Training out of the job, Training off the job, Training near the job, Training on the job sowie Training by the job.
Article
Full-text available
Racial/ethnic concordance between patients and providers concerning the quality of care has received interest over past decades yielding mixed results. Patients seem to prefer clinicians of their own race/ethnicity, but empirical studies have found small or inconsistent effects on the quality of care. Research on the impact of racial/ethnic concordance and treatment duration appears to suggest that racial/ethnic concordance is associated with retention and completion; however, exactly why racial/ethnic concordance improves treatment length remains unexplored. On the other side, the quality of working alliance is a well-established common factor underlying effective treatments. In this study, we examined the interaction between patient-provider racial/ethnic concordance, length of treatment in the therapeutic dyad, and working alliance as evaluated by both patients themselves and objective raters. The study included 312 patients and 74 providers from 13 community and hospital-based outpatient mental health clinics in Massachusetts. Results indicated that among racial/ethnic concordant therapeutic dyads, longer length of treatment was associated with better quality of working alliance as evaluated by objective raters. Implications for clinical practice and future research on racial/ethnic concordance studies linked to the length of treatment are discussed. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
Article
Full-text available
We used the truth and bias model to examine changes in tracking accuracy and under/overestimation (directional bias) on therapists' judgments about clients' satisfaction. We examined 3 factors of clinical experience that could moderate accuracy: (a) overall level of acquaintanceship with a client, operation-alized as treatment length (i.e., less or more time seeing a client), (b) time point in therapy with a specific client, operationalized as session number (i.e., earlier or later in treatment with a client), and (c) order (1st client seen, 2nd client seen. .. last client seen across two years of training in a psychology clinic) in which clients were seen. We conducted a three-level hierarchical linear modeling using data on 6054 sessions, nested in 284 adult clients, nested in 41 doctoral student therapists providing open-ended psychodynamic individual psychotherapy. We found that therapists were able to accurately track client-rated session evaluations with less underestimation (i.e., lower tendency to estimate that clients were less satisfied than they actually were) as they gained experience (both treatment length and client order). Furthermore, therapists exhibited greater tracking accuracy gains over the span of shorter treatments and when working with clients earlier in their clinical training. In longer treatments and with clients seen later in training, tracking accuracy was stable and consistent. Implications for research and practice are discussed. Public Significance Statement We found that therapists improve in their accuracy in assessing clients' satisfaction with clinical experience. Therapist became more accurate as they conducted more sessions with individual clients but did not gain as much accuracy with respect to seeing subsequent clients. Therapists improved significantly in their ability to accurately track client satisfaction in shorter treatments while accuracy remained consistently high and stable in longer treatments.
Article
Full-text available
In addition to assisting with the development of essential competencies for professional practice, growing evidence indicates that supervision may also significantly contribute to client outcomes. However, to date, there have been no explorations of specific supervisor variables that may help us to understand the supervision-client outcome relationship. The current study sought to address this gap in the emerging literature by drawing archival data associated with discharged clients (N = 310) from a large training clinic. Given the available data, we were able to explore 2 supervisor variables in terms of their relationships with client outcomes: 1 categorical variable (faculty status: adjunct vs. tenured/tenure track) and 1 continuous variable (time elapsed since the supervisor attained his or her doctoral degree). The results replicate earlier findings demonstrating that supervisors contribute significantly to client outcomes. Exploration of specific supervisor variables indicates that faculty status is not meaningfully associated with client outcomes. However, the length of time that has elapsed since the supervisor attained his or her doctoral degree is salient, with more recent graduates associated with better client outcomes. Additional research is strongly encouraged in this understudied area, with particular attention to examining the role of supervisor training and supervisory competence.
Article
Full-text available
This document outlines guidelines for supervision of students in health service psychology education and training programs. The goal was to capture optimal performance expectations for psychologists who supervise. It is based on the premises that supervisors (a) strive to achieve competence in the provision of supervision and (b) employ a competency-based, meta-theoretical approach to the supervision process. The Guidelines on Supervision were developed as a resource to inform education and training regarding the implementation of competency-based supervision. The Guidelines on Supervision build on the robust literatures on competency-based education and clinical supervision. They are organized around seven domains: supervisor competence; diversity; relationships; professionalism; assessment/evaluation/feedback; problems of professional competence, and ethical, legal, and regulatory considerations. The Guidelines on Supervision represent the collective effort of a task force convened by the American Psychological Association (APA) Board of Educational Affairs (BEA).
Article
The use of client feedback in clinical supervision provides a way for supervisors to access clients' experiences of the treatment process and monitor clinical progress of their trainees' cases. The present qualitative study investigated a marriage and family therapy training programme's early experience of introducing the Systemic Therapy Inventory of Change (STIC®; Pinsof et al., 2009) into clinical supervision. Supervisors (N = 8) and trainees (N = 14) were interviewed to elicit their experience using the STIC in supervision with a focus on understanding the frequency of use as well as the facilitators and constraints to implementation. The analysis of the narratives resulted in the development of five themes (time, supervisors' expectations, broader training system influences, client feedback training, and perceived helpfulness) that contributed to decreased usage over time. Recommendations to the field for integrating client feedback into empirically informed training and supervision are provided. Practitioner points • Training programmes should consider the potential benefits of utilizing client feedback measures for outcome‐based training and supervision • Training programmes adopting a client feedback measure need to be aware of the potential challenges of implementation • The additional time needed to use a client feedback measure in the clinical supervision process must be considered as it is the most notable constraint to implementation
Article
The authors explored the connection between the facilitative conditions present within the supervisory relationship, the therapeutic relationship, and client outcomes. A correlational research design was used with a sample of 55 counselors-in-training and 88 clients. Results indicated a significant positive relationship between the therapeutic relationship and client outcomes and a significant negative relationship between the supervisory relationship and client outcomes. © 2016 by the American Counseling Association. All rights reserved.
Article
Objective: The present study had 2 aims: (a) to implement West and Kenny's (2011) Truth-and-Bias model to simultaneously assess the temporal congruence and directional discrepancy between clients' and therapists' ratings of the bond facet of the therapeutic alliance, as they cofluctuate from session to session; and (b) to examine whether symptom severity and a personality disorder (PD) diagnosis moderate congruence and/or discrepancy. Method: Participants included 213 clients treated by 49 therapists. At pretreatment, clients were assessed for a PD diagnosis and completed symptom measures. Symptom severity was also assessed at the beginning of each session, using client self-reports. Both clients and therapists rated the therapeutic bond at the end of each session. Results: Therapists and clients exhibited substantial temporal congruence in their session-by-session bond ratings, but therapists' ratings tended to be lower than their clients' across sessions. Additionally, therapeutic dyads whose session-by-session ratings were more congruent also tended to have a larger directional discrepancy (clients' ratings being higher). Pretreatment symptom severity and PD diagnosis did not moderate either temporal congruence or discrepancy at the dyad level; however, during sessions when clients were more symptomatic, therapist and client ratings were both farther apart and tracked each other less closely. Conclusions: Our findings are consistent with a "better safe than sorry" pattern, which suggests that therapists are motivated to take a vigilant approach that may lead both to underestimation and to attunement to fluctuations in the therapeutic bond. (PsycINFO Database Record