Content uploaded by Timothy B Smith
Author content
All content in this area was uploaded by Timothy B Smith on May 13, 2019
Content may be subject to copyright.
Effective Treatment
Nyman, S. J., Nafziger, M. A., & Smith, T. B. (2010). Client outcomes across counselor
training level within a multi-tiered supervision model. Journal of Counseling and
Development, 88, 204-209.
Client Outcomes Across Counselor Training Level
Within a Multi-Tiered Supervision Model
Scott J. Nyman
Genesys Regional Medical Center, Grand Blanc, Michigan
Mark A. Nafziger
Counseling Center, Utah State University
Timothy B. Smith
Department of Counseling Psychology, Brigham Young University
1
Effective Treatment
Abstract
Student trainees in graduate counseling programs provide mental health services to the
public, but trainees may not provide the same quality of services as professional staff.
Three years of outcome data were examined to evaluate treatment effectiveness at a
public university counseling center that utilized a multi-tiered supervision model
consisting of professional staff, pre-doctoral interns, and practicum students. Clients (N =
264) completed self-report inventories of psychological distress at intake and again after
the sixth session. Clients demonstrated significant improvement on all dependent
measures, with no significant differences between those seen by professional staff vs.
supervised trainees. Results appear to provide preliminary support for the clinical
effectiveness of a multi-tiered supervision model. Limitations and implications for future
inquiry are discussed.
Key Words: Counselor training, supervision, effectiveness, and outcome evaluation.
2
Effective Treatment
Client Outcomes across Counselor Training Level
Within a Multi-Tiered Supervision Model
A symbiotic relationship has long existed between mental health service providers
and graduate student trainees seeking real-world counseling experience. Training sites
offer graduate students invaluable opportunities to refine counseling skills, gain
professional experiences, and receive expert supervision (Bernard & Goodyear, 1998;
Boggs & Douce, 2000; Krasner, Howard, & Brown, 1998). In return, these training sites
receive free or low-cost labor from the graduate student trainees (Holland, 1998). In the
current climate of increasing financial pressures and clinical demands placed on service
providers (Constantine & Gloria, 1998), the use of graduate student trainees has become
so commonplace that few scholars question the practice. In this paper, we evaluate the
assumption that counseling services provided by student trainees under close supervision
result in equivalent client outcomes compared to services provided by licensed
professionals.
A recent review of the counseling outcome literature concluded that “there have
been surprisingly few research studies in the area of experience, training, and client
outcome” (Lambert, 2005, p. 861). Studies that have endeavored to evaluate client
outcome across counselor training level have yielded mixed results (Stein & Lambert,
1995). While several reports suggest that counselor experience is not crucial to client
outcome (Michael, Huelsman, & Crowley, 2005; Propst, Paris, & Rosberger, 1994), other
reports favor experienced counselors (Callahan & Hynan, 2005). Research has shown
that experienced counselors conceptualize clients with greater depth and complexity
(Mayfield, Kardash, & Kivlighan, 1999), possess more highly developed cognitive,
emotional, and relational characteristics (Jennings & Skovholt, 1999), and experience
3
Effective Treatment
fewer client drop-outs than do novices (Stein & Lambert, 1995). In particular, novice
counselors may display heightened anxiety, which can be detrimental to counseling
interventions and the counselor-client relationship (e.g., Kelly, Hall, & Miller, 1989).
Under some circumstances, counselor experience level (Roth, 2003) and degree of trainee
experience (Driscoll et al., 2003) can be significant predictors of client improvement.
However, the overall magnitude of the differences between counselors and trainees in
terms of client outcomes has not been established in the literature (Lambert, 2005).
Counselor training sites are responsible for the quality of mental health services
that they provide to their clients. To compensate for the potential limitations in the
effectiveness of clinical services provided by trainees, training sites implement several
resource-intensive procedures. Most notably, training sites provide trainees with direct
clinical supervision, which often includes both individual and group meetings with
licensed professional staff. Typically, the training site will also develop and maintain
supervision guidelines with accompanying oversight to ensure adequate models and
contingencies to address challenges, such as trainee impairment and remediation (Gizara
& Forrest, 2004; Forrest, Elman, Gizara, & Vacha-Haase, 1999). Ideally, training sites
will also regularly evaluate client outcomes to document that the supervised counseling
services provided by trainees are effective. Occasionally, training sites must pursue
additional funding due to the challenge of obtaining third-party reimbursement for
unlicensed counselors (Constantine & Gloria, 1998). Finally, training sites must maintain
close working relationships with the trainees’ university academic programs. All of these
activities require extensive administrative efforts. Therefore, an inevitable consequence
of involving trainees in clinical service provision is the shifting of experienced
4
Effective Treatment
professional staff to greater administrative and supervisory roles. The most experienced
counselors may end up seeing fewer clients.
To overcome the disadvantage of reducing direct client service hours among
experienced counselors, several training sites have implemented a multi-tiered
supervision model, consisting of personnel with varying degrees of experience: licensed
doctoral level professional staff, postdoctoral fellows, pre-doctoral interns, and practicum
students. Supervision is hierarchical, such that licensed staff members supervise fellows
and interns, who subsequently supervise less experienced practicum students. Interns and
fellows receive additional supervision from the licensed staff for their supervisory work
with the practicum students. With post-doctoral fellows and/or pre-doctoral interns
providing supervision to practicum students, professional staff members can retain more
hours for direct client services than in traditional arrangements where professional staff
provide direct supervision to all trainees. Multi-tiered supervision therefore appears to
offer training sites a cost-effective avenue for providing mental health services while at
the same time providing supervised experience in clinical supervision for advanced
trainees. However, empirical analyses are needed to evaluate the effectiveness of
counseling services provided by trainees under this multi-tiered supervision model.
The purpose of the present study is to extend the counseling training and
supervision literature through the evaluation of outcome data from a training site that
employed a multi-tiered supervision model. Specifically, the present study reports client
outcomes across counselor level of training, with beginning practicum students
supervised by pre-doctoral interns and professional staff and with pre-doctoral interns
supervised by licensed professional staff. We tested the hypothesis that clients seeking
5
Effective Treatment
services from a multi-tiered training center will demonstrate significant improvement in
psychological functioning regardless of counselor training level.
Method
Participants. Participant data were culled from a mid-size public university
counseling center in the Western United States. Intake data were obtained from 444
clients, with 264 (59%) clients completing at least 6 counseling sessions, which we set as
the minimum number of sessions required for inclusion in outcome analyses. These
participants were an average of 23.9 years old at intake, with 67% being female.
Participants were 20% freshmen, 17% sophomores, 26% juniors, 23% seniors, and 14%
graduate students. The sample consisted of 91% European Americans, 3% International
Students, 3% Hispanic/Latino Americans, 2% Asian Americans, 1% African Americans,
and 1% Native Americans. The demographics of participants who completed less than 6
sessions did not differ (p > .05) from those who completed 6 or more sessions
Materials. We selected two instruments that were empirically supported for use
with college students, the College Adjustment Scales (CAS; Anton & Reed, 1991) and
the Outcome Questionnaire (OQ-45; Lambert, Hansen, et al., 1996). The CAS is a 108
item self-report inventory composed of nine scales: Anxiety, Depression, Suicidal
Ideation, Substance Abuse, Self-Esteem, Interpersonal Problems, Family Problems,
Academic Problems, and Career Problems. Initial data from 1,146 U.S. university
students demonstrated subscale internal consistency ranging from .80 to .92 (Anton &
Reed, 1991). A subsequent study reported evidence of convergent and discriminant
validity (Wiswell, 1995), and another study found that the CAS significantly
differentiated between clinical and non-clinical samples (Nafziger, Couillard, Smith, &
Wiswell, 1998). In addition, counselors generally supported the CAS for confirming
6
Effective Treatment
client presentation at intake (Nafziger, Couillard, & Smith, 1997). Because the counseling
center evaluated in the present study provided individual mental health services, rather
than academic advisement, career counseling, family therapy, or substance abuse
treatment, data on the CAS Family Problems, Substance Abuse, Academic Problems, and
Career Problems subscales were not relevant outcomes and were therefore omitted from
analyses.
The OQ-45 is a brief 45-item self-report inventory of current psychological
functioning designed to be cost effective, sensitive to change over time, and applicable to
varying degrees of psychological distress (Lambert, Burlingame, et al., 1996). Although
initially designed to measure factors of symptomatic distress, interpersonal problems, and
social role dysfunction, subsequent research appears to support a single factor construct
using a global score (Mueller, Lambert, & Burlingame, 1998). Psychometric properties
for the OQ-45 appear acceptable with respect to 3-week test-retest reliability (.84),
internal consistency (.93), and concurrent validity (Lambert, Burlingame, et al.). In
addition, the OQ-45 has demonstrated sensitivity to client improvement within university
counseling center settings (Vermeersch et al., 2004).
Procedure. Prior to an initial intake session, students seeking services from the
university counseling center completed the CAS and OQ-45. Following intake, the OQ-
45 was administered every 3rd session, while the CAS was administered every 6th
session. During the 3-year data collection period the counseling center was staffed with 5
doctoral-level licensed mental health professionals, 9 pre-doctoral interns (3 per year),
and 18 practicum students (6 per year). A multi-tiered supervision model was utilized in
which professional staff supervised pre-doctoral interns and first-semester practicum
students. Pre-doctoral interns supervised second-semester practicum students and
7
Effective Treatment
received additional supervision specific to that work from licensed professionals. The
counseling center operated an internship program fully accredited by the American
Psychological Association and abided by all accreditation stipulations. All trainees
videotaped and reviewed counseling sessions with supervisors in weekly individual and
group supervision sessions.
Results
As a preliminary step, it was important to verify whether clients assigned to
practicum students, pre-doctoral interns, and licensed professional staff differed in terms
of their initial symptom severity. We therefore conducted a multivariate analysis of
variance (MANOVA) using intake scores on all dependent measures (the OQ-45 and all
CAS subscales) across experience levels. The results were not statistically significant,
Wilks’ Lambda = .97, F(12, 870) = 1.2, p = .27, indicating that clients had a similar level
of self-reported symptoms at intake across the three groups of counselors.
To determine if there were differences in the magnitude of client improvement as
a function of counselor training level, we conducted a repeated measures MANOVA
using scores on the OQ-45 and CAS subscales at both intake and follow-up. The within-
subjects effects of time (intake to follow-up) reached statistical significance for the
multivariate test, Wilks’ Lambda = .74, F(6, 167) = 9.7, p < .0001, and for all univariate
tests across the OQ and CAS subscales (see Table 1). However, neither the between-
subjects effects for the type of therapist (practicum students, pre-doctoral interns, and
licensed professional staff) nor the interaction effect between the type of therapist and
time reached statistical significance, Wilks’ lambda = .92, F(12, 334) = 1.23, p = .26, and
Wilks’ lambda = .92, F(12, 334) = 1.19, p = .28, respectively (see Table 2). Even though
clients reported significant decreases in symptoms across a wide variety of clinical
8
Effective Treatment
assessments (Table 1), there were no differences in client outcomes across counselor
training level using mean score data (Table 2).
Because previous authors (e.g., Lambert & Bergin, 1994) have noted that analyses
of mean score differences do not reveal trends within the data, we next conducted more
specific analyses to disaggregate the type of changes in symptom severity experienced by
individual clients. Using guidelines established by Lambert, Burlingame and colleagues
(1996) on the OQ-45, we classified clients into one of four groups based on scores from
the initial and sixth session OQ-45, which has a clinical cut-off of 63.4 and a reliable
change index of 14. Recovered clients (20%) had initial scores of 64 or higher that
decreased by at least 14 points to a score of 63 or less. Improved clients (21%) had intake
scores that subsequently decreased by 14 points or more but were not yet less than the
clinical cut-off score. Deteriorated clients (12%) increased in their scores by at least 14
points. Unchanged clients (47%) were those who did not meet any of the above criteria.
We then performed a Chi-Square analysis across the categories of client outcome by
counselor training level. The results were not statistically significant, 2 (6, N = 264) =
4.3, p = .64, confirming the findings from the analyses of mean score data.
A final set of analyses investigated client discontinuation of counseling as a
function of counselor training level. Within the dataset, counselors had indicated whether
the client had discontinued counseling without making progress toward counseling goals
or without informing the counselor (i.e., “premature terminations”). It was possible that
clients who had discontinued counseling did so for reasons related to dissatisfaction with
the counselor. We therefore conducted a Chi-Square analysis with these data across
counselor training level. The results did not reach statistical significance, 2 (2, N = 444)
= 2.1, p = .36. Because it was possible that counselors did not reliably record whether a
9
Effective Treatment
client discontinued counseling prematurely, we also evaluated differences across clients
who had attended less than six sessions vs. more than six sessions. These results also did
not reach statistical significance, 2 (2, N = 444) = 4.6, p = .10. There was no evidence
that clients discontinued counseling at higher rates if they were seen by a less
experienced counselor.
Discussion
Clients in this study displayed improvements in psychological functioning that
were independent of the training level of the counselor. Based on the data from the CAS
and the OQ-45, clients who obtained services from a multi-tiered supervised training
center experienced moderate symptom relief over six sessions regardless of whether they
were seen by a licensed doctoral-level counselor, a pre-doctoral intern, or a practicum
student. This finding appears to contradict some previous research that suggests trends
favoring experienced counselors (e.g., Callahan & Hynan, 2005; Driscoll et al., 2003;
Roth, 2003). Although this finding also appears to contradict the basic assumption that
experienced professionals should be more adept at facilitating client improvement than
supervised trainees, there is increasing evidence that this assumption is unfounded
(Lambert, 2005). It is important to note that the counselors in this study were all closely
supervised and might demonstrate greater counseling effectiveness than would newly
licensed counselors with similar experience who no longer receive expert supervision.
Nevertheless, the results of the present study should be interpreted cautiously
when considering the inherent limitations of non-experimental research designs. We
evaluated clinical effectiveness rather than efficacy, which requires greater experimental
rigor (Seligman, 1995). For instance, a control group was not included nor did we involve
concurrent treatment groups from multiple training centers. Studies comparing counselor
10
Effective Treatment
effectiveness across experience levels may also be limited by the within-group variability
of the highest training tier (Stein & Lambert, 1995). Further, the data collection from a
university counseling center located in the rural Western United States restricts the
likelihood that the results may be generalized to other clinical settings, geographical
regions, and client populations.
This study was also limited in that it only evaluated client outcome and
termination data. Although we evaluated multiple outcome measures and although client
outcomes and termination rates are ostensibly the most clinically salient dependent
variables, they reveal only certain aspects of clients’ experiences in therapy. It is likely
client expectations, client trust, and a host of other variables relevant to the client-
counselor relationship moderate and/or mediate client outcomes. For example, it is
possible that clients, who recognize a counselor as a novice, may compensate by altering
their own expectations for improvement (e.g., expecting less assistance from a novice
counselor and therefore being satisfied when they obtain a minimal gain). Similarly, if
clients recognize a counselor as being inexperienced, they may decrease their trust in the
counselor’s ability to handle difficult issues and therefore restrict their discussion to
superficial topics that involve minimal psychological risk, such that the clients obtain
short-term symptom reduction rather than long-term remission. Moreover, client progress
was only tracked over six sessions of counseling, so differences in client outcome may
only become significant over longer courses of treatment. Given these several
possibilities, future research is needed to not only confirm the findings of this study
regarding client outcomes but to also evaluate other pertinent variables regarding client
expectations and the client-counselor relationship.
11
Effective Treatment
Furthermore, the present study did not evaluate the cost-effectiveness of multi-
tiered supervision. This study also did not address the question of whether multi-tiered
supervision is superior to direct supervision by licensed staff only. It remains for future
research to investigate trainee skill acquisition within and across specific training models
(Stein & Lambert, 1995). Useful variables to consider would include trainees’ and
supervisors’ satisfaction ratings and the observed number and nature of apparent ethical
violations. Any data supporting the cost-effectiveness of multi-tiered supervision could be
completely undermined if other data conclusively demonstrate that ethical violations of
greater severity occur when trainees are supervised by advanced trainees as compared to
when trainees are supervised by licensed professionals.
Despite its limitations, the present study provides preliminary support for a multi-
tiered supervision model and may serve as a catalyst for future evaluative studies across
other supervision models. The results also provide notable findings regarding the nature
of clinical effectiveness within a training environment. Counseling training centers
provide invaluable clinical and professional growth opportunities for trainees as well as
licensed supervisors. In many ways, supervised training sites provide foundational
experiences for the clinical practice of future professionals. Given the magnitude of this
influence–and given the commonplace use of trainees in providing clinical services to
clients experiencing genuine distress–the dearth of research investigating client outcomes
across counselor training levels (Lambert, 2005) seems a glaring omission in the
counseling literature. It may be that researchers are loathe to face the possibility that the
extensive efforts involved in educating graduate students to become licensed
professionals results in no observable differences in client outcome. However, in the
interest of empirical inquiry, we urge the field to squarely face the possibility that
12
Effective Treatment
supervised novice counselors may be as effective as experienced counselors and to work
more systematically in uncovering the many moderating and mediating variables
associated with client outcome and client-counselor relationships.
13
Effective Treatment
References
Anton, W. D., & Reed, J. R. (1991). College adjustment scales: Professional manual.
Odessa, FL: Psychological Assessment Resources, Inc.
Bernard, J. M., & Goodyear, P. K., (1998). Fundamentals of clinical supervision (2nd
ed.). Boston: Allyn & Bacon.
Boggs, K. R., & Douce, L. A. (2000). Current status and anticipated changes in
psychology internships: Effects on counseling psychology training. The
Counseling Psychologist, 28, 672-686.
Callahan, J. L., & Hynan, M. T. (2005). Models of Psychotherapy Outcome: Are they
applicable in training clinics? Psychological Services, 2, 65-69.
Constantine, M. G., & Gloria, A. M. (1998). The impact of managed health care on
pre-doctoral internship sites: A national survey. Professional Psychology:
Research and Practice, 29, 195-199.
Driscoll, K. A., Cukrowicz, K. C., Reitzel, L. R., Hernandez, A., Petty, S. C., & Joiner, T.
E. (2003). Behavior Therapy, 34, 165-177.
Forrest, L., Elman, N., Gizara, S., & Vacha-Haase, T. (1999). Trainee impairment: A
review of identification, remediation, dismissal, and legal issues. Counseling
Psychologist, 27, 627-686.
Gizara, S. S., Forrest, L. (2004). Supervisors’ experiences of trainee impairment and
incompetence at APA-accredited internship sites. Professional Psychology:
Research and Practice, 35, 131-140.
Holland, D. (1998). The cost-effective delivery of rehabilitation psychology services: The
responsible utilization of paraprofessionals. Rehabilitation Psychology, 43, 232-
245.
14
Effective Treatment
Jennings, L., & Skovholt, T. M. (1999). The cognitive, emotional, and relational
characteristics of master counselors. Journal of Counseling Psychology, 46, 3-11.
Krasner, R. F., Howard, K. I., & Brown, A. S. (1998). The acquisition of
psychotherapeutic skill: An empirical study. Journal of Clinical Psychology, 54,
895-903.
Kelly, K. R., Hall, A. S., & Miller, K. L. (1989). Relation of counselor intention and
anxiety to brief counseling outcome. Journal of Counseling Psychology, 36, 158-
162.
Lambert, M. J. (2005). Early response in psychotherapy: Further evidence for the
common factors rather than “placebo effects.” Journal of Clinical Psychology, 61,
855-869.
Lambert, M. J., & Bergin, A. E. (1994). The effectiveness of psychotherapy. In A. E.
Bergin and S. L. Garfield (Eds.) Handbook of psychotherapy and behavior
change (pp. 143-189). New York: Wiley.
Lambert, M. J., Burlingame, G. M., Umphress, V., Hansen, N. B., Vermeersch, D. A.,
Clouse, G. C., & Yanchar, S. C. (1996). The reliability and validity of the
Outcome Questionnaire. Clinical Psychology and Psychotherapy, 3, 249-258.
Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G.
M. (1996). Administration and scoring manual for the OQ-45.2. Stevenson, MD:
American Professional Credentialing Services, LLC.
Mayfield, W. A., Kardash, C. M., Kivlinghan, D. M., Jr. (1999). Differences in
experienced and novice counselor’s knowledge models about clients:
Implications for case conceptualization. Journal of Counseling Psychology, 46,
504-514.
15
Effective Treatment
Michael, K.D., Huelsman, T. J., & Crowley, S. L. (2005). Interventions for child and
adolescent depression: Do professional counselors produce better results?
Journal of Child and Family Studies, 14, 223-236.
Mueller, R. M., Lambert, M. J., Burlingame, G. M. (1998). Construct validity of the
Outcome Questionnaire: A Confirmatory factor analysis. Journal of Personality
Assessment, 70, 248-262.
Nafziger, M. A., Couillard, G. C., Smith, T. B., & Wiswell, D. K. (1998). Normative
data on the College Adjustment Scales from a university counseling center.
Journal of College Student Development, 39, 283-289.
Nafziger, M. A., Couillard, G. C., & Smith, T. B. (1997). An evaluation of the clinical
utility of the College Adjustment Scales. College Student Journal, 31, 516-518.
Propst, A. P., Paris, J., & Rosberger, Z. (1994). Do counselor experience, diagnosis, and
functional level predict outcome in short term psychotherapy? Canadian Journal
of Psychiatry, 39, 168- 176.
Roth, S. A., (2003). A program evaluation measuring the effectiveness of service delivery
at a university counseling center. Dissertation Abstracts International Section A:
Humanities and Social Sciences, 63 (12-A), pp. 4235.
Seligman, M. E., (1995). The effectiveness of psychotherapy: The Consumer Reports
Study. American Psychologist, 50, 965-974.
Stein, D. M., & Lambert, M. J. (1995). Graduate training in psychotherapy: Are therapy
outcomes enhanced? Journal of Consulting and Clinical Psychology, 63, 182-196.
Vermeersch, D. A., Whipple, J. L., Lambert, M. J., Hawkins, E. J., Buchfield, C. M., &
Okiishi, J. (2004). Outcome Questionnaire: Is it sensitive to changes in counseling
center clients? Journal of Counseling Psychology, 51, 38-49.
16
Effective Treatment
Wiswell, D. K. (1995). Establishing the validity of the college adjustment scales (CAS)
as
outcome measures in a university counseling center: A test of construct and
convergent validity. Unpublished doctoral dissertation, Utah State University,
Logan.
17
Effective Treatment
Table 1
Client OQ-45 and CAS Data at Intake and Follow-Up
Measure M (SD) M (SD) F* d**
Intake Session 6
OQ-45 78.4 (23.6) 69.4 (21.2) 39.8 .40
CAS Anxiety 60.9 (9.4) 56.5 (10.0) 43.8 .45
CAS Depression 62.7 (10.5) 57.8 (10.0) 53.2 .48
CAS Suicidal Ideation 57.0 (10.5) 53.5 (10.0) 12.4 .34
CAS Self-Esteem 62.1 (9.0) 59.5 (9.0) 20.7 .29
CAS Interpersonal 57.3 (9.4) 54.9 (9.9) 16.3 .25
Note: * = F-value from repeated measures analyses of variance. For all analyses p < .001.
** = standardized mean difference from intake to follow-up.
18
Effective Treatment
Table 2
Means (and Standard Deviations in Parentheses) of Client OQ-45 and CAS Data at Intake and
Follow-Up Across Counselor Training Level
Practicum Pre-doctoral Professional
Student Intern Staff
Measure (n = 47) (n = 95) (n = 122) F*
OQ-45 0.1
Pre 75.0 (23.4) 80.1 (21.7) 81.0 (25.5)
Post 67.9 (23.2) 70.1 (19.5) 69.5 (22.0)
CAS Anxiety 0.3
Pre 62.1 (7.7) 60.1 (9.4) 61.7 (9.4)
Post 57.0 (7.9) 56.2 (8.6) 56.3 (11.2)
CAS Depression 0.5
Pre 62.5 (8.4) 63.5 (9.9) 63.4 (10.5)
Post 58.8 (8.9) 58.6 (8.7) 56.9 (10.6)
CAS Suicidal Ideation 1.4
Pre 55.9 (10.3) 58.2 (10.6) 57.6 (11.3)
Post 52.2 (8.5) 55.2 (9.8) 52.6 (10.2)
CAS Self-Esteem 0.3
Pre 61.3 (8.7) 62.8 (9.0) 62.9 (9.8)
Post 59.0 (8.6) 60.2 (7.4) 59.5 (9.6)
CAS Interpersonal 0.2
Pre 57.0 (8.4) 57.8 (8.7) 57.3 (10.7)
Post 55.0 (8.8) 55.3 (10.3) 54.4 (9.9)
Note: * = F-value from repeated measures analyses of variance across counselor type.
19
Effective Treatment
For all analyses p > .10.
20
