Content uploaded by Mark Blais
Author content
All content in this area was uploaded by Mark Blais on Sep 14, 2017
Content may be subject to copyright.
MEASURING PSYCHODYNAMIC–INTERPERSONAL AND
COGNITIVE–BEHAVIORAL TECHNIQUES: DEVELOPMENT
OF THE COMPARATIVE PSYCHOTHERAPY
PROCESS SCALE
MARK J. HILSENROTH
Adelphi University
MATTHEW D. BLAGYS AND
STEVEN J. ACKERMAN
Erik H. Erikson Institute of the Austen Riggs
Center and Harvard Medical School
DENNIS R. BONGE
University of Arkansas
MARK A. BLAIS
Massachusetts General Hospital and
Harvard Medical School
Many instruments have been developed
to assess techniques and interventions
in a variety of psychotherapies. How-
ever, existing scales are limited by sev-
eral factors such as relatively weak
psychometric properties, applicability
to only a single form of treatment or
manual, and extensive time required for
completion. The authors report on the
development of a new measure, the
Comparative Psychotherapy Process
Scale (CPPS). The CPPS is designed to
assess the distinctive features of
psychodynamic–interpersonal and
cognitive– behavioral treatments. Data
are presented on the psychometric
properties, reliability, and validity of
the CPPS. The findings suggest that the
scale possesses excellent interrater reli-
ability and internal consistency as well
as promising validity. Clinical utility,
potential limitations, and future re-
search of the CPPS are discussed.
Keywords: psychotherapy process, ther-
apist technique, adherence,
psychodynamic–interpersonal,
cognitive– behavioral
A number of instruments have been developed
to assess treatment adherence, technique, and
process for a variety of therapies (Barber & Crits-
Christoph, 1996; Barber, Liese & Abrams, 2003;
DeRubeis, Hollon, Evans, & Bemis, 1982; Gas-
ton & Ring, 1992; Goldfried, Newman, & Hayes,
1989; Hollon et al., 1988; Jones, 1985; Ogrod-
niczuk & Piper, 1999; O’Malley et al., 1988;
Samoilov, Goldfried, & Shapiro, 2000; Shapiro
& Startup, 1990; Shaw et al., 1999; Young &
Beck, 1980). However, some of these scales are
limited by various factors. First, some existing
measures suffer from relatively low interrater re-
liability coefficients (intraclass correlation coef-
ficients in the poor–fair range, ⬍.60; Fleiss,
1981) for specific scales or items (Barber &
Crits-Christoph, 1996; Barber et al., 2003; Crits-
Christoph et al., 1998; Gaston & Ring, 1992;
Goldfried, Castonguay, Hayes, Drozd, & Sha-
piro, 1997; Goldfried, Raue, & Castonguay,
Mark J. Hilsenroth, Derner Institute of Advanced Psycho-
logical Studies, Adelphi University; Matthew D. Blagys and
Steven J. Ackerman, Erik H. Erikson Institute of the Austen
Riggs Center and Harvard Medical School; Dennis R. Bonge,
Department of Psychology, University of Arkansas; Mark A.
Blais, Department of Psychiatry, Massachusetts General Hos-
pital and Harvard Medical School.
Additional materials are on the Web at http://dx.doi.org/
10.1037/0033-3204.42.3.340.supp
Correspondence regarding this article should be addressed
to Mark J. Hilsenroth, PhD, ABAP, Derner Institute of Ad-
vanced Psychological Studies, Adelphi University, 220 Wein-
berg Building, 158 Cambridge Avenue, Garden City, NY
11530. E-mail: hilsenro@adelphi.edu
Psychotherapy: Theory, Research, Practice, Training Copyright 2005 by the Educational Publishing Foundation
2005, Vol. 42, No. 3, 340–356 0033-3204/05/$12.00 DOI: 10.1037/0033-3204.42.3.340
340
1998; Hollon et al., 1988; Jacobson, 1998; Jacob-
son et al., 1996; Ogrodniczuk & Piper, 1999;
Shaw et al., 1999; Vallis, Shaw, & Dobson,
1986). Second, measures of adherence are, by
definition, wedded to a specific manual or treat-
ment (Waltz, Addis, Koerner, & Jacobson, 1993).
Therefore, the applicability and utility of these
instruments to more real-world therapies that (a)
do not use a treatment manual, (b) do not use the
specific manual for which the instrument was
developed, or (c) use techniques from different
manualized treatments (both within or across the-
oretical orientations) are largely unknown. Third,
some existing instruments include a relatively
large number of items that require a substantial
amount of time to be completed and may limit
their applied utility (Barber & Crits-Christoph,
1996; Goldfried et al., 1989; Hollon et al., 1988;
Jones, 1985; Shapiro & Startup, 1990).
The current study is a psychometric investigation
of a new measure, the Comparative Psychotherapy
Process Scale (CPPS). The scale is based on the
findings of two reviews of the empirical compara-
tive psychotherapy process literature (Blagys &
Hilsenroth, 2000, 2002). These reviews sought to
identify significant differences between the tech-
niques used in psychodynamic–interpersonal (PI;
defined broadly to include psychodynamic,
psychodynamic–interpersonal, and interpersonal
therapies) and cognitive– behavioral (CB; defined
broadly to include cognitive, cognitive– behavioral,
and behavioral therapies) treatments. Blagys and
Hilsenroth (2000) identified seven general tech-
niques that consistently and significantly distin-
guished PI from CB treatments: (a) focusing on
patients’ affect and the expression of emotion; (b)
exploring patients’ attempts to avoid topics or en-
gage in activities that hinder the progress of treat-
ment; (c) focusing session on patterns in patients’
actions, thoughts, feelings, and relationships; (d)
exploring patients’ past experiences; (e) focusing on
patients’ interpersonal experiences; (f) focusing dis-
cussion on the therapeutic relationship; and (g) ex-
ploring patients’ wishes, dreams, or fantasies.
Blagys and Hilsenroth (2002) reported six general
techniques that consistently and significantly distin-
guished CB from PI treatments: (a) assigning home-
work and outside of session activities; (b) actively
directing session activity; (c) teaching specific cop-
ing skills; (d) focusing on patients’ future experi-
ences; (e) providing patients with information about
their treatment, disorder, or symptoms; and (f) fo-
cusing on patients’ cognitive experiences (e.g., dys-
functional or irrational beliefs).
CPPS items were written to reflect the
between-treatment differences identified in the
empirical literature reviews in order to assess the
distinctive features of these alternative ap-
proaches to therapy. The scale is intended to be a
brief measure assessing the degree to which any
delivered therapy used global techniques and ad-
hered to the distinctive features of PI and CB
treatments. Thus, the scale has the potential to (a)
classify treatments broadly as either PI or CB, (b)
compare various types of CB and PI therapies,
and (c) examine the effects of PI and CB tech-
niques in a given session or treatment. The CPPS
is not intended to replace existing manual specific
instruments but rather is intended to offer a reli-
able alternative that has general real-world appli-
cability to a variety of treatments (both within
and across theoretical orientations).
The CPPS is distinctive from other existing
measures in several ways. First, unlike adherence
and competence instruments such as the Cogni-
tive Therapy Adherence and Competence Scale
(Liese, Barber, & Beck, 1995), Collaborative
Study Psychotherapy Rating Scale (CSPRS; Hol-
lon et al., 1988), Sheffield Psychotherapy Rating
Scale (SPRS; Shapiro & Startup, 1990), Cogni-
tive Therapy Scale (Young & Beck, 1980), and
the Penn Adherence-Competence Scale for
Supportive–Expressive Psychotherapy (PACS-
SE; Barber & Crits-Christoph, 1996), the CPPS
was not designed to assess the interventions of a
specific treatment and its corresponding manual.
Rather, it was developed to be a more general
instrument with applicability to different forms of
therapy. In this respect, the CPPS may be partic-
ularly useful for examining real-world (i.e., not
manualized) treatments. Second, some instru-
ments assess techniques from only one theoreti-
cal perspective (i.e., either PI or CB). For exam-
ple, the PACS-SE (Barber & Crits-Christoph,
1996) and the Interpretive and Supportive Tech-
nique Scale (Ogrodniczuk & Piper, 1999) were
designed to assess the interventions of psychody-
namic treatments. Therefore, the applicability
and relevance of these measures to CB treatments
are limited. In contrast, the CPPS assesses key
aspects of both PI and CB treatments, allowing
for comparisons between these different forms of
therapy.
A third distinctive feature of the CPPS is its
focus on the distinguishing features of PI and CB
CPPS Development
341
therapies as found in the empirical literature. That
is, this measure was developed from actual re-
search findings (i.e., bottom-up) rather than
solely theoretical expectations (i.e., top-down).
As noted by Waltz et al. (1993), the evaluation of
interventions both prescribed and proscribed by a
treatment is important for capturing the complex-
ity of a therapy session and assessing treatment
adherence. In this respect, the CPPS may be said
to include items described by Waltz et al. (1993)
as “unique and essential” and “proscribed” by PI
and CB approaches, respectively. The CPPS is
the only measure of which we are aware that
attempts to quantify and assess the specific tech-
niques that make PI and CB treatments different.
Therefore, the CPPS may be especially useful in
assessing the amount of unique aspects of PI and
CB treatment included in a session. This is an
important distinction given the results of several
studies that suggest features of an alternative
treatment may be unintentionally provided in a
therapy (e.g., such as psychodynamic techniques
used in a CB treatment session) and contribute to
treatment outcome independent of the intended,
treatment-specific interventions (e.g., Ablon &
Jones, 1998; Castonguay, Goldfried, Wiser,
Raue, & Hayes, 1996; Hayes, Castonguay, &
Goldfried, 1996; Hayes & Strauss, 1998; Jones &
Pulos, 1993). Last, the CPPS was designed to be
a brief and efficient measure. In contrast, several
existing measures contain a large number of
items that are relatively time consuming and la-
bor intensive such as the Psychotherapy Process
Q-Sort (Jones, 1985), CSPRS (Hollon et al.,
1988), PACS-SE (Barber & Crits-Christoph,
1996), SPRS (Shapiro & Startup, 1990), and the
Coding System of Therapeutic Focus (Goldfried
et al., 1989).
Although there have already been several exam-
ples of the clinical validity of the CPPS, across two
separate research groups (Ackerman, Hilsenroth, &
Knowles, 2005; Hilsenroth, Ackerman & Blagys,
2001; Hilsenroth, Ackerman, Blagys, Baity, &
Mooney, 2003; Hilsenroth, DeFife, Blagys, & Ack-
erman, in press; Price, Hilsenroth, Callahan,
Petretic-Jackson, & Bonge, 2004; Thompson-
Brenner & Westen, in press, 2005; Westen, No-
votny, & Thompson-Brenner, 2004), the specific
aims of the current study are to present original data
on the CPPS along two related lines of interest.
First, we present the initial reliability and psycho-
metric properties from the CPPS development. Sec-
ond, we examine six separate validity analyses,
across several different contexts, to address our a
priori predictions that (a) the CPPS PI subscale will
demonstrate a significant relationship with other
therapist activity measures evaluating similar forms
of psychodynamic or supportive– expressive (SE)
psychotherapy; (b) the CPPS items and subscales
will be able to distinguish between different thera-
peutic modalities; (c) the CPPS PI and CPPS CB
scales will delineate prototypes of each respective
approach to psychotherapy process as rated by ex-
pert clinicians; (d) the CPPS will allow naı¨ve (i.e.,
undergraduate) raters to distinguish the session
characteristics of PI and CB treatments; (e) patients
in psychodynamic psychotherapy will report signif-
icantly more PI than CB treatment techniques dur-
ing their sessions; (f) therapists’ and supervisors’
ratings of both PI and CB treatment techniques will
be significantly related to one another.
Method
Participants
Two patient samples were used in the initial
reliability and validity analyses. The first con-
sisted of 36 patients (18 women, 18 men) con-
secutively admitted to a psychodynamic psycho-
therapy treatment team (PPTT; Hilsenroth, 2002)
over a 26-month period at a university-based
community outpatient psychological clinic. Six-
teen individuals were single, 11 were married,
and 9 were divorced. The mean age for this
psychodynamic (PD) treatment group was 31.14
years (SD ⫽11.50 years).
To provide a comparison for this PD sample,
another group of participants was included. This
second sample consisted of 6 patients from the
same outpatient clinic admitted to other (nonpsy-
chodynamic) treatment practicums during the
same 26-month period. The 6 patients comprising
this group were chosen because they each had
videotaped sessions available and were described
by the therapists conducting the treatment and
their supervisors as nonpsychodynamic treat-
ments. All 6 patients in this nonpsychodynamic
(non-PD) treatment sample were single, with a
mean age of 26.05 years (SD ⫽7.77 years).
Patients were not randomly assigned to the PD
and non-PD treatment conditions. Rather, they
were assigned to student clinicians of various
treatment practicums in an ecologically valid
manner based on clinicians’ availability and were
accepted into treatment regardless of disorder or
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
342
comorbidity. In each treatment sample, semi-
structured diagnostic interviews were used to
evaluate symptomatology from the Diagnostic
and Statistical Manual of Mental Disorders (4th
ed.; DSM–IV; American Psychiatric Association,
1994; for a more detailed description of this as-
sessment process, see Hilsenroth, 2002; Hilsen-
roth, Peters, & Ackerman, 2004). The DSM–IV
Axis I diagnoses in the PD therapy patient sample
were as follows: mood disorder (n⫽24), adjust-
ment disorder (n⫽5), V-code relational problem
(n⫽5), anxiety disorder (n⫽1), and substance
abuse (n⫽1). In addition, 17 individuals were
diagnosed with an Axis II personality disorder,
and 8 others were found to have subclinical per-
sonality disorder traits or features. The DSM–IV
Axis I diagnoses in the non-PD therapy sample
were as follows: anxiety disorder (n⫽3), so-
matoform disorder (n⫽2), V-code problems
related to abuse or neglect (n⫽2), substance
abuse disorder (n⫽1), sexual and gender iden-
tity disorders (n⫽1), and V-code relational
problems (n⫽1). In addition, 3 individuals were
diagnosed with an Axis II personality disorder,
and 2 others had subclinical personality disorder
traits or features. All patients receiving services
at the clinic where these data were collected (not
just the participants in this research project) give
their informed consent to be included in program
evaluation research and to have their therapy
sessions videotaped.
Therapists, Training, and Treatment
Therapists in the PPTT were 13 advanced
graduate students (8 women, 5 men) enrolled in
an American Psychological Association (APA)-
accredited clinical psychology doctorate pro-
gram. Training of the PPTT therapists included
(a) a minimum of 1.5 hr of individual clinical
case supervision per week with the PPTT super-
visor (a licensed clinical psychologist), which
included the viewing of session videotape and the
discussion of case conceptualization, interven-
tions, and techniques; (b) 2 hr of small-group
supervision per week with the PPTT supervisor,
which included didactic presentations on the
principles and techniques of SE psychotherapy
(Luborsky, 1984) accompanied by readings from
the treatment manuals of Luborsky (1984), Book
(1998), Wachtel (1993), and Strupp and Binder
(1984) and illustrated by clinical examples and
discussions of videotaped sessions conducted by
participants in and the supervisor of the PPTT;
(c) 1 to 2 hr per week of self-critique in which
therapists were encouraged to review their own
session videotape; (d) additional supervision time
provided by the PPTT supervisor on an as-needed
basis; and (e) an optional 1.5 hr per week con-
tinuing case conference on a training case pro-
vided by the PPTT supervisor (for a more de-
tailed description of this training process, see
Hilsenroth et al., in press).
The treatment provided by the PPTT therapists
was psychodynamic. Although therapists were
aided and informed by the technical guidelines
delineated in the training materials noted previ-
ously, they were not used to prescribe session
activity. As such, the treatment provided was not
manualized. Rather, the training materials were
used to teach psychodynamic technique and in-
terventions from a variety of PD approaches, and
theories were used and integrated into the treat-
ment in a flexible, case-based manner. Treatment
focused on personal insight, support, expression
of emotion, interpretation, and interpersonal–
relational patterns and themes. Each patient re-
ceived one or two sessions of psychotherapy per
week, and treatment was open ended in length.
The length of treatment was determined by the
clinician’s judgment, the patient’s decision to end
therapy, the patient’s progress toward his or her
goals, and changes in the patient’s life.
Therapists in the non-PD treatment group were
5 advanced graduate students (2 women, 3 men)
enrolled in the same APA-accredited clinical psy-
chology doctoral program. Training of the
non-PD therapists was provided by supervisors
who were licensed clinical psychologists and in-
cluded (a) a minimum of 1 hr per week of indi-
vidual supervision with their clinical supervisor,
which included case management, review of ses-
sion videotape, discussion of techniques, inter-
ventions, patient process, and treatment planning;
(b) 2 hr of small-group supervision per week,
which included case review, group discussions,
and viewing of videotaped case material; and (c)
additional individual supervision provided by the
clinical supervisor on an as-needed basis. The
training materials used by the therapists in the
non-PD sample reflect primarily CB approaches
to psychotherapy and include the works of Beck
and Freeman (1990); Beck, Rush, Shaw, and
Emery (1979); Bourne (1995), Davis, Eshelman,
and McKay (1995); Goldfried and Davison
(1994); Craighead, Craighead, Kazdin, and Ma-
CPPS Development
343
honey (1994); and Masters, Burish, Hollon, and
Rimm (1987). Therapists in the non-PD treatment
condition were also encouraged, but not required,
to review their own session videotape.
Therapists in the PPTT were part of an ongo-
ing psychodynamic treatment alliance, process,
and outcome study, which required more super-
vision time and training. Therapists in the
non-PD treatment group, in contrast, were not
involved in a treatment study and generally re-
ceived supervision as usual in the university-
based community outpatient clinic. As such, ther-
apists in this treatment condition may have
received somewhat less supervision than those in
the PPTT. The therapy provided in this group was
nonpsychodynamic. Treatment focused on symp-
tom reduction, directly challenging dysfunc-
tional, illogical, or irrational thoughts–thinking,
actively having patients’ accept responsibility for
their actions– behaviors, teaching specific coping
skills, exposure methods, relaxation techniques,
and homework. For 3 of the 6 non-PD patients,
treatment was of a fixed duration (one 16 and two
12 session treatments) as agreed on by the patient
and therapist. The remaining 3 patients received
treatment that was open ended in length. As in the
PD treatment group, the treatment of the patients
in the non-PD group was not manualized. Rather,
the training materials were used in a flexible
manner based on the needs of the patients.
CPPS
As described, the CPPS (see Appendixes) is
based on the findings of two reviews of the em-
pirical comparative psychotherapy process liter-
ature (Blagys & Hilsenroth, 2000, 2002). The
CPPS is a brief measure of the distinctive fea-
tures of PI and CB treatments designed to assess
therapist activity and techniques used and occur-
ring during the therapeutic hour. The measure
consists of 20 randomly ordered items rated on a
7-point Likert-type scale (0 ⫽not at all charac-
teristic; 2⫽somewhat characteristic; 4⫽char-
acteristic; 6⫽extremely characteristic). The
CPPS may be completed by a patient, therapist,
or external rater. Based on the previously de-
scribed reviews of the empirical literature, the
CPPS was constructed to contain two subscales:
one measuring PI features and one measuring CB
features. The CPPS PI subscale is expected to
measure therapist activities and techniques em-
phasized significantly more in PI than CB ther-
apy. The CPPS CB subscale is expected to mea-
sure techniques and therapist activities that are
emphasized significantly more in CB than PI
treatment.
In completing the CPPS, the rater’s task is to
search for evidence that a therapist activity has
occurred in the session. Although a manual for
scoring criteria
1
provides greater details, a gen-
eral principle across each item is that a score of 1
or 2 suggests some attempt by the therapist to
engage in the behavior or action delineated by the
item, with limited follow-up exploration. A score
of 4 indicates that the technique tapped by the
item is addressed on separate occasions by the
therapist with some follow-up investigation. A
score of 5 or 6 suggests continued efforts by the
therapist to exhibit the behavior or action indi-
cated by the item with sustained follow-up.
Judges and Procedure
Judges were two advanced clinical psychology
graduate students enrolled in an APA-approved
clinical psychology doctoral program. Before the
rating sessions included in the current study, the
two coders underwent supervised training in the
use of the manual and rating video recordings of
both PI and CB sessions using the CPPS. Fifteen
sessions independently rated by the two judges
during the training phase of the project were used
to compute a preliminary analysis of interrater
reliability (intraclass correlation coefficient) on
which judges showed a good level of initial in-
terrater agreement (⬎.60; Fleiss, 1981). The 15
training sessions were obtained from three differ-
ent sources: (a) PI and CB training tapes from the
APA Psychotherapy Videotape Series; (b) other
available training tapes of expert PI and CB ther-
apists; and (c) videotaped sessions of patients
included in the larger programmatic PPTT study
(these sessions were not included in the statistical
analyses of the current study). After the prelimi-
nary interrater reliability analysis, judges began
rating videotaped sessions of patients in the re-
search study.
For the 36 patients receiving PD therapy, vid-
eotapes of the 3rd, 9th, 15th, 21st, 27th, 36th, and
57th sessions were used when available. The total
number of sessions viewed and rated for each
1
The manual is available on the Web at http://dx.doi.org/
10.1037/0033-3204.42.3.340.supp
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
344
patient depended on the length of his or her
treatment. In all, 105 sessions were viewed and
rated in the PD therapy sample. For the 6 patients
in the non-PD treatment group, sessions were
rated based on their availability. In all, 19 ses-
sions (Sessions 1–3, 2–3, 8 –9, 8–12, 9–12, 21,
30, 31) were gathered, viewed, and rated, for a
sample of 124 rated therapy sessions. Of these
124 sessions, 80 were rated by both judges and 44
were rated by one of the two judges. Although we
recognize that the data collection for the non-PD
treatment sample was not as standardized as in
the PD sample, a comparison group was included
to provide a greater range of scores on the CPPS
and to provide a preliminary investigation of the
validity of the CPPS. It is important to note that
this procedure and the size of this non-PD sample
are comparable to those reported in two other
investigations of PD treatment adherence and ac-
tivity (Barber & Crits-Christoph, 1996; Barber,
Crits-Christoph, & Luborsky, 1996).
For each patient, psychotherapy sessions were
arranged in random order and entire sessions
were viewed by the two judges. Immediately
after viewing a videotaped session, judges inde-
pendently completed the CPPS, and raters alter-
nated their completion of the CPPS items (i.e.,
1–20, then 20 –1) and completed the CPPS in a
random order to decrease order effects. Regular
reliability meetings were held during the coding
process to prevent rater drift.
Results
Reliability
On the basis of 80 sessions rated by both
judges, the interrater reliability of the CPPS PI
and CPPS CB was examined using the two-way
random-effect model intraclass correlation coef-
ficient [ICC (2, 1); (Shrout & Fleiss, 1979)] as
well as Spearman-Brown correction for the two-
way random-effect model ICC representing the
mean reliability across two raters [ICC (2, 2);
(Shrout & Fleiss, 1979)]. As shown in Tables 1
and 2, all of the CPPS PI and CPPS CB items
achieved ICC (2, 1) values in the good (.60 –.74;
Fleiss, 1981) to excellent (ⱖ.75; Fleiss, 1981)
range, and all ICC (2, 2) values were in the
excellent (ⱖ.75) range. In addition, all mean
ICCs for the CPPS PI and CPPS CB items were
also in the excellent range, as were the ICCs for
the CPPS PI and CPPS CB total subscale scores.
Psychometric Characteristics
Descriptive statistics. Means and standard
deviations were computed for each of the CPPS
TABLE 1. Interrater Reliability, Descriptive Statistics, and Adjusted Item-to-Scale Correlations for the CPPS PI Subscale
Item
no. Item
ICC PD sample
Non-PD
sample Adjusted
item-to-scale
r(2, 1) (2, 2) MSDMSD
1 Explore uncomfortable feelings .90 .95 4.47 1.23 2.08 1.39 .87
4 Feelings & percepts linked to past exp. .79 .88 3.21 1.45 1.24 0.96 .74
5 Similar relationships over time .84 .91 3.76 1.34 1.32 1.06 .84
7 Focus on patient–therapist relationship .91 .95 2.57 2.08 .66 1.14 .38
8 Experience and expression of feelings .90 .94 4.75 1.10 1.79 1.12 .87
10 Address avoid topics & shift in mood .66 .80 2.12 1.39 .53 0.77 .63
13 Alternative understanding of experiences .75 .85 3.72 1.13 2.13 1.04 .78
14 Recurrent patterns of action/feel/exp. .75 .85 4.02 1.23 2.16 1.11 .81
16 Patient initiates discussion .85 .90 4.38 0.75 1.82 0.98 .56
19 Explore wish, fantasy, dream, EM .80 .89 3.35 1.22 1.13 1.27 .73
Mean CPPS PI .82 .89 3.63 0.90 1.48 0.66 .72
CPPS PI subscale .93 .97 36.33 8.98 14.84 6.55
CPPS PI: coefficient alpha .92
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS PI ⫽Comparative Psychotherapy Process Scale Psychodynamic–Interpersonal subscale; ICC (2, 1) ⫽
intraclass correlation coefficient two-way random-effects model; ICC (2, 2) ⫽intraclass correlation coefficient
Spearman-Brown correction for the two-way random-effects model; PD ⫽psychodynamic. nfor ICC ⫽80. nfor CPPS
PI item means and SD for PD sample ⫽105 sessions. nfor CPPS PI item means and SD for the non-PD sample ⫽19
sessions. Nfor adjusted item-to-scale correlations ⫽124.
CPPS Development
345
items as well as the CPPS PI and CPPS CB
subscale scores in the PD and non-PD samples.
As shown in Table 1, CPPS PI item mean scores
ranged from a low of 2.12 to a high of 4.75 in the
PD sample. In the non-PD sample, CPPS PI item
means ranged from a low of .53 to a high of 2.16.
The mean for the CPPS PI subscale score was
3.63 in the PD sample and 1.48 in the non-PD
sample. Table 2 shows the descriptive statistics
for the CPPS CB items and subscale in the PD
and non-PD sample. The means for the CPPS CB
items ranged from a low of 0.28 to a high of 3.67
in the PD sample. In the non-PD sample, CPPS
CB item means ranged from a low of 3.05 to a
high of 4.37. The mean for the CPPS CB subscale
score was 1.17 in the PD sample and 3.97 in the
non-PD sample.
Internal consistency. Adjusted item-to-scale
correlations provide an estimate of the conver-
gence between the item being evaluated and the
rest of the items in its subscale. In calculating the
adjusted item-to-scale correlations, the item be-
ing evaluated was excluded from the total sub-
scale score so as not to inflate the correlation. An
item is considered to possess adequate conver-
gence if its adjusted item-to-scale correlation is
equal to or greater than .30 (Nunnally & Bern-
stein, 1994). Adjusted item-to-scale correlations
were then computed based on the sample of 124
rated sessions for the CPPS PI and CPPS CB
scales and are shown in Tables 1 and 2. For the
CPPS PI, all of the items achieved adjusted item-
to-scale correlations greater than .30. Similarly,
for the CPPS CB subscale, all of the items ob-
tained adjusted item-to-scale correlations greater
than .30 except for the original version of Item 12
(“The focus of session is primarily on current life
situations”; r⫽.12, p⫽.19).
Coefficient alpha is also an internal reliability
statistic that provides an evaluation of the internal
consistency of the items defining a subscale. A
coefficient alpha of .70 or greater is generally con-
sidered to represent adequate internal consistency
(Nunnally & Bernstein, 1994). In addition to dis-
playing the adjusted item-to-scale correlations
noted previously, coefficient alphas were computed
for the CPPS PI and CPPS CB subscales. As shown
in Tables 1 and 2, the coefficient alpha based on the
sample of 124 rated sessions for the CPPS PI and
CPPS CB were .92 and .94, respectively, indicating
high internal consistency.
Initial Validation
Concurrent validity of the CPPS PI subscale.
To evaluate the concurrent validity of the CPPS
PI subscale, the relationships between this sub-
scale and other therapist activity measures eval-
uating similar forms of psychodynamic or SE
psychotherapy, as found in this study, were ex-
TABLE 2. Interrater Reliability, Descriptive Statistics, and Adjusted Item-to-Scale Correlations for the CPPS CB Subscale
Item
no. Item
ICC PD sample
Non-PD
sample Adjusted
item-to-scale
r(2, 1) (2, 2) MSDMSD
2 Explicit advice or direct suggestion .79 .88 1.06 0.98 3.71 0.73 .79
3 Therapist initiation of topics and activity .82 .89 2.17 0.88 4.37 0.47 .64
6 Focus on irrational/illogical belief system .70 .82 1.41 0.91 4.26 1.02 .75
9 Specific outside-of-session activity or task .89 .94 .37 0.66 4.16 1.40 .89
11 Explain rationale, technique, or treatment .81 .89 .75 0.87 3.05 0.96 .76
12 Focus primarily on current life situations .67 .79 3.67 1.26 4.05 0.85 .12
15 Provide information symp, disorder, or tx .81 .89 .99 0.93 4.18 1.17 .83
17 Practice behaviors between sessions .91 .95 .28 0.58 4.11 1.09 .91
18 Teach specific techniques to patient .95 .97 .29 0.55 3.58 1.77 .83
20 Interacts in teacher-like (didactic) manner .89 .93 .67 0.67 4.24 1.10 .89
Mean CPPS CB .82 .90 1.17 0.43 3.97 0.60 .74
CPPS CB subscale .95 .98 11.66 4.31 39.71 5.98
CPPS CB: coefficient alpha .94
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS CB ⫽Comparative Psychotherapy Process Scale Cognitive–Behavioral subscale; ICC (2, 1) ⫽intraclass
correlation coefficient two-way random-effects model; ICC (2, 2) ⫽intraclass correlation coefficient Spearman-Brown
correction for the two-way random-effects model; PD ⫽psychodynamic. nfor ICC ⫽80. nfor CPPS CB item and
subscale means and SD in PD sample ⫽105 sessions. nfor CPPS CB item and subscale means and SD in the non-PD
sample ⫽19 sessions. Nfor adjusted item-to-scale correlations ⫽124.
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
346
amined using Pearson rcorrelations. Following
the same procedures detailed previously, 76 vid-
eotaped sessions were rated by both judges
on the Vanderbilt Therapeutic Strategies Scale
(VTSS; Butler, Henry, & Strupp, 1992) and the
PACS-SE (Barber & Crits-Christoph, 1996).
Psychotherapy sessions were arranged in ran-
dom order, and entire sessions were viewed by
the two judges. Immediately after viewing a
videotaped session, judges independently rated,
in random order to decrease order effects, the
General Psychodynamic (GD) interviewing
style and Time Limited Dynamic Psychotherapy
(TLDP) specific strategies scales of the VTSS as
well as the Supportive–Adherence, Supportive–
Competence, Expressive–Adherence, and
Expressive–Competence scales of the PAC-SE.
Regular reliability meetings were held during the
coding process to prevent rater drift.
Table 3 presents the ICC (2, 1) and all ICC (2,
2) values for the VTSS and PACS-SE scales; all
were in the good (.60 –.74) or excellent (ⱖ.75)
range. In addition, Table 3 presents Pearson r
correlations between the VTSS and PACS-SE
measures of therapist activity with CPPS-PI rat-
ings from an early (third or fourth) session of 35
patients in psychodynamic psychotherapy. Find-
ings from these analyses revealed that the most
robust relationship was demonstrated between
the CPPS-PI and the GD technique scale of the
VTSS. Large effect size correlations (⬎.50; Co-
hen, 1988) were found between the CPPS PI with
the VTSS TLDP specific strategies scales as well
as the adherence and competence of expressive
techniques. Moderate (⬎.30; Cohen, 1988) to
large effect size correlations were found between
the CPPS PI with the adherence and competence
of supportive techniques.
Criterion validity. To evaluate the criterion
validity of the CPPS (i.e., its ability to distinguish
between different therapeutic modalities), an
analysis of variance (ANOVA) was performed
comparing the mean item and subscale scores in
two different treatment groups. Before conduct-
ing the ANOVA, 2 of the 19 non-PD sample
sessions (both from the same case) were excluded
from this validity analysis. Although these two
sessions were allowed to be included in our in-
vestigation of CPPS reliability, at the request of
the supervising psychologist for this case they
were excluded for the between-group compari-
sons (supervisor consent was also a requirement
for inclusion of data in this study). The remaining
17 rated non-PD sessions were matched on pa-
tient gender, age, global assessment of functional
scale, and session number (none significantly dif-
ferent; p⬎.05) with 17 PD sample sessions.
Mean ratings and standard deviations were
computed for each individual item and subscale
total score across these matched sessions and
were then compared across treatment orienta-
tions. It was hypothesized that the CPPS PI items
and total subscale score would be rated signifi-
cantly higher than the CPPS CB items and total
subscale score in the PD therapy sample. Con-
versely, we expected that the CPPS CB items and
total subscale score would be rated significantly
higher than the CPPS PI items and total subscale
score in the non-PD therapy sample. Tables 4 and
5 present the means and standard deviations for
each item for the 17 matched PD and non-PD
sessions, the Fvalue for each planned compari-
son, the corresponding pvalue, and effect sizes
for each comparison using Cohen’s d(Cohen,
1988) to provide more clinically relevant infor-
mation. On the basis of Cohen’s (1988) recom-
mendation, dvalues of .2, .5, and .8 were used to
represent small, medium, and large effects, re-
spectively. In the calculation of effect sizes, the
pooled standard deviation was used in the de-
nominator. Also, a Bonferroni correction (.05/20)
for this set of CPPS analyses yields a significant
pvalue of .0025.
TABLE 3. Concurrent Validity Correlations for the CPPS
PI Subscale
Scale
ICC CPPS PI
a
(2, 1) (2, 2) rp
VTSS
General Psychodynamic
interviewing style .92 .96 .87 ⬍.0001
TLDP specific strategies .85 .92 .63 ⬍.0001
PACS-SE
Supportive–Adherence .71 .83 .49 .003
Supportive–Competence .79 .88 .49 .003
Expressive–Adherence .85 .92 .84 ⬍.0001
Expressive–Competence .92 .96 .85 ⬍.0001
Note. CPPS PI ⫽Comparative Psychotherapy Process
Scale Psychodynamic–Interpersonal subscale; ICC (2, 1) ⫽
intraclass correlation coefficient two-way random-effects
model; ICC (2, 2) ⫽intraclass correlation coefficient
Spearman-Brown correction for the two-way random-
effects model; nfor ICC ⫽76. VTSS ⫽Vanderbilt Thera-
peutic Strategies Scale; TLDP ⫽Time Limited Dynamic
Psychotherapy; PACS-SE ⫽Penn Adherence-Competence
Scale for Supportive–Expressive Psychotherapy.
a
Early session, n⫽35.
CPPS Development
347
As Table 4 illustrates, each of the CPPS PI
items was found to have significantly higher
mean ratings in the PD therapy sample than in the
non-PD sample, as did the CPPS PI total subscale
score. Also note that all of these differences were
representative of large effects (⬎.80). Similarly,
Table 5 shows that all but one of the CPPS CB
items were found to have significantly higher
mean ratings in the non-PD sample than in the
PD therapy sample, as did the CPPS CB total
subscale score. Again, all of the significant dif-
ferences were representative of large effects (⬎
.80). Only the original version of Item 12 (“The
focus of session is primarily on current life situ-
ations”) did not significantly differentiate the two
treatments. However, it is important to note that
the mean ratings for the original version of Item
12 across the two treatment samples were very
similar.
Because the original version of Item 12 (“The
focus of session is primarily on current life situ-
ations”) did not achieve a significant adjusted
item-to-scale correlation and failed to discrimi-
nate PD and non-PD therapy in the ANOVA,
replacing the present-focused Item 12 with one
that more accurately reflects the differences
found between PI and CB therapy in the literature
reviews may improve the scale and define the
CPPS CB subscale more precisely. One possible
reason for this finding is that, in the initial liter-
ature reviews, short-term PI and CB treatments
were both found to emphasize a patient’s current
TABLE 4. Analysis of Variance Comparing PD- With Non-PD-Treated Patients on the CPPS PI Subscale
Item
no. Item
PD (n⫽17)
Non-PD
(n⫽17)
FpdMSDMSD
1 Explore uncomfortable feelings 4.56 0.68 2.06 1.47 40.59 ⬍.0001 2.19
4 Feelings & percepts linked to past exp. 3.03 1.11 1.09 0.89 31.68 ⬍.0001 1.93
5 Similar relationships over time 3.62 0.86 1.21 1.06 53.10 ⬍.0001 2.50
7 Focus on patient–therapist relationship 2.71 2.19 .50 0.85 14.95 .0005 1.33
8 Experience and expression of feelings 4.65 0.77 1.65 1.10 85.10 ⬍.0001 3.16
10 Address avoid topics & shift in mood 2.21 1.43 .47 0.78 19.38 .0001 1.51
13 Alternative understanding of experiences 3.65 0.93 2.21 1.08 17.43 .0002 1.34
14 Recurrent patterns of action/feel/exp. 3.97 0.93 2.09 1.15 27.65 ⬍.0001 1.80
16 Patient initiates discussion 4.41 0.44 1.88 1.01 89.79 ⬍.0001 3.25
19 Explore wish, fantasy, dream, EM 3.18 0.83 1.12 1.34 29.02 ⬍.0001 1.85
CPPS PI subscale 35.97 5.84 14.27 6.68 101.68 ⬍.0001 3.46
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS PI ⫽Comparative Psychotherapy Process Scale Psychodynamic–Interpersonal subscale; PD ⫽psychody-
namic; EM ⫽early memory.
TABLE 5. Analysis of Variance Comparing PD- With Non-PD-Treated Patients on the CPPS CB Subscale
Item
no. Item
PD (n⫽17)
Non-PD
(n⫽17)
FpdMSDMSD
2 Explicit advice or direct suggestion 1.24 1.37 3.65 0.70 41.70 ⬍.0001 2.22
3 Therapist initiation of topics and activity 2.12 0.70 4.29 0.75 76.72 ⬍.0001 3.00
6 Focus on irrational/illogical belief system 1.38 0.60 4.29 1.06 96.92 ⬍.0001 3.38
9 Specific outside of session activity or task .41 0.85 4.15 1.50 82.50 ⬍.0001 3.12
11 Explain rationale, technique, or treatment .79 0.56 3.09 0.91 78.89 ⬍.0001 3.05
12 Focus primarily on current life situations 4.41 0.62 4.06 0.88 1.83 .1900 0.46
15 Provide information symp, disorder, or tx .77 0.89 4.29 1.11 105.59 ⬍.0001 3.52
17 Practice behaviors between sessions .32 0.77 4.18 1.12 137.15 ⬍.0001 4.02
18 Teach specific techniques to patient .47 0.65 3.56 1.88 41.06 ⬍.0001 2.20
20 Interacts in teacher-like (didactic) manner .62 0.80 4.18 1.12 113.94 ⬍.0001 3.66
CPPS CB subscale 12.53 4.53 39.74 6.08 218.76 ⬍.0001 5.07
Note. Items are abbreviated in table because of formatting requirements; see Appendixes for complete item descrip-
tions. CPPS CB ⫽Comparative Psychotherapy Process Scale Cognitive–Behavioral subscale; PD ⫽psychodynamic.
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
348
life experiences (Ablon & Jones, 1998; Goldfried
et al., 1997; Jones & Pulos, 1993). The fact that
the PD treatment delivered in the current study
was generally of shorter duration may account for
the lack of differences between the PD and
non-PD therapy groups in terms of their focus on
the patients’ current life situation. Therefore, af-
ter additional review of this literature, before any
further analyses, we made a slight modification to
the wording of Item 12 so that the revised item
now reads “The therapist focuses discussion on
the patient’s future life situations.”
Additional Validation and Clinical Utility
Clinical training staff prototype ratings. Fol-
lowing procedures based on Ablon and Jones
(1998), we solicited the clinical training staff
from the Department of Psychiatry at Harvard
Medical School to “please rate each of the 20
items on the scale provided (from 0 –6) according
to how characteristic each item is of an ideally
conducted session that adheres to the principles
of your theoretical orientation.” Forty-three staff
members (23 women, 20 men) representing a
wide variety of graduate training in mental health
(26 PhDs, 10 MDs, 4 LCSWs, and 3 NPs) re-
sponded to this request. The responses of these
training staff on the CPPS were used to ascertain
how well the PI and CB subscales would capture
prototypes of each respective approach to psy-
chotherapy process.
In this group of clinical training staff, 30 iden-
tified their primary theoretical orientation as psy-
chodynamic, PI, or psychoanalytic and 13 others
as CB, cognitive, or behavioral. All of these
training staff had several years of postgraduate
clinical experience (M⫽8 years) as well as an
average of 3 years additional postgraduate train-
ing experience in specific PI or CB treatments
consistent with their major theoretical orienta-
tion. This group of training staff had also been
responsible for supervising therapists within their
major theoretical orientation for, on average, 4 to
5 years, and most had several publications (M⫽
14) concerning their approach to psychotherapy
(technique, application, theory, process, or out-
come). Before proceeding, it is important to note
that there were no significant differences (p⬎
.05) regarding these experience variables be-
tween the two groups.
Coefficient alpha reliabilities based on the rat-
ings of the clinical training staff demonstrated a
high level of internal consistency for both the PI
(.89) and CB (.96) subscales. CPPS PI subscale
scores were significantly different, F(1, 41) ⫽
107.25, p⬍.0001, d⫽3.4, between the PI (n⫽
30; M⫽5.04, SD ⫽.57) and CB (n⫽13; M⫽
3.05, SD ⫽.61) clinicians’ ratings of a prototyp-
ical session. In addition, CPPS CB subscale
scores were also significantly different, F(1,
41) ⫽269.92, p⬍.0001, d⫽5.4, between the PI
(n⫽30; M⫽1.43, SD ⫽.61) and CB (n⫽13;
M⫽4.75, SD ⫽.61) clinicians’ ratings of a
prototypical session. Thus, given these extreme
effect size differences (i.e., d⬎3 represents
different distributions), it seems that the CPPS
was able to robustly differentiate the prototypic
session characteristics of both PI and CB ap-
proaches to treatment made by clinical training
staff from these theoretical orientations.
Novice ratings. We sought to examine how
well the CPPS items would perform in use by
naı¨ve raters evaluating the session characteristics
of PI and CB treatments. One hundred fifteen
undergraduates were recruited to participate in a
study on psychotherapy techniques for course
extra credit. Participants were randomized into
one of two different conditions that watched a
brief segment (approximately 8 min) of a psycho-
therapy session from a commercially available
training tape on approaches to psychotherapy in
which the same patient received treatment from
either a psychodynamic or CB approach to ther-
apy (Insight Media, 1998). All students watched
the video segment in a small-group format, were
given an opportunity to review the CPPS items
both before and during the video segment before
making their ratings, and were provided with the
same instructions from a single investigator who
administered all groups (“Please rate each of the
20 items on the scale provided [from 0 –6] ac-
cording to how characteristic each item was of
the session segment you observed.”)
Of the 115 participants in the study, after ran-
domization, 57 participants observed the psy-
chodynamic treatment segment and 58 partici-
pants observed the CB treatment segment. When
examining group main effects for which session
segment the participants observed (i.e., PD vs.
CB), both the CPPS PI, F(1, 113) ⫽5.0, p⫽.03,
d⫽.42, and CPPS CB, F(1, 113) ⫽48.0, p⬍
.0001, d⫽1.3, subscale scores demonstrated
significant differences. The CPPS PI and CPPS
CB subscales were effective in differentiating
session activity of both the psychodynamic and
CPPS Development
349
CB therapists made by ratings of participants
with no clinical training.
Patient ratings. Following procedures simi-
lar to Silove, Parker, and Manicavasagar (1990),
we sought to examine how well the CPPS items
would perform when used by patients in psy-
chodynamic psychotherapy to evaluate their ses-
sions for both PI and CB treatment techniques.
For this analysis, we used ratings from 136 ses-
sions made by 42 patients receiving psychody-
namic psychotherapy in the treatment research
program described previously (Hilsenroth, 2002).
The demographic characteristics of these 42 pa-
tients are described in greater detail elsewhere
(Hilsenroth et al., 2004). Each participant pro-
vided written informed consent to be included in
this program evaluation research. In addition, pa-
tients were informed both verbally and in writing
that their therapist would not have access to their
responses on the psychotherapy process measures
they completed.
Patients who received psychodynamic psycho-
therapy described their session activity in a man-
ner that clearly differentiated PI and CB technical
interventions, was statistically significant, and
demonstrated robust effects. A paired t-test anal-
ysis demonstrated significant differences in the
amount of PI (M⫽4.01, SD ⫽.93) and CB (M⫽
1.76, SD ⫽.10) techniques patients described in
their psychotherapy sessions across treatment,
N⫽136, t(135) ⫽26.36, p⬍.0001, d⫽2.4.
Therapist and supervisor ratings. We sought
to ascertain how similar CPPS ratings of both PI
and CB treatment techniques would be for ther-
apist and supervisor perspectives on session pro-
cess. For this analysis we used independent rat-
ings from 131 sessions made by 18 therapists and
their supervisor providing psychodynamic psy-
chotherapy in the treatment research program de-
scribed previously (Hilsenroth, 2002). The demo-
graphic characteristics of these therapists as well
as their training, treatment, and supervision are
described in greater detail elsewhere (Hilsenroth
et al., in press; Hilsenroth et al., 2004). Indepen-
dent therapist and supervisor ratings of PI and CB
technical interventions were significantly related
and demonstrated moderate to large effects. Pear-
son rcorrelations demonstrated a significant re-
lationship between therapist and supervisor rat-
ings for both the amount of PI (N⫽131, r⫽.48,
p⬍.0001) and CB (N⫽131, r⫽.47, p⬍.0001)
techniques used in psychotherapy sessions across
treatment.
Discussion
The current study was an initial investigation
of the development, reliability, psychometric
properties, and validity of a new measure assess-
ing psychotherapy techniques used in treatment
sessions: the CPPS. Based on two reviews of the
empirical comparative psychotherapy process lit-
erature (Blagys & Hilsenroth, 2000, 2002), the
CPPS measures the distinctive features of PI and
CB treatments. The measure was not developed
for any highly specific form of therapy or treat-
ment manual. Rather, the scale is intended to be
a more general measure, applicable to a wide
range of PI and CB therapies as found in applied
settings.
These results suggest that the CPPS possesses
excellent interrater reliability and internal consis-
tency, with items and subscales comparing favor-
ably to (if not an improvement over) reliability
statistics of other therapist activity measures re-
ported in the literature (Barber & Crits-Christoph,
1996; Barber et al., 2003; Castonguay et al.,
1996; DeRubeis & Feeley, 1990; Feeley DeRu-
beis, & Gelfand, 1999; Gaston & Ring, 1992;
Gaston, Thompson, Gallagher, Cournoyer, &
Gagnon, 1998; Goldfried et al., 1997; Hill,
O’Grady, & Elkin, 1992; Hollon et al., 1988;
Ogrodniczuk & Piper, 1999; Samoilov et al.,
2000; Shaw et al., 1999; Startup & Shapiro,
1993). In addition, the psychometric features of
the CPPS were consistent with the a priori, em-
pirically based conceptualization of psychother-
apy process designed to differentiate the session
characteristics of PI and CB approaches to
treatment.
In the initial examination of the scale validity,
the CPPS PI subscale was found to be signifi-
cantly related to other extant scales of psychody-
namic treatment activity (in order of relative
magnitude) such as general psychodynamic inter-
viewing style, the competent use of expressive
techniques, the amount of expressive techniques
used, and specific strategies of time-limited dy-
namic psychotherapy. Also, the CPPS PI sub-
scale demonstrated significant but less robust re-
lationships to the amount and competent use of
supportive techniques. Thus, higher scores for the
CPPS items may be related to the competent use
of a technique or intervention being utilized. In
fact, these findings are consistent with the extant
research that demonstrates competence and ad-
herence, across a range of techniques, are not
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
350
necessarily orthogonal but often moderate to
highly related to one another (Barber et al., 2003;
Miller & Binder, 2002). Finally, although some
of these correlations were quite high, it is impor-
tant to keep in mind the unique features of the
CPPS as an empirically derived, brief, general
measure of both PI and CB techniques more
likely to be found in applied clinical practice.
In addition, when comparing mean CPPS rat-
ings in matched samples of PD and non-PD ses-
sions, all but one of the CPPS items (as well as
the CPPS PI and CPPS CB subscale total scores)
were able to discriminate PD from non-PD ses-
sions. A number of studies have shown that forms
of PD–IP therapy could be distinguished from
CB treatment using different measures of psycho-
therapy process, adherence, and competence in
different samples (for reviews, see Blagys &
Hilsenroth, 2000, 2002). The results of the cur-
rent study replicate this previous research and
provide initial evidence supporting the validity of
the CPPS.
Additional validity analyses were also under-
taken to demonstrate the clinical utility of this
measure. The CPPS was able to robustly differ-
entiate the prototypic session characteristics of
both PI and CB approaches to treatment made by
clinical training staff from these theoretical ori-
entations. Consistent with the findings of Ablon
and Jones (1998), the current use of the CPPS
provides a further demonstration that the ratings
of experienced PI and CB clinicians have two
distinct conceptualizations of ideal psychother-
apy process. In addition, the CPPS represents
adequate item coverage to assess such treatment
prototypes. The CPPS PI and CPPS CB subscales
also performed well when used by undergraduate
raters to evaluate the session characteristics of
psychodynamic and CB treatments. Thus, it
seems that the CPPS items are written in clear,
descriptive, experience-near terms that allow na-
ı¨ve raters to assess the use of a technique or
intervention in a session and does not require
inferences about internal mental processes or
highly specialized clinical and theoretical knowl-
edge. This finding is further buttressed by the
results of the patient ratings, whereby patients
who received psychodynamic psychotherapy de-
scribed their session activity in a manner that
significantly differentiated PI and CB technical
interventions with robust effects. Finally, it ap-
pears that therapist and supervisor perspectives
on session process were significantly related to
one another based on independent ratings of PI
and CB treatment interventions from the CPPS.
Despite this broad set of analyses, the results of
the reliability and validity analyses should be
considered preliminary, and some potential limi-
tations are discussed. First, the study lacked a
well-standardized CB treatment comparison
group and, because of the nature of the primary
treatment sample (i.e., short-term psychodynamic
psychotherapy), some of the analyses were better
suited to explore the validity of the PI subscale.
Given this limitation, the validity and applicabil-
ity of the CPPS for a standardized CB treatment
should be examined in future work. Although it is
important to highlight that nonpsychodynamic
sessions were included to provide a preliminary
comparison and initial assessment of the validity
of the scale, the CB prototype ratings from the
clinical training staff were quite robust, and un-
dergraduates were able to accurately differentiate
the delivery of CB treatment using the CB sub-
scale. Second, for several of the analyses, thera-
pists conducting the treatments were graduate
students in training. The extent to which the work
of these clinicians differs from the practice of
more experienced therapists potentially reduces
the external validity of the measure. Last, finan-
cial and time constraints prevented the use of
more coders or those with postgraduate experi-
ence in this initial project. However, it is unlikely
that the use of graduate student judges limited the
generalizability of the CPPS for use in psycho-
therapy research because, in almost all extant
reports of therapist activity or process scales,
raters are trained undergraduates, master’s-level,
or doctorate graduate students.
The results of the current study represent the
initial phase of research in scale development of
the CPPS and suggest that further evaluation is
warranted because the clinical utility of this mea-
sure extends into three overlapping domains.
First, in terms of research, the CPPS identifies
and operationally defines some of the theoreti-
cally central and distinctive activities of non-
manualized PI and CB therapy. In this respect,
the CPPS represents a potentially useful measure
of therapist activity in real-world treatments and
may be used in effectiveness studies conducted
in naturalistic settings, given that the scale was
not developed for a specific therapy or manual
but focuses on more general principles of PI
and CB treatment. Conversely, in more con-
trolled psychotherapy trials, the CPPS may be
CPPS Development
351
used to compare techniques both within and
between various types of CB and PI therapies
to distinguish specific or broad features of
these alternative treatments. Second, the CPPS
can also be used as a training or teaching tool
to help new students and trainees learn about
the general treatment techniques of each ther-
apeutic approach (see Hilsenroth et al., in
press). The instrument may provide a founda-
tion from which to build skills advocated by
the two theoretical orientations, because teach-
ers can point to the CPPS as a clear illustration
of empirically derived differences between the
two treatments. In addition, supervisors will be
able to focus their educational efforts on these
salient, fundamental, orientation-specific activ-
ities and processes. In terms of clinical prac-
tice, the CPPS can provide PI or CB therapists
with a guide for session activity. Clinicians
from each orientation can use the CPPS to
monitor their own use of the salient aspects of
each orientation. Finally, the CPPS may be
useful in the integration of PI and CB ap-
proaches to treatment. By operationally defin-
ing some of the differences between the two
sets of activities, clinicians can begin to eval-
uate the effectiveness of each activity alone or
in combination and attempt to incorporate these
procedures into their treatment repertoire.
References
ABLON,J.S.,&JONES, E. E. (1998). How expert clini-
cian’s prototypes of an ideal treatment correlate with
outcome in psychodynamic and cognitive-behavior
therapy. Psychotherapy Research, 8, 71–83.
ACKERMAN, S., HILSENROTH,M.,&KNOWLES, E. (2005).
Ratings of therapist dynamic activities and alliance
early and late in psychotherapy. Psychotherapy, 42,
225–231.
American Psychiatric Association. (1994). Diagnostic
and statistical manual of mental disorders (4th ed.).
Washington, DC: Author.
BARBER,J.P.,&CRITS-CHRISTOPH, P. (1996). Develop-
ment of a therapist adherence/competence rating scale
for supportive-expressive psychotherapy: A prelimi-
nary report. Psychotherapy Research, 6, 81–94.
BARBER, J. P., CRITS-CHRISTOPH,P.,&LUBORSKY,L.
(1996). Effects of therapist adherence and competence
on patient outcome in brief dynamic therapy. Journal
of Consulting and Clinical Psychology, 64, 619–622.
BARBER, J. P., LIESE,B.,&ABRAMS, M. (2003). Devel-
opment of the Cognitive Therapy Adherence and Com-
petence Scale. Psychotherapy Research, 13, 205–221.
BECK,A.T.,&FREEMAN, A. (1990). Cognitive therapy of
personality disorders. New York: Guilford Press.
BECK, A. T., RUSH, A. J., SHAW,B.F.,&EMERY,G.
(1979). Cognitive therapy for depression. New York:
Guilford Press.
BLAGYS,M.D.,&HILSENROTH, M. J. (2000). Distinctive
features of short-term psychodynamic-interpersonal
psychotherapy: A review of the comparative psycho-
therapy process literature. Clinical Psychology: Science
and Practice, 7, 167–188.
BLAGYS,M.D.,&HILSENROTH, M. J. (2002). Distinctive
features of short-term cognitive-behavioral psychother-
apy: A review of the comparative psychotherapy pro-
cess literature. Clinical Psychology Review, 22, 671–706.
BOOK, H. (1998). How to practice brief psychodynamic
psychotherapy: The core conflictual relationship theme
method. Washington, DC: American Psychological
Association.
BOURNE, E. J. (1995). The anxiety and phobia work book
(2nd ed.). Oakland, CA: New Harbinger.
BUTLER, S., HENRY,W.,&STRUPP, H. (1992). Measuring
adherence and skill in time-limited dynamic psychother-
apy. Unpublished manuscript, Vanderbilt University.
CASTONGUAY, L. G., GOLDFRIED, M. R., WISER, S.,
RAUE,P.J.,&HAYES, A. M. (1996). Predicting the
effect of cognitive therapy for depression: A study of
unique and common factors. Journal of Consulting and
Clinical Psychology, 64, 497–504.
COHEN, J. (1988). Statistical power analysis for the behav-
ioral sciences (3rd ed.). New York: Academic Press.
CRAIGHEAD, L. W., CRAIGHEAD, W. E., KAZDIN, A. E., &
MAHONEY, M. J. (1994). Cognitive and behavioral in-
terventions. Boston: Allyn & Bacon.
CRITS-CHRISTOPH, P., SIQUELAND, L., CHITTAMS, J., BAR-
BER, J., BECK, A., FRANK, A., ET AL. (1998). Training in
cognitive, supportive-expressive, and drug counseling
therapies for cocaine dependence. Journal of Consult-
ing and Clinical Psychology, 66, 484–492.
DAVIS,M.ESHELMAN,E.R.,&MCKAY, M. (1995). The
relaxation and stress reduction workbook (4th ed.).
Oakland, CA: New Harbinger.
DERUBEIS,R.J.,&FEELEY, M. (1990). Determinants of
change in cognitive therapy for depression. Cognitive
Therapy and Research, 14, 469–482.
DERUBEIS, R. J., HOLLON, S. D., EVANS,M.D.,&BEMIS,
K. M. (1982). Can psychotherapies for depression be
discriminated? A systematic investigation of cognitive
therapy and Interpersonal therapy. Journal of Consult-
ing and Clinical Psychology, 50, 744–756.
FEELEY,M.DERUBEIS,R.J.,&GELFAND, L. A. (1999).
The temporal relation of adherence and alliance to
symptom change in cognitive therapy for depression.
Journal of Consulting and Clinical Psychology, 67, 578–
582.
FLEISS, J. L. (1981). Statistical methods for rates and pro-
portions (2nd ed.). New York: Wiley.
GASTON,L.,&RING, J. M. (1992). Preliminary results on
the Inventory of Therapeutic Strategies. Journal of Psy-
chotherapy Practice and Research, 1, 135–146.
GASTON, L., THOMPSON, L., GALLAGHER, D., COURNOYER,
L.,&G
AGNON, R. (1998). Alliance, technique, and their
interactions in predicting outcome of behavioral, cogni-
tive, and brief dynamic therapy. Psychotherapy Research,
8, 190–209.
GOLDFRIED, M. R., CASTONGUAY, L. G., HAYES,A.M.,
DROZD,J.F.,&SHAPIRO, D. A. (1997). A comparative
analysis of the therapeutic focus in cognitive-behavioral
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
352
and psychodynamic-interpersonal sessions. Journal of
Consulting and Clinical Psychology, 65, 740–748.
GOLDFRIED,M.R.,&DAVISON, G. C. (1994). Clinical
behavior therapy. New York: Wiley.
GOLDFRIED, M. R., NEWMAN,C.F.,&HAYES,A.M.
(1989). The coding system of therapeutic focus. Unpub-
lished manuscript, State University of New York at
Stony Brook.
GOLDFRIED, M. R., RAUE,P.J.,&CASTONGUAY,L.G.
(1998). The therapeutic focus in significant sessions of
master therapists: A comparison of cognitive-
behavioral and psychodynamic-interpersonal interven-
tions. Journal of Consulting and Clinical Psychology,
66, 803–810.
HAYES, A. M., CASTONGUAY,L.G.,&GOLDFRIED,M.R.
(1996). Effectiveness of targeting vulnerability factors
of depression in cognitive therapy. Journal of Consult-
ing and Clinical Psychology, 64, 623–627.
HAYES,A.M.,&STRAUSS, J. L. (1998). Dynamic systems
theory as a paradigm for the study of change in psycho-
therapy: An application to cognitive therapy for de-
pression. Journal of Consulting and Clinical Psychol-
ogy, 66, 939–947.
HILL, C. E., O’GRADY,K.E.,&ELKIN, I. (1992). Apply-
ing the Collaborative Study Psychotherapy Rating
Scale to rate therapist adherence in cognitive-
behavioral therapy, interpersonal therapy, and clinical
management. Journal of Consulting and Clinical Psy-
chology, 60, 73–79.
HILSENROTH, M. (2002). Adelphi University: Psychody-
namic psychotherapy process and outcome research
team. In P. Fonagy, J. Clarkin, A. Gerber, H. Ka¨ chele,
R. Krause, E. Jones, et al. (Eds.), An open door review
of outcome studies in psychoanalysis (2nd ed., pp. 241–
247). London: International Psychoanalytical Association.
HILSENROTH, M., ACKERMAN,S.,&BLAGYS, M. (2001).
Evaluating the phase model of change during short-
term psychodynamic psychotherapy. Psychotherapy
Research, 11, 29–47.
HILSENROTH, M., ACKERMAN, S., BLAGYS, M., BAITY, M.,
&M
OONEY, M. (2003). Short-term psychodynamic psy-
chotherapy for depression: An evaluation of statistical,
clinically significant, and technique specific change.
Journal of Nervous and Mental Disease, 191, 349–357.
HILSENROTH, M., DEFIFE, J., BLAGYS,M.,&ACKERMAN,
S. (IN PRESS). Effects of training in short-term psy-
chodynamic psychotherapy: Changes in graduate clini-
cian technique. Psychotherapy Research.
HILSENROTH, M., PETERS, E., & ACKERMAN, S. (2004).
The development of therapeutic alliance during psy-
chological assessment: Patient and therapist perspec-
tives across treatment. Journal of Personality Assess-
ment, 83, 332–344.
HOLLON, S. D., EVANS, M. D., AUERBACH, A., DERU-
BEIS, R. J., ELKIN, I., LOWERY, A., ET AL. (1988). De-
velopment of a system for rating therapies for depres-
sion: Differentiating cognitive therapy, interpersonal
therapy, and clinical management pharmacotherapy.
Unpublished manuscript, Department of Psychology,
Vanderbilt University.
Insight Media. (1998). Psychology: Approaches to ther-
apy. New York: Author.
JACOBSON, N. (1998, JULY). How important are cognitive
interventions in cognitive therapy? World Congress of
Behavior and Cognitive Therapies, Acapulco, Mexico.
JACOBSON, N., DOBSON, K., TRUAX, P., ADDIS, M.,
KOERNER, K., GOLLAN,J.K.,ET AL. (1996). A compo-
nent analysis of cognitive-behavioral treatment for de-
pression. Journal of Consulting and Clinical Psychol-
ogy, 64, 295–304.
JONES, E. E. (1985). Manual for the Psychotherapy Pro-
cess Q Sort. Unpublished manuscript, University of
California, Berkeley.
JONES,E.E.,&PULOS, S. M. (1993). Comparing the
process in psychodynamic and cognitive-behavioral
therapies. Journal of Consulting and Clinical Psychol-
ogy, 61, 306–316.
LIESE, B., BARBER,J.,&BECK, A. (1995). The Cognitive
Therapy Adherence and Competence Scale. Unpub-
lished manuscript, University of Kansas Medical Center.
LUBORSKY, L. (1984). Principles of psychoanalytic psy-
chotherapy: A manual for supportive-expressive treat-
ment. New York: Basic Books.
MASTERS, J. C., BURISH, T. G., HOLLON,S.D.,&RIMM,
D. C. (1987). Behavior therapy: Techniques and empir-
ical findings. New York: Harcourt Brace Jovanovich.
MILLER,S.,&BINDER, J. (2002). The effects of manual-
based training on treatment fidelity and outcome: A
review of the literature on adult individual psychother-
apy. Psychotherapy, 39, 184–198.
NUNNALLY,J.,&BERNSTEIN, I. (1994). Psychometric the-
ory (3rd ed.). New York: McGraw-Hill.
O’MALLEY, S. S., FOLEY, S. H., ROUNSAVILLE,B.J.,
WATKINS, J. T., SOTSKY, S. M., IMBER,S.D.,&ELKIN,
I. (1988). Therapist competence and patient outcome in
interpersonal psychotherapy of depression. Journal of
Consulting and Clinical Psychology, 56, 496–501.
OGRODNICZUK,J.S.,&PIPER, W. E. (1999). Measuring
therapist technique in psychodynamic psychotherapies:
Development and use of a new scale. Journal of Psy-
chotherapy Practice and Research, 8, 142–154.
PRICE, J., HILSENROTH, M., CALLAHAN, K., PETRETIC-
JACKSON,P.,&BONGE, D. (2004). A pilot study of
psychodynamic psychotherapy for adult survivors of
childhood sexual abuse. Clinical Psychology and Psy-
chotherapy, 11, 378–391.
SAMOILOV, A., GOLDFRIED,M.R.,&SHAPIRO,D.A.
(2000). Coding system of therapeutic focus on action
and insight. Journal of Consulting and Clinical Psychol-
ogy, 68, 513–514.
SHAPIRO,D.A.,&STARTUP, M. J. (1990). Raters’ manual
for the Sheffield Psychotherapy Rating Scale (Memo 1154,
MRC/ESRC). Sheffield, UK: University of Sheffield.
SHAW, B. F., ELKIN, I., YAMAGUCHI, J., OLMSTED, M.,
VALLIS, T. M., DOBSON,K.S.,ET AL. (1999). Therapist
competence ratings in relation to clinical outcome in
cognitive therapy of depression. Journal of Consulting
and Clinical Psychology, 67, 837–846.
SHROUT,P.E.,&FLEISS, J. L. (1979). Intraclass correla-
tions: Uses in assessing rater reliability. Psychological
Bulletin, 86, 420–428.
SILOVE, D., PARKER,G.,&MANICAVASAGAR, V. (1990).
Perceptions of general and specific therapist behaviors.
Journal of Nervous and Mental Diseases, 178, 292–299.
STARTUP,M.,&SHAPIRO, D. A. (1993). Therapist treatment
fidelity in prescriptive vs. exploratory psychotherapy. Brit-
ish Journal of Clinical Psychology, 32, 443–456.
CPPS Development
353
STRUPP,H.,&BINDER, J. (1984). Psychotherapy in a new
key: A guide to time-limited dynamic psychotherapy.
New York: Basic Books.
THOMPSON-BRENNER,H.&WESTEN,D.(IN PRESS). A
naturalistic study of psychotherapy for bulimia nervosa,
Pt. 2: Therapeutic interventions and outcome in the
community. Journal of Nervous and Mental Disease.
THOMPSON-BRENNER,H.,&WESTEN, D. (2005). Person-
ality subtypes in eating disorders: Validation of a clas-
sification in a naturalistic sample. British Journal of
Psychiatry, 186, 516–524.
VALLIS, T., SHAW,B.,&DOBSON, K. (1986). The Cogni-
tive Therapy Scale: Psychometric properties. Journal of
Consulting and Clinical Psychology, 54, 381–385.
WACHTEL, P. L. (1993). Therapeutic communication. New
York: Guilford Press.
WALTZ, J., ADDIS, M. E., KOERNER,K.,&JACOBSON,N.
(1993). Testing the integrity of a psychotherapy proto-
col: Assessment of adherence and competence. Journal
of Consulting and Clinical Psychology, 61, 620–630.
WESTEN, D., NOVOTNY,C.,&THOMPSON-BRENNER,H.
(2004). The empirical status of empirically supported
therapies: Assumptions, findings, and reporting in
controlled clinical trials. Psychological Bulletin, 130,
631–663.
YOUNG,J.,&BECK, A. T. (1980). Cognitive Therapy
Scale rating manual. Unpublished manuscript, Univer-
sity of Pennsylvania, Philadelphia.
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
354
(Appendixes continue)
Appendix A
CPPS–Form ER/T
Therapist Session #
Patient ID Rater
Instructions: Using the scale provided below, please rate how characteristic each statement was of the therapy session. For
each item, please write the scale rating number on the blank line provided.
Scale:
0123456
Not at all characteristic Somewhat characteristic Characteristic Extremely characteristic
(1) The therapist encourages the exploration of feelings regarded by the patient as uncomfortable (e.g., anger,
envy, excitement, sadness, or happiness).
(2) The therapist gives explicit advice or direct suggestions to the patient.
(3) The therapist actively initiates the topics of discussion and therapeutic activities.
(4) The therapist links the patient’s current feelings or perceptions to experiences of the past.
(5) The therapist focuses attention on similarities among the patient’s relationships repeated over time, settings,
or people.
(6) The therapist focuses discussion on the patient’s irrational or illogical belief systems.
(7) The therapist focuses discussion on the relationship between the therapist and patient.
(8) The therapist encourages the patient to experience and express feelings in the session.
(9) The therapist suggests specific activities or tasks (homework) for the patient to attempt outside of session.
(10) The therapist addresses the patient’s avoidance of important topics and shifts in mood.
(11) The therapist explains the rationale behind his or her technique or approach to treatment.
(12) The therapist focuses discussion on the patient’s future life situations.
(13) The therapist suggests alternative ways to understand experiences or events not previously recognized by
the patient.
(14) The therapist identifies recurrent patterns in patient’s actions, feelings, and experiences.
(15) The therapist provides the patient with information and facts about his or her current symptoms, disorder,
or treatment.
(16) The therapist allows the patient to initiate the discussion of significant issues, events, and experiences.
(17) The therapist explicitly suggests that the patient practice behavior(s) learned in therapy between sessions.
(18) The therapist teaches the patient specific techniques for coping with symptoms.
(19) The therapist encourages discussion of patient’s wishes, fantasies, dreams, or early childhood memories
(positive or negative).
(20) The therapist interacts with the patient in a teacher-like (didactic) manner.
CPPS Development
355
Appendix B
CPPS–Form P
Therapist Session #
Patient ID Rater
Instructions: Using the scale provided below, please rate how characteristic each statement was of the therapy session. For
each item, please write the scale rating number on the blank line provided.
Scale:
0123456
Not at all characteristic Somewhat characteristic Characteristic Extremely characteristic
(1) My therapist encouraged me to explore feelings that are hard for me to talk about (e.g., anger, envy,
excitement, sadness, or happiness).
(2) My therapist gave me explicit advice or direct suggestions for solving my problems.
(3) My therapist actively initiated the topics of discussion and activities during the session.
(4) My therapist linked my current feelings or perceptions to experiences in my past.
(5) My therapist brought to my attention similarities between my past and present relationships.
(6) Our discussion centered on irrational or illogical belief systems.
(7) The relationship between the therapist and myself was a focus of discussion.
(8) My therapist encouraged me to experience and express feelings in the session.
(9) My therapist suggested specific activities or tasks (homework) for me to attempt outside of the session.
(10) My therapist addressed my avoidance of important topics and shifts in my mood.
(11) My therapist explained the rationale behind his or her technique or approach to treatment.
(12) The focus of our session was primarily on future life situations.
(13) My therapist suggested alternative ways to understand experiences or events I had not previously
recognized.
(14) My therapist identified recurrent patterns in my actions, feelings, and experiences.
(15) My therapist provided me with information and facts about my current symptoms, disorder, or treatment.
(16) I initiated the discussion of significant issues, events, and experiences.
(17) My therapist explicitly suggested that I practice behavior(s) learned in therapy between sessions.
(18) My therapist taught me specific techniques for coping with my symptoms.
(19) My therapist encouraged discussion of wishes, fantasies, dreams, or early childhood memories (positive or
negative).
(20) My therapist interacted with me in a teacher-like (didactic) manner.
Hilsenroth, Blagys, Ackerman, Bonge, and Blais
356
A preview of this full-text is provided by American Psychological Association.
Content available from Psychotherapy Theory Research Practice Training
This content is subject to copyright. Terms and conditions apply.
