ArticlePDF Available

Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial

  • University of Huddersfield and South West Yorkshire Partnership NHS Foundation Trust

Abstract and Figures

Background: Previous research suggests that using outcome feedback technology can enable psychological therapists to identify and resolve obstacles to clinical improvement. This study aimed to evaluate the effectiveness of an outcome feedback quality assurance system applied in stepped care psychological services. Methods: This multi-site cluster randomised controlled trial (registration DOI: 10.1186/ISRCTN12459454) included 2233 patients with depression and anxiety disorders accessing at least 2 sessions of individual psychological therapy delivered by 77 therapists across 8 healthcare organisations. Therapists were randomised to a feedback intervention group (N = 39) or a treatment-as-usual control group (N = 38). The feedback technology alerted therapists to cases that were “not on track”, and primed them to review these in clinical supervision. Post-treatment symptom severity on validated depression (PHQ-9) and anxiety (GAD-7) measures was compared between groups using multilevel modelling, controlling for cluster (therapist) effects, following an intention-to-treat approach. Findings: Cases classified as not on track had significantly less severe symptoms after treatment if they were allocated to the feedback group (PHQ-9 d = 0.23, B = -1.03 [95% CI = -1.84, -0.23], p = 0.012; GAD-7 d = 0.19, B = -0.85 [95% CI = -1.56, -0.14], p = 0.019). There were no between-group differences in the odds of reliable improvement (OR = 1.32 [0.93, 1.89], p = 0.12); however, control cases classed as not on track had significantly greater odds of reliable deterioration (OR = 1.73 [1.18, 2.54], p = 0.0050). Interpretation: Supplementing psychological therapy with low-cost feedback technology prevents deterioration in cases at risk of poor response to treatment. This evidence supports the implementation of outcome feedback in stepped care psychological services.
Content may be subject to copyright.
Author’s Manuscript
Note: This is a pre-print peer reviewed article, accepted for publication on
12.04.18. Please do not copy or share without the authors permission.
Citation: Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J.,
Gilbody, S., Ali, S., Aguirre, E., Appleton, M., Nevin, J., O’Hayon, H., Patel,
U., Sainty, A., Spencer, P., & McMillan, D. (2018). Feedback-informed
treatment versus usual psychological treatment for depression and anxiety:
a multisite, open-label, cluster randomised controlled trial. Lancet
Psychiatry, 5, 56472. doi: 10.1016/S2215-0366(18)30162-7
Feedback-informed treatment versus usual psychological treatment
for depression and anxiety: a multisite, open-label,
cluster randomised controlled trial
Jaime Delgadillo
, Kim de Jong2, Mike Lucock3,12, Wolfgang Lutz4, Julian Rubel4,
Simon Gilbody5, Shehzad Ali5, Elisa Aguirre6, Mark Appleton7, Jacqueline Nevin8,
Harry O’Hayon9, Ushma Patel10, Andrew Sainty11, Peter Spencer12,
and Dean McMillan5
1. Clinical Psychology Unit, Department of Psychology, University of Sheffield, Sheffield, United Kingdom
2. Institute of Psychology, Leiden University, The Netherlands
3. Centre for Applied Research in Health, University of Huddersfield, Huddersfield, United Kingdom
4. Department of Psychology, University of Trier, Germany
5. Department of Health Sciences, University of York, United Kingdom
6. North East London NHS Foundation Trust, London, United Kingdom
7. Pennine Care NHS Foundation Trust, Hyde, United Kingdom
8. Cheshire and Wirral Partnership NHS Foundation Trust, Cheshire, United Kingdom
9. Whittington Health NHS Trust, London, United Kingdom
10. Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, United Kingdom
11. Humber NHS Foundation Trust, Hessle, United Kingdom
12. South West Yorkshire Partnership NHS Foundation Trust, Barnsley, United Kingdom
Declaration of interests: None.
Correspondence: Dr Jaime Delgadillo, Clinical Psychology Unit, University of Sheffield, Floor F, Cathedral
Court, 1 Vicar Lane, Sheffield S1 2LT, UK.
Background: Previous research suggests that using outcome feedback technology
can enable psychological therapists to identify and resolve obstacles to clinical
improvement. This study aimed to evaluate the effectiveness of an outcome
feedback quality assurance system applied in stepped care psychological services.
Methods: This multi-site cluster randomised controlled trial (registration DOI:
10.1186/ISRCTN12459454) included 2233 patients with depression and anxiety
disorders accessing at least 2 sessions of individual psychological therapy
delivered by 77 therapists across 8 healthcare organisations. Therapists were
randomised to a feedback intervention group (N = 39) or a treatment-as-usual
control group (N = 38). The feedback technology alerted therapists to cases that
were “not on track”, and primed them to review these in clinical supervision. Post-
treatment symptom severity on validated depression (PHQ-9) and anxiety (GAD-7)
measures was compared between groups using multilevel modelling, controlling
for cluster (therapist) effects, following an intention-to-treat approach.
Findings: Cases classified as not on track had significantly less severe symptoms
after treatment if they were allocated to the feedback group (PHQ-9 d = 0.23, B = -
1.03 [95% CI = -1.84, -0.23], p = 0.012; GAD-7 d = 0.19, B = -0.85 [95% CI = -1.56,
-0.14], p = 0.019). There were no between-group differences in the odds of reliable
improvement (OR = 1.32 [0.93, 1.89], p = 0.12); however, control cases classed as
not on track had significantly greater odds of reliable deterioration (OR = 1.73 [1.18,
2.54], p = 0.0050).
Interpretation: Supplementing psychological therapy with low-cost feedback
technology prevents deterioration in cases at risk of poor response to treatment.
This evidence supports the implementation of outcome feedback in stepped care
psychological services.
Research in context
Evidence before this study
Previous research suggests that using inexpensive quality improvement strategies
such as routine outcome monitoring and feedback can improve psychological
treatment outcomes, in particular for cases that are prone to deterioration. The
generalisability of previous trials is limited by their application in specialist
university or psychotherapy clinics, and observational studies in primary care were
likely to be statistically underpowered.
Added value of this study
This large-scale, pragmatic, randomised controlled trial was adequately powered
to detect small effect size differences, and designed to evaluate the generalisability
of feedback effects across multiple primary care psychological therapy services.
The findings indicate that feedback-informed treatment can improve outcomes for
patients classed as not on track, and with remarkable consistency across different
services, therapists, and intensities of treatment.
Implications of all the available evidence
There is now a compelling evidence base to support the implementation of outcome
monitoring and feedback technologies in mainstream psychological services.
Implementing this low-cost, automated feedback and quality assurance system
can help to prevent deterioration for cases that are at risk of poor treatment
A number of psychological interventions, ranging from brief guided self-help to
more intensive psychotherapies, are effective for the treatment of depression and
anxiety disorders.1 Large-scale evaluations of such treatments applied in routine
care are generally favourable, although it is also known that at least 30% of
patients do not show statistically reliable improvement and some deteriorate.2-3
Previous studies have shown that patients at risk of poor response to treatment
can be identified early using outcome feedback methods.4 Outcome feedback is a
quality assurance method which involves routinely monitoring a patient’s
condition using standardised measures which are compared to data from a
normative clinical sample.5 Using data charts or automated electronic monitoring
technologies, cases that are not on track are detected when their symptoms are
significantly worse than those of similar cases.
Several reviews of experimental and practice-based studies suggest that
using outcome feedback can help to improve treatment outcomes by comparison
to usual psychological care.4,6-8 Simply collecting patient-reported outcome
measures in clinical practice is not associated with improved outcomes,9 so it is
plausible that the risk signal element of feedback technologies serves to
effectively prompt therapists to identify and to resolve obstacles to improvement.
This mechanism of action is supported by evidence from controlled trials where
therapy supported with risk signalling yielded better outcomes than routine
psychological care.6-8 An early meta-analysis suggested that supplementing the
signal with clinical decision-making and support tools further enhances its
effectiveness,6 although a more recent meta-analysis contradicts this finding.9 It
has also been proposed that outcome feedback specifically helps to prevent
deterioration in cases classed as not on track.6-8 A recent systematic review of the
literature concluded that studies that applied risk signalling technology show some
evidence of improved outcomes for not on track cases, but the effect sizes were
small (standardised mean difference of -0.22).9 Furthermore, some studies have
not found a differential effect of feedback in the not on track subgroup10-12 and one
study found that using feedback possibly deteriorates outcomes for not on track
cases with cluster B personality disorders.13
Overall, the literature shows mixed and inconclusive evidence for the use of
feedback technologies, and the methodological quality of studies has been rated
as generally low.9 This variability raises questions about the generalisability of
feedback, justifying the need to carefully evaluate its acceptability, feasibility and
effectiveness prior to adoption in routine care.14 Some studies have suggested that
outcome feedback may be particularly helpful in short-term evidence-based
therapies such as cognitive behavioural therapy, and could enhance the efficiency
of treatment.10,11 A recent study reported qualitative evidence that feedback-
assisted brief psychological interventions were acceptable to patients with
depression and anxiety disorders, and feasible to implement in a routine primary
care setting.11 This study also suggested that outcome feedback could reduce the
cost and enhance the efficiency of treatment, although it was limited by the use of
historical control group data in a non-randomized design. In spite of these
promising results, more rigorous experimental evidence is necessary to establish
the generalisability and efficacy of feedback in primary care settings. The present
study aimed to address this gap in the literature through a multi-site randomised
controlled trial applied in primary care psychological services for common mental
health problems.
Study design
This was a pragmatic, multi-site, cluster randomised controlled trial. The objective
was to assess the clinical effect of feedback-assisted psychological treatments, in
comparison to routinely delivered psychological care. The central hypothesis was
that using feedback would result in lower mean symptom severity for not on track
cases, in comparison to usual care. The primary outcome was depression and
anxiety symptom severity assessed at the last treatment session using validated
patient-reported outcome measures described below. Secondary outcomes
included work and social adjustment, treatment duration, reliable improvement,
reliable deterioration, treatment dropout rates and the percentage of cases
classified as not on track.
The design involved randomising participating therapists (and all of their
patients meeting inclusion criteria described below) to a feedback intervention
group or a treatment-as-usual control group. The rationale for this design was two-
fold. First, randomising therapists would minimise the risk of contamination of
controls through practice effects, which could occur if the same therapist were to
treat some patients with and others without using outcome feedback technology.
Secondly, this cluster design adequately represents the natural nesting of patients
within therapists, thus enabling us to control for variability in outcomes
attributable to therapists (therapist effects15).
Using the Optimal Design Software for Multi-level and Longitudinal
Research (Version 3.01)16, we estimated that a minimum of 60 therapists (30 per
group) each of whom treated an average of 10 patients was required to detect a
small effect size with an alpha level of α = 0.05 and 80% power. This calculation
assumed an intracluster correlation coefficient of ICC = 0.05, guided by previous
studies investigating therapist effects in naturalistic samples.15,17 We aimed to
recruit up to 80 therapists to account for attrition.
The study was approved by the London - City & East NHS Research Ethics
Committee (06/01/2016, Ref: 15/LO/2200) and the protocol was registered in an
international database prior to recruitment (DOI: 10.1186/ISRCTN12459454).
Setting and interventions
The study was conducted in eight National Health Service (NHS) Trusts in
England. Together, these services covered a large primary care population across
London, Cambridge, Cheshire & Wirral, Bury, Heywood, Middleton, Rochdale,
Oldham, Stockport, Tameside & Glossop, Trafford, Barnsley, and East Riding.
All participating services were part of the national Improving Access to
Psychological Therapies (IAPT) programme, which offers protocol-driven, evidence-
based psychological interventions for depression and anxiety disorders organised
in a stepped care model.18 Low intensity guided self-help based on principles of
cognitive behavioural therapy (LiCBT) was offered as an initial treatment in most
cases with mild-to-moderate depression and/or anxiety problems. LiCBT is
delivered by trained coaches (psychological wellbeing practitioners) in a variety of
different formats (e.g., individual or group psychoeducation, computerised CBT
with telephone support) and typically lasts under 8 sessions. Those with more
severe or complex problems, and those who did not respond to LiCBT were
“stepped up” to high intensity (up to 20 sessions) psychotherapies including CBT,
interpersonal psychotherapy, and counselling for depression. The specific
treatment recommendation for each case followed standard clinical guidelines.19
Treatment was supported by regular (weekly or fortnightly) clinical supervision
delivered in a peer-supervision model organised within each service.
Therapists qualified to deliver low or high intensity interventions were eligible to
take part, with the exception of (1) therapists with short-term employment
contracts or (2) trainees who were not yet fully qualified. The trial included all
patients that accessed individual (low and/or high intensity) therapy with
participating therapists, excluding patients who accessed group therapies and
those who attended less than 2 individual therapy sessions. The latter condition
was applied because: (1) outcome measures for patients that accessed 1 session
reflect symptom severity for a pre-treatment period of 2 weeks, and (2) the outcome
feedback technology requires at least 2 sessions to provide a progress feedback
signal taking session 1 as a baseline score. The allocation of patients to therapists
in routine care was quasi-random, where patients on waiting list were allocated
sequentially based on therapist availability.
Outcome feedback quality assurance system
Therapists in all participating services routinely recorded their patients’
clinical outcomes using an electronic clinical record system called Patient Case
Management Information System (PCMIS; PCMIS
includes outcome monitoring graphs which chart depression and anxiety symptom
severity scores at every session. Therapists randomised to the experimental group
had access to enhanced outcome monitoring graphs which included expected
treatment response curves. The expected treatment response curves represent 80%
prediction intervals, which are estimated using growth curve modelling in data
from a normative clinical sample.5,20-21 Expected treatment response curves were
calculated for subgroups of cases with the same baseline symptom severity, using
a large clinical dataset of cases treated in IAPT (further details described
elsewhere22). These enhanced outcome monitoring graphs automatically generated
a “red signal” to alert therapists to not on track cases whose depression and/or
anxiety symptoms surpassed the 80% upper boundary of the expected treatment
response curves. Control group therapists only had access to standard outcome
tracking graphs, but without expected treatment response curves or automated
risk signals.
Therapists randomised to the feedback group attended a standardised 6.5-
hour training programme which covered: outcome feedback theory and evidence-
base; instructions on how to use the feedback tool; clinical trouble-shooting skills.
The training required therapists to follow the following process: (1) review outcome
feedback graphs with patients at the start of every session; (2) if the graph shows
a risk signal, discuss this with the patient to collaboratively identify potential
obstacles to improvement; (3) prioritise discussing not on track cases with your
clinical supervisor; (4) use information from points 2 and 3 to develop a plan to
address obstacles; (5) use outcome feedback graphs to assess how your plan is
working. Therapists were also primed to be aware of variables that have been
empirically shown to be associated with treatment outcomes (patient, therapist,
process, and wider context factors). This information and evidence-base was
synthesised in a clinical guideline that therapists assigned to the feedback group
received after training.23
Outcome measures and secondary data
Patients accessing the participating services routinely self-completed standardised
outcome measures before each session; the measures obtained at the last
treatment session were taken as primary outcomes in the trial. The Patient Health
Questionnaire (PHQ-9) is a nine-item screening tool for depression, where each
item is rated on a 0 to 3 scale, yielding a total depression severity score between
027.24 A cut-off 10 has been recommended to screen for major depression,24
and a difference of ≥6 points between assessments is indicative of reliable change.25
The Generalized Anxiety Disorder questionnaire (GAD-7) is a seven-item
measure developed to screen for anxiety disorders.26 It is also rated using a 0 to 3
scale, yielding a total anxiety severity score between 021. A cut-off score ≥8 is
recommended to identify the likely presence of a diagnosable anxiety disorder,26
and a difference of ≥5 points is indicative of reliable change.25
Secondary data sources included demographics (age, gender, ethnicity,
employment status), stepped care pathway information, number of treatment
sessions, primary diagnoses recorded in clinical records and functional
impairment measured using the Work and Social Adjustment Scale (WSAS).27
Recruitment, randomisation and data collection
Recruitment took place between January and July 2016. A participant information
sheet and consent form were shared via email with all therapists working in
participating services. Therapists had an opportunity to clarify questions with the
principal investigator before providing signed consent forms directly to the
research team. Parallel-group random allocation was independently performed by
a researcher using a computer-generated (1:1) randomisation algorithm to prevent
selection bias within services. Given the nature of the outcome feedback
technology, this was an open-label trial where therapists were aware of their
allocation. Session-by-session depression (PHQ-9) and anxiety (GAD-7) outcome
measures were collected for all eligible patients who accessed individual therapy
with participating therapists during a one-year study period.
Data analysis
Patients’ characteristics were compared between groups (those included and
excluded from the trial sample) using Mann-Whitney U tests for continuous
variables and chi-square tests for categorical variables. A small number of cases
(N = 98; 4.4% of the trial sample) had missing post-treatment outcome measures
which were imputed by averaging the imputed values from 25 estimated datasets
using an expectation maximization method.28 This imputation was carried out so
that we could conduct intention-to-treat analyses, including post-treatment
outcomes for all cases regardless of completion or dropout status.
The primary analysis was carried out using multilevel modelling (MLM) with
separate models for PHQ-9 and GAD-7 outcomes. Following conventional model
building guidelines,29 we initially examined the hierarchical structure of the
dataset using unconditional models predicting post-treatment symptom severity.
The “site variable was not statistically significant in a three-level model (patients
within therapists within sites), so subsequent analyses used two-level models
(patients within therapists). Next, we considered different covariance structures,
assessed non-linear (i.e., quadratic, log-linear) trends in the number of treatment
sessions, and assessed goodness-of-fit (using AIC, BIC, −2 log likelihood statistics).
After initial model checking, the primary analysis applied a twolevel model,
including random intercepts for therapists, with an unstructured covariance
matrix, and an identity link-function. No cases included in the trial sample had
two interventions delivered by different therapists (e.g., low followed by high
intensity therapy), so crossed random effects were not modelled. Continuous
variables were grand-mean centred and an intracluster correlation coefficient (ICC)
was calculated to assess the proportion of variance in outcomes attributable to
therapists. An initial conditional model included the following predictors: baseline
severity of symptoms, log transformed number of sessions, and group (feedback
vs. control), which compared between-group differences in post-treatment
symptom severity. Next, a fully adjusted model additionally included a case
classification (case classified as on track vs. not on track), and a group *
classification interaction term (main hypothesis test). This MLM strategy was
repeated in a sensitivity analysis controlling for age and step of care (low vs. high
intensity treatment).
Secondary analyses assessed other relevant clinical outcomes. The fully
adjusted MLM was repeated using the WSAS as a dependent variable to assess
potential effects of feedback on functional impairment. Poisson MLM was used to
compare between-group differences in treatment duration, controlling for baseline
PHQ-9 and GAD-7. Logistic MLM was used to compare between-group probabilities
(odds ratios) of meeting post-treatment criteria for reliable improvement (RI), after
controlling for baseline severity (PHQ-9 and GAD-7). The RI classification required
patients to have statistically reliable improvement in at least one of the outcome
measures, as long as the other measure did not show reliable deterioration.
Logistic MLM was also used to estimate between-group odds ratios for the % of
cases with reliable deterioration (in at least one outcome measure), the percentage
of cases classed as not on track, and the percentage of cases that dropped-out of
treatment. These models were computed using the full sample and repeated in the
not on track subsample.
Role of the funding source
The study was partly supported by research capability funding awarded by the
English National Health Service (NHS) and partly funded by a visiting research
fellowship awarded to the principal investigator by the Department of Health
Sciences, University of York. The funding organisations had no role in the decision
to publish the study.
Sample characteristics
In total, 79 therapists were recruited but 2 did not participate (see Figure 1). Of
the 77 participating therapists, 39 (50.6%) were randomised to the feedback group
and 38 (49.4%) to the control group. Of these, 48 (62.3%) delivered high intensity
CBT, 23 (29.9%) delivered low intensity CBT, and 6 (7.8%) delivered counselling
for depression. Most therapists were females (84.4%) from a white British
background (84.4%), with an average of 7 years’ experience in delivering
psychological interventions (range = 9 months to 31 years). The number of trial
cases treated by each therapist ranged between 1 and 113 (median = 25, mean =
30.77, SD = 24.54). Further sample characteristics are summarised in Table 1.
Altogether, 2233 patients meeting case selection criteria described above
were included in the trial (1176 feedback cases, 1057 controls). According to
clinical records, 34.5% had a primary affective disorder (major depression episode,
recurrent depression), 14.2% had mixed anxiety and depression disorder, 14.6%
had generalized anxiety disorder, 6.0% had post-traumatic stress disorder, and
other anxiety problems were less prevalent. The mean number of weekly therapy
sessions was 6.45 (SD = 3.67, median = 6, range = 2 to 25) in the full study sample;
6.35 (SD = 3.60, median = 6, range = 2 to 25) in the control group and 6.54 (SD =
3.73, median = 6, range = 2 to 22) in the OF group. Demographics and clinical
characteristics are summarised in Table 1.
The trial sample excluded 651 cases that did not access individual therapy
(e.g., group psycho-education cases) or who only attended a single session.
Excluded cases had similar baseline characteristics compared to trial cases, but a
higher proportion of unemployed patients (22.3% vs. 18.1%; p = 0.040) and
marginally higher baseline PHQ-9 scores (mean difference = 0.35; p = 0.007).
[Figure 1]
[Table 1]
Primary analysis
The main effect for trial group was not statistically significant in the initial
conditional models testing between-group differences (shown in supplementary
appendix), nor in the fully adjusted models testing interaction terms (shown in
Table 2). The negative coefficients for the group * classification interaction terms
indicated that not on track cases tended to have lower post-treatment symptoms if
they were in the feedback group, as depicted in Figure 2. The interaction was
statistically significant in the depression model (B = -1.03, SE = 0.41, p = 0.012),
and in the anxiety model (B = -0.85, SE = 0.36, p = 0.019). Approximately 11% of
variability in depression (ICC = 0.107) and anxiety (ICC = 0.114) outcomes was
attributable to therapist effects. Effect size differences between groups were PHQ-
9 d = 0.17 and GAD-7 d = 0.13 in the whole sample (N = 2233); the corresponding
values in the not on track subsample (N = 1288) were PHQ-9 d = 0.23 and GAD-7
d = 0.19. Sensitivity MLM analyses controlling for age and intensity of treatment
(low vs. high) confirmed the same results (see supplementary appendix).
[Figure 2]
[Table 2]
Secondary analyses
The fully adjusted MLM results using WSAS as a dependent variable followed the
same pattern as described above. The main effect for group was not significant (B
= 0.46, SE = 0.77, p = 0.55), but the group * classification interaction term was
statistically significant (B = -1.75, SE = 0.62, p = 0.0050) yielding an effect size of
d = 0.22 in the not on track subgroup. The poisson MLM results indicated no
significant differences in treatment duration between groups (B = -0.05, SE = 0.05,
p = 0.37); and no significant group * classification interaction (B = -0.02, SE = 0.04,
p = 0.62). Full outputs from these MLM analyses are in the supplementary
Table 3 summarises indices of clinical effectiveness. MLM results controlling
for therapist effects indicated that there were no significant between-group
differences in the odds of reliable improvement in the full sample (OR = 1.21, p =
0.29) or in the not on track subsample (OR = 1.32, p = 0.12). However, control cases
had greater odds of reliable deterioration (full sample OR = 1.48, p = 0.023; not on
track subsample OR = 1.73, p = 0.0050). There were no significant between-group
differences in the odds of treatment dropout or of being classed as not on track.
[Table 3]
Findings in context
This large-scale, multi-site trial conducted in stepped care IAPT services
demonstrated that using low-cost outcome feedback technology can improve
outcomes for cases that are at risk of poor response to treatment. No main effect
of feedback was found overall; instead an interaction effect indicated that feedback
is specifically helpful for cases classified as not on track. These findings are largely
consistent with reviews and meta-analyses of previous trials in university and
outpatient psychotherapy centres, which conclude that the effect of feedback is
mostly observed in not on track cases,4,6,8,9 although there are also exceptions such
as the trial by Amble et al.12 which found main effects for feedback in the full
sample but not in the not on track subgroup. Effect sizes of d = 0.23 for depression,
d = 0.19 for anxiety, and d = 0.22 for work and social adjustment favouring the
feedback group were observed. These effect sizes are small by conventional
standards, but nevertheless remarkable considering the automated nature of the
risk signalling technology and the low cost incurred by services in requiring
outcome feedback users to attend a single-day training session. In addition, given
that the feedback intervention prioritises clinical supervision resources for not on
track cases, it is important to highlight that this did not disadvantage the on track
cases in terms of clinical outcomes or dropout rates. Overall, this low-cost quality
assurance system effectively integrates the use of routine outcome measures,
outcome prediction technology and clinical supervision.
Given that usual treatment in IAPT stepped care services utilises standard
outcome tracking charts and regular clinical supervision, we might expect modest
effect size differences when supplementing this with risk signalling technology.
Usual care (control) cases had higher rates of deterioration compared to feedback
cases, although the odds ratios in this trial (full sample OR = 1.48; not on track
subsample OR = 1.73) were lower by comparison to the OR = 2.3 reported in the
meta-analysis by Shimokawa et al.6 This difference may be influenced by the low
base rate of cases with reliable deterioration in the participating services (<7.5%),
whereas other psychotherapy settings have typically observed deterioration rates
in the order of 10%.1 This is plausibly explained by differences in case-mix, since
IAPT services mostly support people with mild-to-moderate mental health
Contrary to recent studies applying evidence-based CBT interventions,10-11
we found no significant effects of feedback on treatment duration. One
methodological explanation could be that prior quasi-experimental studies did not
have contemporaneous controls, and their effects on duration could be explained
by other unmeasured factors that changed over time. An alternative explanation
could be that the inclusion of counselling and LiCBT interventions in the present
trial may have obscured effects that may be specific to conventional CBT. The
potential influence of feedback on treatment duration and costs requires further
Strengths and limitations
The inclusion of services across diverse regions in England is a key strength of this
study, offering compelling evidence of generalisability in contrast to earlier single-
centre pilot studies.11,30 The risk signalling technology was developed using
historical data from a service and region that did not take part in this trial,11 thus
offering a strong test of the generalisability and predictive power of the outcome
feedback model. The study was adequately powered to detect a small effect and to
control for therapist effects. The latter feature is an important advance, confirming
that the use of feedback technology improves response rates after accounting for
variations in therapeutic aptitude across multiple practitioners. It should be noted
that the therapist effect estimate (approximately 11%) in this study explains a
considerably larger proportion of variance than the effect of feedback, so attention
to the factors that characterise underperforming therapists is clearly warranted. It
is, of course, plausible that some therapists may make better use of feedback than
others, and future studies could investigate the personal attitudes, skills or
organisational conditions that optimise adequate use of feedback.31-33
Some limitations should also be borne in mind when interpreting the
present results. Although we included a sizeable group of therapists delivering a
range of low and high intensity interventions, our study participants nevertheless
volunteered to take part in the trial. We did not have information about the total
size or professional characteristics of the workforce across all participating
services, so we cannot assume that trial therapists are necessarily representative
of the wider workforce. Furthermore, we did not have the resources to closely
monitor competence in treatment delivery or in feedback utilisation. A central
feature of this feedback model involves discussing risk signals with patients and
clinical supervisors; however, we did not have objective data to assess the extent
to which these features were adhered to. A further methodological issue relates to
potential ceiling effects. Cases with high baseline severity scores (e.g., PHQ-9 ≥ 22)
whose symptoms increased during treatment could not be classified as showing
reliable deterioration, which is mostly an artefact of the measurement tools and
reliable change indices used in the study. It is therefore possible that the true
extent of reliable deterioration rates could be underestimated. In addition, like
most other feedback studies conducted to date,8 this trial only had a short-term
observation period since outcomes were assessed at the end of treatment. It is
therefore unknown if the observed effects of feedback may have a durable impact
on longer-term symptoms and functioning.
We found generalisable evidence that supplementing psychological therapy with a
low-cost quality assurance system using outcome feedback technology helps to
prevent deterioration in cases that are particularly prone to poor treatment
The outcome feedback and signalling technology used in this study was developed
by PCMIS at the Department of Health Sciences, University of York
1. Lambert MJ. The efficacy and effectiveness of psychotherapy. In M. J. Lambert
(Ed.), Bergin and Garfield's handbook of psychotherapy and behavior change
(6th ed.). Wiley & Sons: New York, 2013.
2. Hansen NB, Lambert MJ, Forman EM. The psychotherapy doseresponse effect
and its implications for treatment delivery services. Clin Psychol-Sci Pr 2002;
3. NHS Digital. Psychological Therapies: Annual Report on the use of IAPT services,
England, 2015-16. Health and Social Care Information Centre, 2016. Retrieved
4. Carlier IV, Meuldijk D, Van Vliet IM, Van Fenema E, Van der Wee NJ, Zitman
FG. Routine outcome monitoring and feedback on physical or mental health
status: evidence and theory. J Eval Clin Pract 2012; 18:104-10.
5. Finch AE, Lambert MJ, Schaalje BG. Psychotherapy quality control: The
statistical generation of expected recovery curves for integration into an early
warning system. Clin Psychol Psychother 2001; 8:231-42.
6. Shimokawa K, Lambert MJ, Smart DW. Enhancing treatment outcome of
patients at risk of treatment failure: Meta-analytic and mega-analytic review
of a psychotherapy quality assurance system. J Consult Clin Psychol 2010;
7. Castonguay LG, Barkham M, Lutz W, McAleavey AA. Practice-oriented
research: approaches and applications. In M. J. Lambert (Ed.), Bergin and
Garfield's handbook of psychotherapy and behavior change (6th ed.). New York,
Wiley & Sons, 2013.
8. Knaup C, Koesters M, Schoefer D, Becker T, Puschner B. Effect of feedback of
treatment outcome in specialist mental healthcare: meta-analysis. Br J
Psychiatry 2009; 195:15-22.
9. Kendrick T, ElGohary M, Stuart B, Gilbody S, Churchill R, Aiken L,
Bhattacharya A, Gimson A, Brütt AL, de Jong K, Moore M. Routine use of
patient reported outcome measures (PROMs) for improving treatment of
common mental health disorders in adults. Cochrane Libr 2016; 7:CD011119.
10. Janse PD, De Jong K, Van Dijk MK, Hutschemaekers GJ, Verbraak MJ.
Improving the efficiency of cognitive-behavioural therapy by using formal client
feedback. Psychother Res 2017; 27:525-38.
11. Delgadillo J, Overend K, Lucock M, Groom M, Kirby N, McMillan D, Gilbody
S, Lutz W, Rubel JA, de Jong K. Improving the efficiency of psychological
treatment using outcome feedback technology. Behav Res Ther 2017; 99:89-
12. Amble I, Gude T, Stubdal S, Andersen BJ, Wampold BE. The effect of
implementing the Outcome Questionnaire-45.2 feedback system in Norway: A
multisite randomized clinical trial in a naturalistic setting. Psychother Res
2015; 25:669-77.
13. de Jong K, Segaar J, Ingenhoven T, van Busschbach J, Timman R. Adverse
Effects of Outcome Monitoring Feedback in Patients With Personality
Disorders: A Randomized Controlled Trial in Day Treatment and Inpatient
Settings. J Pers Disord 2017. DOI: 10.1521/pedi_2017_31_297
14. Lutz W, De Jong K, & Rubel J. (2015). Patient-focused and feedback research
in psychotherapy: Where are we and where do we want to go? Psychother Res
2015; 25: 625-632.
15. Baldwin SA, Imel ZE. Therapist Effects: Findings and Methods. In M. J.
Lambert (Ed.), Bergin and Garfield's handbook of psychotherapy and behavior
change (6th ed.). Wiley & Sons: New York, 2013.
16. Spybrook J, Bloom H, Congdon R, Hill C, Liu X, Martinez A, Raudenbush S.
Optimal Design Software for Multi-level and Longitudinal Research (Version
3.01) [Software], 2013. Available from
17. Schiefele AK, Lutz W, Barkham M, Rubel J, Böhnke J, Delgadillo J, Kopta M,
Schulte D, Saxon D, Nielsen SL, Lambert MJ. Reliability of therapist effects in
practice-based psychotherapy research: A guide for the planning of future
studies. Adm Policy Ment Health 2017; 44: 598-613.
18. Clark DM. Implementing NICE guidelines for the psychological treatment of
depression and anxiety disorders: the IAPT experience. Int Rev Psychiatry
2011; 23:318-27.
19. National Institute for Health and Care Excellence. Common mental health
disorders: Identification and pathways to care. [CG123]. London, National
Institute for Health and Care Excellence, 2011. Retrieved from
20. Lutz W, Martinovich Z, & Howard KI. Patient profiling: An application of
random coefficient regression models to depicting the response of a patient to
outpatient psychotherapy. J Consult Clin Psychol 1999, 67: 571-577.
21. Lutz W, Martinovich Z, Howard KI, & Leon SC. Outcome management,
expected treatment response and severity adjusted provider profiling in
outpatient psychotherapy. J Clin Psychol 2002, 58: 1291-1304.
22. Delgadillo J, Moreea O, Lutz W. Different people respond differently to
therapy: A demonstration using patient profiling and risk stratification. Behav
Res Ther 2016; 79:15-22.
23. Delgadillo J, Lucock M, de Jong K. Using outcome feedback in psychological
therapy: A guideline for IAPT practitioners. Department of Health Sciences,
University of York: York, 2015.
24. Kroenke K, Spitzer RL, & Williams JB. The PHQ-9: Validity of a brief
depression severity measure. J Gen Intern Med 2001; 16:606-613.
25. Richards DA, Borglin G. Implementation of psychological therapies for anxiety
and depression in routine practice: two year prospective cohort study. J Affect
Disord 2011; 133:51-60.
26. Kroenke K, Spitzer RL, Williams JBW, Monahan PO, & Löwe B. Anxiety
disorders in primary care: Prevalence, impairment, comorbidity, and detection.
Ann Intern Med 2007; 146:317325.
27. Mundt JC, Marks IM, Shear MK, Greist JM. The Work and Social Adjustment
Scale: a simple measure of impairment in functioning. Br J Psychiatry 2002;
28. Schafer JL, Olsen MK. Multiple imputation for multivariate missing-data
problems: A data analyst's perspective. Multivariate Behav Res 1998; 33:545-
29. Raudenbush, S. W. Hierarchical linear models and experimental design. In L.
Edwards (Ed.), Applied analysis of variance in behavioral science (pp. 459-496).
Marcel Dekker: New York, 1993.
30. Lucock M, Halstead J, Leach C, Barkham M, Tucker S, Randal C, Middleton
J, Khan W, Catlow H, Waters E, Saxon D. A mixed-method investigation of
patient monitoring and enhanced feedback in routine practice: Barriers and
facilitators. Psychother Res 2015; 25:633-46.
31. de Jong K, van Sluis P, Nugter MA, Heiser WJ, Spinhoven P. Understanding
the differential impact of outcome monitoring: Therapist variables that
moderate feedback effects in a randomized clinical trial. Psychother Res 2012;
22: 464-74.
32. de Jong K, de Goede M. Why do some therapists not deal with outcome
monitoring feedback? A feasibility study on the effect of regulatory focus and
personorganization fit on attitude and outcome. Psychother Res 2015; 25:
33. Lutz W, Zimmermann D, Müller VN, Deisenhofer AK, Rubel JA. Randomized
controlled trial to evaluate the effects of personalized prediction and adaptation
tools on treatment outcome in outpatient psychotherapy: study protocol. BMC
Psychiatry 2017; 17: 306.
Figure 1. CONSORT diagram
Full sample
OF group
Control group
N = 77
N = 39
N = 38
65 (84.4%)
30 (76.9%)
35 (92.1%)
Mean age (SD)
40.81 (11.13)
40.26 (11.29)
41.37 (11.10)
White British
65 (84.4%)
32 (82.1%)
33 (86.8%)
12 (15.6%)
7 (17.9%)
5 (13.2%)
Mean years of experience (SD)
7.42 (5.79)
7.46 (5.88)
7.38 (5.77)
54 (70.1%)
27 (69.2%)
27 (71.1%)
23 (29.9%)
12 (30.8%)
11 (28.9%)
N = 2233
N = 1176
N = 1057
1465 (65.7%)
751 (63.9%)
714 (67.7%)
Mean age (SD)
39.22 (15.02)
38.40 (14.66)
40.14 (15.38)
286 (18.1%)
164 (20.3%)
122 (15.7%)
White British
1824 (88.5%)
979 (89.1%)
845 (87.8%)
237 (11.5%)
120 (10.9%)
117 (12.2%)
Clinical characteristics
Affective disorder
771 (34.5%)
413 (35.1%)
358 (33.9%)
Mixed anxiety and depression
316 (14.2%)
154 (13.1%)
162 (15.3%)
Generalized anxiety disorder
326 (14.6%)
170 (14.5%)
156 (14.8%)
Other diagnosis
820 (36.7%)
439 (37.3%)
381 (36.0%)
Baseline PHQ-9 mean (SD)
15.29 (6.20)
14.96 (5.96)
15.65 (6.43)
Baseline GAD-7 mean (SD)
13.99 (4.93)
13.82 (4.78)
14.19 (5.09)
Baseline WSAS mean (SD)
19.29 (9.40)
19.08 (9.22)
19.52 (9.57)
Mean treatment sessions (SD)
6.45 (3.67)
6.54 (3.73)
6.35 (3.60)
OF = outcome feedback; HIT = high intensity therapy; LIT = low intensity therapy; PHQ-9 = measure of depressions
symptoms; GAD-7 = measure of anxiety symptoms; WSAS = work and social adjustment scale; * percentages are
calculated using cases with available data, some cases with missing demographic data were excluded
Depression (PHQ-9) model
Anxiety (GAD-7) model
Fixed effects
Fixed effects
95% CI
95% CI
6.25, 7.63
5.42, 6.70
Sessions (Log)
-10.38, -8.63
-9.65, -8.07
Baseline severity (mc)
0.51, 0.57
0.43, 0.51
-0.76, 1.15
-0.57, 1.20
5.05, 6.24
4.65, 5.71
Group * Classification
-1.84, -0.23
-1.56, -0.14
Variance components
(ICC = 0.107)
Variance components
(ICC = 0.114)
Random intercept
Sessions: log-linear transformation for number of treatment sessions; Baseline severity (mc): mean centred values for PHQ-9 in the depression model, or
GAD-7 in the anxiety model; Group: 0 = controls, 1 = Outcome Feedback cases; Classification: 0 = cases classified as “on track”, 1 = cases classified as “not
on track”; note that there were two symptom-specific classifications, one for PHQ-9 and one for GAD-7; Group * Classification: this interaction term is the
main hypothesis test; B: regression coefficient; SE: standard error; CI: confidence intervals; ICC: intracluster correlation coefficient
Full sample
N = 2233
NOT subsample
N = 1288
OF cases
N = 1176
N = 1057
OF cases
N = 678
N = 610
Clinical effectiveness
PHQ-9 pre-treatment mean (SD)
14.41 (5.96)
14.85 (6.46)
14.47 (5.80)
15.45 (6.30)
PHQ-9 post-treatment mean (SD)
8.61 (6.60)
9.75 (7.12)
10.89 (7.17)
12.53 (7.37)
PHQ-9 Cohen’s d
GAD-7 pre-treatment mean (SD)
13.42 (4.85)
13.54 (5.24)
13.82 (4.77)
14.25 (5.00)
GAD-7 post-treatment mean (SD)
7.96 (5.78)
8.76 (6.12)
10.06 (6.12)
11.26 (6.37)
GAD-7 Cohen’s d
WSAS pre-treatment mean (SD)
19.58 (8.67)
19.88 (9.12)
20.29 (8.70)
21.03 (8.91)
WSAS post-treatment mean (SD)
12.65 (9.57)
14.11 (9.98)
15.54 (10.23)
17.72 (9.95)
WSAS Cohen’s d
Reliable improvement
N = 796 (67.7%)
N = 630 (59.6%)
N = 412 (60.8%)
N = 317 (52.0%)
OR (95% CI)
1.21ns (0.85, 1.71)
1.32ns (0.93, 1.89)
Reliable deterioration
N = 49 (4.2%)
N = 76 (7.2%)
N = 44 (6.5%)
N = 68 (11.1%)
OR (95% CI)
1.48* (1.06, 2.07)
1.73** (1.18, 2.54)
N = 284 (24.1%)
N = 253 (23.9%)
N = 167 (24.6%)
N = 151 (24.8%)
OR (95% CI)
1.00ns (0.70, 1.43)
1.03ns (0.71, 1.50)
Classed as NOT
N = 678 (57.7%)
N = 610 (57.7%)
OR (95% CI)
1.07ns (0.86, 1.32)
Notes: NOT = cases classified as “not on track” during therapy; PHQ-9 = depression measure; GAD-7 = anxiety measure; WSAS = work and social adjustment
measure; SD = standard deviation; Cohen’s d = post-treatment effect size difference between groups; OR = odds ratio, adjusting for baseline severity; * p <
0.05; ** p < 0.01; ns = not statistically significant
... positive outcomes (de Jong et al., 2021;Delgadillo et al., 2018;Lambert, 2007). Studies that have employed these methods with psychological treatment data include analyses of symptoms of depression (Lutz, Stulz, & Köck, 2009), panic disorder (Lutz et al., 2014) and post-traumatic stress disorder (PTSD; Dewar, Paradis, & Fortin, 2020), which each reported between three and five trajectory classes and some associations with baseline variables such as comorbid anxiety and depression symptoms. ...
... To be implemented clinically, these findings need to be extended into a predictive tool that combines baseline variables and outputs a patient's most likely symptom trajectory. Once validated in an independent dataset, this could be used to inform treatment plans at the start of therapy and also monitor whether patients are 'on-track' according to their predicted trajectory, which has been shown to improve patient outcomes (Delgadillo et al., 2018). This would be especially relevant for patients in the gradual improvement class who on average did not show reliable improvement until session six or seven. ...
Full-text available
Background: There is substantial variation in patient symptoms following psychological therapy for depression and anxiety. However, reliance on endpoint outcomes ignores additional interindividual variation during therapy. Knowing a patient's likely symptom trajectories could guide clinical decisions. We aimed to identify latent classes of patients with similar symptom trajectories over the course of psychological therapy and explore associations between baseline variables and trajectory class. Methods: Patients received high-intensity psychological treatment for common mental health problems at National Health Service Improving Access to Psychological Therapies services in South London (N = 16 258). To identify trajectories, we performed growth mixture modelling of depression and anxiety symptoms over 11 sessions. We then ran multinomial regressions to identify baseline variables associated with trajectory class membership. Results: Trajectories of depression and anxiety symptoms were highly similar and best modelled by four classes. Three classes started with moderate-severe symptoms and showed (1) no change, (2) gradual improvement, and (3) fast improvement. A final class (4) showed initially mild symptoms and minimal improvement. Within the moderate-severe baseline symptom classes, patients in the two showing improvement as opposed to no change tended not to be prescribed psychotropic medication or report a disability and were in employment. Patients showing fast improvement additionally reported lower baseline functional impairment on average. Conclusions: Multiple trajectory classes of depression and anxiety symptoms were associated with baseline characteristics. Identifying the most likely trajectory for a patient at the start of treatment could inform decisions about the suitability and continuation of therapy, ultimately improving patient outcomes.
... Mechanisms that may improve care effectiveness include evidence-based practice [8], measurement-based care [9][10][11] and person-centred care [12,13]. Evidencebased practice involves the integration of the best available research evidence with clinical expertise and service user values to inform clinical and policy decision-making [14]. ...
Full-text available
Evidence-based and person-centred care requires the measurement of treatment outcomes that matter to youth and mental health practitioners. Priorities, however, may vary not just between but also within stakeholder groups. This study used Q-methodology to explore differences in outcome priorities among mental health practitioners from two countries in relation to youth depression. Practitioners from the United Kingdom (UK) ( n = 27) and Chile ( n = 15) sorted 35 outcome descriptions by importance and completed brief semi-structured interviews about their sorting rationale. By-person principal component analysis (PCA) served to identify distinct priority profiles within each country sample; second-order PCA examined whether these profiles could be further reduced into cross-cultural “super profiles”. We identified three UK outcome priority profiles ( Reduced symptoms and enhanced well-being; improved individual coping and self-management; improved family coping and support ), and two Chilean profiles ( Strengthened identity and enhanced insight; symptom reduction and self-management ). These could be further reduced into two cross-cultural super profiles: one prioritized outcomes related to reduced depressive symptoms and enhanced well-being; the other prioritized outcomes related to improved resilience resources within youth and families. A practitioner focus on symptom reduction aligns with a long-standing focus on symptomatic change in youth depression treatment studies, and with recent measurement recommendations. Less data and guidance are available to those practitioners who prioritize resilience outcomes. To raise the chances that such practitioners will engage in evidence-based practice and measurement-based care, measurement guidance for a broader set of outcomes may be needed.
... They will then rate each goal as part of the routine outcome measures for each session. Goal-based outcomes have been shown to improve treatment retention, clinical outcomes and client progress [62,63]. ...
Full-text available
Loneliness is a significant problem for young people and is associated with a range of physical and mental health difficulties. Meta-analyses have identified that interventions aimed at young people who report loneliness as their primary problem are lacking within the literature. In adults, the most effective interventions for loneliness are those which target the underlying maladaptive social cognitions. Therefore, we have developed a modular Cognitive Behavioural Therapy (CBT) intervention for children and young people. The aim of this study is to conduct a multiple baseline single-case experimental design (SCED) to assess the efficacy, feasibility and acceptability of this intervention. In total 6–8 11–18-year-olds and their families will be recruited. The design consists of AB+ post-intervention, where A is the baseline phase, B is the intervention phase and then a post-intervention phase. Participants will complete a baseline assessment, before being randomised to one of four different baseline lengths (12 days, 19 days, 26 days or 33 days). Participants will then complete an average of 12 sessions of CBT, with the aim being to reduce their feelings of loneliness. Participants will then complete a 12-day post-intervention phase. Participant loneliness will be repeatedly assessed throughout the three phases of the intervention using the Three-item Loneliness Scale, which will be the primary outcome. Secondary outcomes will be reliable and clinically meaningful change on the UCLA Loneliness Scale, Revised Child Anxiety and Depression Scale (RCADS) and Strengths and Difficulties Questionnaire (SDQ). Feasibility and participant satisfaction will also be assessed and reported. Trial registration: trial registration number: NCT05149963 (Date registered: 07.12.2021). .
... Beyond CF, it has been shown that practising data-informed psychotherapy with technological augmentation tools (i.e., CDSS, ROM and feedback systems, CSTs) contributes to better treatment outcomes (Cohen et al., 2021;de Jong et al., 2021;Delgadillo et al., 2018). For instance, in a prospective evaluation of the TTN, Lutz et al. (2021b) found that therapists who followed the recommended treatment strategies in the first 10 sessions obtained a differential effect size of Cohen's d = .30 ...
Full-text available
The current article presents a way of practising case formulation using technology augmentation in the context of data-informed psychotherapy. First, we explain how case formulation guides the decision-making processes in psychotherapy and how decisions made under an intuitive clinical judgement can be biased. The use of actuarial methods is pointed out as a way of addressing these biases, mainly through clinical decision support systems based on statistical tools and machine learning algorithms. We present the Trier Treatment Navigator (TTN), a clinical decision support system developed in a University research-based clinical training programme. We show how the TTN can contribute to an initial case formulation and its dynamic adaptations during treatment with a clinical case. Finally, we discuss how case formulation and data-informed psychotherapy are aimed at the same goal: treatment personalisation. We argue that case formulation and data-informed psychotherapy enrich and feedback on each other.
This article is an introduction to this special issue dedicated to promoting the building of bridges between research and clinical practice. The purpose of this compendium is to delve into the relevance and possibilities that the Practice-Based Evidence (PBE) paradigm offers, which includes establishing a two-way dialogue between clinical practice and research, through the studies carried out by researchers from different countries and with different backgrounds, their approaches to the construction of knowledge from the ‘bottom up’ and the application of their findings to routine practice. PBE seeks to develop research whose findings are relevant for practitioners in routine clinical settings, from dissimilar countries and cultural contexts, where the central interest is improving the quality of treatment provided to users of mental health services.
An implementation fidelity study evaluated the quality of the therapeutic alliance (TA) in the counseling services of a Canadian Employee Assistance Program (EAP). The aims were to evaluate the level of TA experienced by EAP users during counseling, to assess the influence of client gender, and to determine the associations between the level of TA and mental health and work functioning outcomes. The TA was assessed with the Brief Therapeutic Alliance Scale (BTAS-5), a self-report quantitative measure developed to rapidly assess multiple elements of the client-therapist alliance from the perspective of the client. The majority of the EAP users (N = 1277) reported experiencing a high level of TA during counseling and men and women reported equal levels. Pretest and posttest analysis with ANCOVA showed that EAP users (N = 505) reporting higher levels of TA had greater mental well-being, lower depression symptoms (Patient Health Questionnaire-2), lower work presenteeism (Workplace Outcome Suite-5; WOS-5), and greater life satisfaction (WOS-5) at follow-up after end of counseling. The TA from the perspective of EAP users can serve as an indicator of well-implemented employee counseling services and its assessment can provide additional evidence of counseling quality and effectiveness, supplementing the findings of traditional outcomes-based EAP studies.
This third edition provides a thorough real-world exploration of the scientist-practitioner model, enabling clinical psychology trainees to develop the core competencies required in an increasingly interdisciplinary healthcare environment. The book has been comprehensively revised to reflect shifts towards transdiagnostic practice, co-design principles, and personalized medicine, and features new chapters on low intensity psychological interventions and private practice. Fully updated for the DSM-5 and ICD-11, provides readers with a contemporary account of diagnoses. It covers practical skills such as interviewing, diagnosis, assessment, case formulation, treatment, case management, and process issues with emphasis on the question 'how would a scientist-practitioner think and act?' The book equips trainees to deliver the accountable, efficient, and effective client-centred service demanded of professionals in the modern integrated care setting by demonstrating how an evidence-base can influence every decision of a clinical psychologist. Essential reading for all those enrolled in, or contemplating, postgraduate studies in clinical psychology.
Recent AI advances in AI and machine learning (ML) promise significant transformations in the future delivery of healthcare. Despite a surge in research and development, few works have moved beyond demonstrations of technical feasibility and algorithmic performance. However, to realize many of the ambitious visions for how AI can contribute to clinical impact requires the closer design and study of AI tools or interventions within specific health and care contexts. This paper outlines our collaborative, human-centered approach to developing an AI application that predicts treatment outcomes for patients who are receiving human-supported, internet-delivered Cognitive Behavioral Therapy (iCBT) for symptoms of depression and anxiety. Intersecting the fields of HCI, AI and healthcare, we describe how we addressed the specific challenges of: (1) identifying clinically relevant AI applications ; and (2) designing AI applications for sensitive use contexts like mental health. Aiming to better assist the work practices of iCBT supporters, we share how learnings from an interview study with 15 iCBT supporters surfaced their practices and information needs, and revealed new opportunities for the use of AI. Combined with insights from the clinical literature and technical feasibility constraints, this led to the development of two clinical outcome prediction models. To clarify their potential utility for use in practice, we conducted 13 design sessions with iCBT supporters that utilized interface mock-ups to concretize the AI output and derive additional design requirements. Our findings demonstrate how design choices can impact interpretations of the AI predictions as well as supporter motivation and sense of agency. We detail how this analysis and the design principles derived from it enabled the integration of the prediction models into a production interface. Reporting on identified risks of over-reliance on AI outputs and needs for balanced information assessment and preservation of a focus on individualized care, we discuss and reflect on what constitutes a responsible, human-centered approach to AI design in this healthcare context.
Full-text available
Introduction Integrated behavioural health, a model of care that embeds mental health services in primary care, can potentially increase access to mental healthcare. With the increase in health information technologies, remote measurement-based care (RMBC) presents an opportunity to improve support of integrated care. This scoping review will comprehensively examine what common procedures are followed when RMBC for mental health is tested in integrated care settings. Methods and analysis Based on an established six-step framework for conducting scoping reviews, we will search PubMed, Embase, PsycINFO, Cochrane, EBSCOhost and Web of Science with search terms related to ‘integrated care’ and ‘RMBC’. Articles published from 2015 onwards, in English, including an intervention that meets our definition of RMBC, and are conducted in collaboration with primary care or in a primary care setting will be included. After data extraction, we will categorise key findings along the following dimensions: (1) common delivery practices of RMBC; (2) common technologies and instruments used and (3) most common barriers and facilitators when implementing RMBC in an integrated care model. Ethics and dissemination Ethics approval is not required for this scoping review. For maximum impact, we will disseminate the findings to the scientific community (via publication in a peer-reviewed journal and at national conferences) and to the broader healthcare community. We will share findings with the broader healthcare community through our research centre’s existing stakeholder communication structures and through guidance from our multidisciplinary research team. These key stakeholder relationships will continue to guide our subsequent RMBC research following the review.
In this chapter, we provide a brief explanation of what machine learning is and explore how this approach to predictive modeling could add value to child and adolescent mental health services (CAMHS). In particular, we highlight how promising (and less promising) potential areas of application can be identified and discuss some of the general challenges of applying machine learning methods. We provide illustrative examples where they exist. We draw, where possible, from mental health services, and especially those that serve children and adolescents or have implications for common developmental or early-onset disorders. Machine learning has the potential to enhance CAMHS and particularly the effectiveness and reach of psychological therapies. However, a number of challenges remain before these possibilities are realised.
Full-text available
Aims: This study evaluated the impact of applying computerized outcome feedback (OF) technology in a stepped care psychological service offering low and high intensity therapies for depression and anxiety. Methods: A group of therapists were trained to use OF based on routine outcome monitoring using depression (PHQ-9) and anxiety (GAD-7) measures. Therapists regularly reviewed expected treatment response graphs with patients and discussed cases that were “not on track” in clinical supervision. Clinical outcomes data were collected for all patients treated by this group (N = 594), six months before (controls = 349) and six months after the OF training (OF cases = 245). Symptom reductions in PHQ-9 and GAD-7 were compared between controls and OF cases using longitudinal multilevel modelling. Treatment duration and costs were compared using MANOVA. Qualitative interviews with therapists (N = 15) and patients (N = 6) were interpreted using thematic analysis. Results: OF technology was generally acceptable and feasible to integrate in routine practice. No significant between-group differences were found in post-treatment PHQ-9 or GAD-7 measures. However, OF cases had significantly lower average duration and cost of treatment compared to controls. Conclusions: After adopting OF into their practice, this group of therapists attained similar clinical outcomes but within a shorter space of time and at a reduced average cost per treatment episode. We conclude that OF can improve the efficiency of stepped care.
Full-text available
Background Psychotherapy is successful for the majority of patients, but not for every patient. Hence, further knowledge is needed on how treatments should be adapted for those who do not profit or deteriorate. In the last years prediction tools as well as feedback interventions were part of a trend to more personalized approaches in psychotherapy. Research on psychometric prediction and feedback into ongoing treatment has the potential to enhance treatment outcomes, especially for patients with an increased risk of treatment failure or drop-out. Methods/design The research project investigates in a randomized controlled trial the effectiveness as well as moderating and mediating factors of psychometric feedback to therapists. In the intended study a total of 423 patients, who applied for a cognitive-behavioral therapy at the psychotherapy clinic of the University Trier and suffer from a depressive and/or an anxiety disorder (SCID interviews), will be included. The patients will be randomly assigned either to one therapist as well as to one of two intervention groups (CG, IG2). An additional intervention group (IG1) will be generated from an existing archival data set via propensity score matching. Patients of the control group (CG; n = 85) will be monitored concerning psychological impairment but therapists will not be provided with any feedback about the patients assessments. In both intervention groups (IG1: n = 169; IG2: n = 169) the therapists are provided with feedback about the patients self-evaluation in a computerized feedback portal. Therapists of the IG2 will additionally be provided with clinical support tools, which will be developed in this project, on the basis of existing systems. Therapists will also be provided with a personalized treatment recommendation based on similar patients (Nearest Neighbors) at the beginning of treatment. Besides the general effectiveness of feedback and the clinical support tools for negatively developing patients, further mediating and moderating variables on this feedback effect should be examined: treatment length, frequency of feedback use, therapist effects, therapist’s experience, attitude towards feedback as well as congruence of therapist’s and patient’s evaluation concerning the progress. Additional procedures will be implemented to assess treatment adherence as well as the reliability of diagnosis and to include it into the analyses. Discussion The current trial tests a comprehensive feedback system which combines precision mental health predictions with routine outcome monitoring and feedback tools in routine outpatient psychotherapy. It also adds to previous feedback research a stricter design by investigating another repeated measurement CG as well as a stricter control of treatment integrity. It also includes a structured clinical interview (SCID) and controls for comorbidity (within depression and anxiety). This study also investigates moderators (attitudes towards, use of the feedback system, diagnoses) and mediators (therapists’ awareness of negative change and treatment length) in one study. Trial registration Current Controlled Trials NCT03107845. Registered 30 March 2017.
Full-text available
Background: Routine outcome monitoring of common mental health disorders (CMHDs), using patient reported outcome measures (PROMs), has been promoted across primary care, psychological therapy and multidisciplinary mental health care settings, but is likely to be costly, given the high prevalence of CMHDs. There has been no systematic review of the use of PROMs in routine outcome monitoring of CMHDs across these three settings. Objectives: To assess the effects of routine measurement and feedback of the results of PROMs during the management of CMHDs in 1) improving the outcome of CMHDs; and 2) in changing the management of CMHDs. Search methods: We searched the Cochrane Depression Anxiety and Neurosis group specialised controlled trials register (CCDANCTR-Studies and CCDANCTR-References), the Oxford University PROMS Bibliography (2002-5), Ovid PsycINFO, Web of Science, The Cochrane Library, and International trial registries, initially to 30 May 2014, and updated to 18 May 2015. Selection criteria: We selected cluster and individually randomised controlled trials (RCTs) including participants with CMHDs aged 18 years and over, in which the results of PROMs were fed back to treating clinicians, or both clinicians and patients. We excluded RCTs in child and adolescent treatment settings, and those in which more than 10% of participants had diagnoses of eating disorders, psychoses, substance use disorders, learning disorders or dementia. Data collection and analysis: At least two authors independently identified eligible trials, assessed trial quality, and extracted data. We conducted meta-analysis across studies, pooling outcome measures which were sufficiently similar to each other to justify pooling. Main results: We included 17 studies involving 8787 participants: nine in multidisciplinary mental health care, six in psychological therapy settings, and two in primary care. Pooling of outcome data to provide a summary estimate of effect across studies was possible only for those studies using the compound Outcome Questionnaire (OQ-45) or Outcome Rating System (ORS) PROMs, which were all conducted in multidisciplinary mental health care or psychological therapy settings, because both primary care studies identified used single symptom outcome measures, which were not directly comparable to the OQ-45 or ORS.Meta-analysis of 12 studies including 3696 participants using these PROMs found no evidence of a difference in outcome in terms of symptoms, between feedback and no-feedback groups (standardised mean difference (SMD) -0.07, 95% confidence interval (CI) -0.16 to 0.01; P value = 0.10). The evidence for this comparison was graded as low quality however, as all included studies were considered at high risk of bias, in most cases due to inadequate blinding of assessors and significant attrition at follow-up.Quality of life was reported in only two studies, social functioning in one, and costs in none. Information on adverse events (thoughts of self-harm or suicide) was collected in one study, but differences between arms were not reported.It was not possible to pool data on changes in drug treatment or referrals as only two studies reported these. Meta-analysis of seven studies including 2608 participants found no evidence of a difference in management of CMHDs between feedback and no-feedback groups, in terms of the number of treatment sessions received (mean difference (MD) -0.02 sessions, 95% CI -0.42 to 0.39; P value = 0.93). However, the evidence for this comparison was also graded as low quality. Authors' conclusions: We found insufficient evidence to support the use of routine outcome monitoring using PROMs in the treatment of CMHDs, in terms of improving patient outcomes or in improving management. The findings are subject to considerable uncertainty however, due to the high risk of bias in the large majority of trials meeting the inclusion criteria, which means further research is very likely to have an important impact on the estimate of effect and is likely to change the estimate. More research of better quality is therefore required, particularly in primary care where most CMHDs are treated.Future research should address issues of blinding of assessors and attrition, and measure a range of relevant symptom outcomes, as well as possible harmful effects of monitoring, health-related quality of life, social functioning, and costs. Studies should include people treated with drugs as well as psychological therapies, and should follow them up for longer than six months.
Full-text available
This paper aims to provide researchers with practical information on sample sizes for accurate estimations of therapist effects (TEs). The investigations are based on an integrated sample of 48,648 patients treated by 1800 therapists. Multilevel modeling and resampling were used to realize varying sample size conditions to generate empirical estimates of TEs. Sample size tables, including varying sample size conditions, were constructed and study examples given. This study gives an insight into the potential size of the TE and provides researchers with a practical guide to aid the planning of future studies in this field.
Full-text available
Objective: Feedback from clients on their view of progress and the therapeutic relationship can improve effectiveness and efficiency of psychological treatments in general. However, what the added value is of client feedback specifically within cognitive-behavioural therapy (CBT), is not known. Therefore, the extent to which the outcome of CBT can be improved is investigated by providing feedback from clients to therapists using the Outcome Rating Scale (ORS) and Session Rating Scale (SRS). Method: Outpatients (n = 1006) of a Dutch mental health organization either participated in the "treatment as usual" (TAU) condition, or in Feedback condition of the study. Clients were invited to fill in the ORS and SRS and in the Feedback condition therapists were asked to frequently discuss client feedback. Results: Outcome on the SCL-90 was only improved specifically with mood disorders in the Feedback condition. Also, in the Feedback condition, in terms of process, the total number of required treatment sessions was on average two sessions fewer. Conclusion: Frequently asking feedback from clients using the ORS/SRS does not necessarily result in a better treatment outcome in CBT. However, for an equal treatment outcome significantly fewer sessions are needed within the Feedback condition, thus improving efficiency of CBT.
Full-text available
Objective: To investigate the barriers and facilitators of an effective implementation of an outcome monitoring and feedback system in a UK National Health Service psychological therapy service. Method: An outcome monitoring system was introduced in two services. Enhanced feedback was given to therapists after session 4. Qualitative and quantitative methods were used, including questionnaires for therapists and patients. Thematic analysis was carried out on written and verbal feedback from therapists. Analysis of patient outcomes for 202 episodes of therapy was compared with benchmark data of 136 episodes of therapy for which feedback was not given to therapists. Results: Themes influencing the feasibility and acceptability of the feedback system were the extent to which therapists integrated the measures and feedback into the therapy, availability of administrative support, information technology, and complexity of the service. There were low levels of therapist actions resulting from the feedback, including discussing the feedback in supervision and with patients. Conclusions: The findings support the feasibility and acceptability of setting up a routine system in a complex service, but a number of challenges and barriers have to be overcome and therapist differences are apparent. More research on implementation and effectiveness is needed in diverse clinical settings.
Full-text available
In the last 15 years feedback interventions have had a significant impact on the field of psychotherapy research and have demonstrated their potential to enhance treatment outcomes, especially for patients with an increased risk of treatment failure. This article serves as an introduction to the special issue on "Patient-focused and feedback research in psychotherapy: Where are we and where do we want to go?" Current investigations on feedback research are concerned with potential moderators and mediators of these effects, as well as the design and the implementation of feedback into routine care. This introduction summarizes the current state of feedback research and provides an overview of the three main research topics in this issue: (1) How to implement feedback systems into routine practice and how do therapist and patient attitudes influence its effects?, (2) How to design feedback reports and decision support tools?, and (3) What are the reasons for patients to become at risk of treatment failure and how should therapists intervene with these patients? We believe that the studies included in this special issue reflect the current state of feedback research and provide promising pathways for future endeavors that will enhance our understanding of feedback effects.
This study aimed to evaluate the differential effect of outcome monitoring feedback to therapists and to patients on outcomes in cluster B, cluster C, and personality disorder not otherwise specified (PD-NOS) patients. Day treatment patients (n = 112) and inpatients (n = 94) were randomly assigned to a feedback to therapist (FbT), feedback to therapist and patient (FbTP), or no feedback (NFb) condition. Feedback was based on weekly administrations of the Outcome Questionnaire-45 (OQ-45). In cluster B and PD-NOS patients FbTP resulted in increased OQ-45 scores during the first 6 months of therapy for not on track (NOT) patients. In cluster C patients, no adverse effects of feedback were found. These results suggest that for certain personality disorder patient groups, providing feedback during treatment may not always be beneficial, although more research is needed to further assess these effects.
Background: This study aimed to identify patient characteristics associated with poor outcomes in psychological therapy, and to develop a patient profiling method. Method: Clinical assessment data for 1347 outpatients was analysed. Final treatment outcome was based on reliable and clinically significant improvement (RCSI) in depression (PHQ-9) and anxiety (GAD-7) measures. Thirteen patient characteristics were explored as potential outcome predictors using logistic regression in a cross-validation design. Results: Disability, employment status, age, functional impairment, baseline depression and outcome expectancy predicted post-treatment RCSI. Regression coefficients for these factors were used to derive a weighting scheme called Leeds Risk Index (LRI), used to assign risk scores to individual cases. After stratifying cases into three levels of LRI scores, we found significant differences in RCSI and treatment completion rates. Furthermore, LRI scores were significantly correlated with the proportion of treatment sessions classified as ‘not on track’. Conclusions: The LRI tool can identify cases at risk of poor progress to inform personalized treatment recommendations for low and high intensity interventions.