Development and initial evaluation of blended cognitive behavioural
treatment for major depression in routine specialized mental health care
Lisa C. Kooistra
, Jeroen Ruwaard
, Jenneke E. Wiersma
, Patricia van Oppen
, Rosalie van der Vaart
Julia E.W.C. van Gemert-Pijnen
Faculty of Behavioural and Movement Sciences, Department of Clinical-, Neuro- and Developmental Psychology, VU University Amsterdam, Van der Boechorststraat 1, BT 1081 Amsterdam, The
EMGO Institute for Health Care and Research, VU University Medical Centre, Van der Boechorststraat 1, BT 1081 Amsterdam, The Netherlands
Department of Psychiatry, GGZ inGeest and VU University Medical Centre, P.O. Box 7057, Amsterdam MB 1007, the Netherlands
Health, Medical and Neuropsychology Unit, Leiden University, Wassenaarseweg 52, AK 2333, Leiden, The Netherlands
Department of Psychology, Health & Technology, University of Twente, Drienerlolaan 5, NB 7522, Enschede, The Netherlands
Faculty of Health Sciences, the Institute of Clinical Research /Telepsychiatric Centre, Mental Health Services in the Region of Southern Denmark, University of Southern Denmark, Winsløwparken
19, DK-5000 Odense, Denmark
Received 4 August 2015
Received in revised form 24 December 2015
Accepted 25 January 2016
Available online 27 January 2016
Background: Blended care combines face-to-face treatment with web-based components in mental health care set-
tings. Blended treatment could potentially improve active patient participation, by letting patients work though part
of the protocol autonomously. Further, blended treatment might lower the costs of mental health care, by reducing
treatment duration and/or therapist contact. However, knowledge on blended care for depression is still limited.
Objectives: To develop a blended cognitive behavioural treatment (bCBT) for depressed patients in an outpatient
specialized mental health care centre and to conduct a preliminary evaluation of this bCBT protocol.
Method: A bCBT protocol was developed, taking recommendations into account from depressed patients (n = 3)
and therapists and experts in the ﬁeld of e-health (n = 18). Next, an initial evaluation of integrated high-
intensive bCBT was conducted with depressed patients (n = 9) in specialized mental health care. Patients' clinical
proﬁles were established based on pre-treatment diagnostic information and patient self-reports on clinical mea-
sures. Patient treatment adherence rates were explored, together with patient ratings of credibility and expectancy
(CEQ) before treatment, and system usability (SUS) and treatment satisfaction after treatment (CSQ-8). During and
after treatment, the blended treatment protocol was evaluated in supervision sessions with the participating ther-
apists (n = 7).
Results: Seven out of nine patients started bCBT, of whom ﬁve completed ≥ 90% of treatment. System usability was
evaluated as being above average (range 63 to 85), and patients were mostly to very satisﬁed with bCBT (range 16 to
32). Patients reported improvements in depression, health-related quality of life and anxiety. We observed that
therapists evaluated the highly structured blended treatment as a helpful tool in providing evidence-based treat-
ment to this complex patient group.
Discussion: Although no conclusions can be drawn based on the current study, our observations suggest that a blend-
ed CBT approach might shorten treatment duration and has the potential to be a valuable treatment option for pa-
tients with severe depression in specialized mental health care settings. Further exploration of the effectiveness of
our bCBT protocol by means of a randomized controlled trial is warranted.
© 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
Blended cognitive behavioural therapy
Outpatient specialized mental health care
Cognitive behavioural treatment (CBT) for depression has been
studied extensively and has proved to be a clinically effective
psychotherapy (Butler et al., 2006; Cuijpers et al., 2013a, b). More re-
cently, studies have shown that CBT for depression can be effectively ad-
ministered in web-base d settings (Andersson and Cuijpers, 2009;
Andrews et al., 2010; Kelders et a l., 2015; Richards and Richardson,
2012). Furthermore, web-based treatment appears to be acceptable to
both patients and therapists (Andrews and Williams, 2014; Becker
and Jensen-Doss, 2013).
Although most studies focussed on patients with mild to moderate
symptoms (Richards and Richardson, 2012), recent studies also show
Internet Interventions 4 (2016) 61–71
⁎ Corresponding author at: Faculty of Behavioural and Movement Sciences, Department
of Clinical-, Neuro- and Developmental Psychology, VU University Amsterdam, Van der
Boechorststraat 1, BT 1081 Amsterdam, The Netherlands.
E-mail address: email@example.com (L.C. Kooistra).
2214-7829/© 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Contents lists available at ScienceDirect
journal homepage: www.elsevier.com/locate/invent
promising treatment effects and acceptability for patients with more
severe symptoms (Andrews and Williams, 2014; Hedman et al., 2014;
Ruwaard et al., 2012; Williams and Andrews, 2013). Furthermore,
internet interventions guided by a professional have been shown
to have similar treatment effects to face -to-face treatment
(Andersson et al., 2014), although the number of studies that exam-
ined the relative efﬁcacy of face-to-fa ce versus online psychotherapy
An important potential beneﬁt of web-based treatment is that it can
facilitate the delivery of evidence-based treatment protocols, such as
CBT (Andrews and Williams, 2014). Research sugge sts that only a
limited amount of patients in routine practise actually receives
evidence-ba sed treatment (Gyani et al., 2014; Harvey and Gumport,
2015). This is caused both by under-treatment of mental disorders
such as depression (Demyttenaere et al., 2004; Harvey and Gumport,
2015) and therapist drift from evidence-based treatment protoc ols
(Waller, 2009). By providing CBT in a web-based f ormat, therapist
adherence to evidence-based treatment protocols can potentially be
improved (Andersson, 2010; Månsso n et al., 2013), because the
online treatment environment provides all core treatment constructs
(Andr ews and Willia ms, 2014). In addition, online treatment is
believed to improve the accessibility and affordability of evidence-
based mental health care. Studies suggest that online treatments
may reduce therapist time per patient, because patients are encouraged
to work through the treatment protocol more autonomously, and ther-
apists can provide feedback online instead of during face-to-face ses-
sions at the clinic (Hedman et al., 2014; Kenter et al., 2015). This, in
turn, may lower treatment costs and allow therapists to take on more
Within the Dutch h ealth care syste m these potential beneﬁts are
highly relevant to specialized mental health care, because mental health
services in this setting focus on more complex, chronic and severe pa-
tients. Therefore, treatment costs tend to be higher compared to prima-
ry care (Spijker et al., 2013) in combination with long waiting lists due
to treatment duration and limited ﬁnancial resources (Bower and
Gilbody, 2005; Lovell and Richards, 2000).
Despite the potential beneﬁts of online treatment, only a small
number of patients are reached with online therapies in routine practise,
particularly in specialized mental health care (Bremmer and van Es,
2013; Kenter et al., 2015). A possible reason for the relatively low
uptake in routine practise could be that end-users, such as patients
and therapists, lack knowledge about the potential costs and beneﬁts
of online treatment (Bremmer and van Es, 2013). Further, therapists
are sceptical about whether online treatment could beneﬁt treatment
outcomes compared to face-to-face treatment (Becker and Jensen-
The integration of on line treatment into routine mental health
care could potentially be stimulated by offering treatment in a blended
format (Cuijpers and Riper, 2014). This form of treatment integrates
face-to-face treatment sessions and online sessions into one treatment
protocol (Riper et al., 2013). Blended treatment aims to preserve
personal contact and the therapeutic relationship that is associated
with stand-alone face-to-face psychotherapy, while utilizing web-
based treatment to stimulate active patient participation and im-
prove the accessibility and affordability of treatmen t (
Kenter et al. ,
nother possible beneﬁt of blended treatment is that it can facilitate
increased treatment intensity, for example by adding one online session
per week alongsid e a face-to-face session. A recent met aregression
analysis (Cuijpers et al., 2013b) indicated that intensifying treatment
augments the effectiveness of face-to-face psychotherapy, with a treat-
ment intensity of two sessions per week increasing the effect size with
g = 0.45 compared to one session per week.
Although high intensity blended treatment has not yet been studied,
preliminary evidence that a blended treatment format can offer CBT
effectively was provid ed by the uncontrolled study of Månsson and
colleagues (Månsson et al., 2013), focussin g on a community-based
sample of patients with moderate anxiety or depression (n = 15).
Further, a recent Delphi study suggested that blended treatment is pos-
itively perceived by patients and therapists (Van der Vaart et al., 2014).
Other available studies focussed on combined cognit ive behav-
ioural face- to-face and online treatment for depression. The results
suggest that this combination treatment can achieve promising clin-
ical results (Hickie et al., 2010; Høifødt et al., 2013; Kenter et al.,
2013; Robertson et al., 2006). However, combining the two treat-
ment formats rather than blending them int o one treatment protocol
can also lead to increased treatm ent dosage and higher costs (Kenter
et al., 2015).
The current study expands on the aforementioned studies by devel-
oping a highly structured and integrated blended CBT (bCBT) protocol
for depressed patients in specialized mental health care. This paper de-
scribes the development of the protocol and initial experiences with
2.1. Development of blended cognitive behavioural treatment (bCBT)
Our primary objective was to develop a bCBT protocol for depression
in specialized mental health care, because, to the best of our knowledge,
such a protocol was not yet available.
2.1.1. Therapist and expert recommendations
In order to acquire input on how online and face-to-face treatment
sessions could be integrated, we consulted CBT therapists working at a
specialized mental health care centre in Amsterdam, the Netherlands,
and Dutch experts in the ﬁeld of web-based treatment (n = 18) in
four two-hour group discussions. During these meetings, we discussed
possible beneﬁts and limitations of online a nd face-to-face sessions,
and participants could express speciﬁc recommendations for the blend-
ed CBT protocol. Sessions were recorded and transcribed,
were taken during the sessions by the ﬁrst author (LK).
Next, authors one (LK), two (JR), three (JW), four (PvO) and seven
(HR) discussed the ﬁndings. Based on group consensus, the following
therapist recommendations were incorporated into the treatment
– Tre atment starts with a face-to-face session, in order to establish
a therapeutic relati onship, motivate patients for tr eatment and
explain working with the online treatment environment to
– Face-t o-face sessions and online sessions are provided in equal
measure (50%/50% ratio). Therapists expected that the proposed
ratio would enable them to provide adequate therapist support
to p atients, thus promoting treatment motivation a nd preventing
patients from dropping out of treatment.
– Face-to-face sessions focus on adapting the treatment content to
individual patient needs, for example by p ractis ing skills in rol e
plays and helping patients t o identify their core problems. Online
sessions are used to offer background information, record mood
ratings and provide homework excercises.
– The treatment is structured as a ﬁxed sequence of treatment
modules, instead of tailoring online content to individual pa tients
by al lowing therapists and p atients to choose from tr eatment
modules and/or adjust th e order of module s. This was done pri-
marily to ensure delivery of the full CBT protocol. Therapists
also noted that a ﬂexible rather than ﬁxed approach would
Due to technical problems, the audio recordings of one therapist group-session and
one expert group session were not usable.
62 L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
require them to have extensive knowledge on the content of
the protocol and experience with working with the web-based
treatment environment. This led th em to prefer working
with a ﬁxed treatment protocol duri ng the initial eval uation
– Online sessions in clude an optional, open-ended sessi on evalua -
tion question, to allow patients 1) to comment on the online ses-
sions, and 2) to promote reﬂective thinking on the meaning and
impact of the homework excersise s.
– Email remi nders are sent in ord er to encourage patients to access
the online platform and engag e in treatment.
– Online therapist feedback is provided after each online session , in
order to monitor and motiv ate patients between the face-to-face
and online sessions.
2.1.2. Patient recommendations
In order to incorporate the patient perspective, we showed the pro-
totypes of the bCBT protocol (MS Word document) and web-based
treatment delivery system (Minddistrict; www.minddistrict.com), to a
convenience sample of patients (n = 3) during a 90-min group meeting.
The patients (two males, one female) were in the ﬁnal phase of face-to-
face CBT treatment for depression at an outpatient clinic of a specialized
mental health care centre in Amsterdam, the Netherlands. The meeting
took place at the mental health care centre and was led by the third au-
thor (JW). Minutes were taken during the meeting by the ﬁrst author
(LK). Next, authors one (LK), two (JR), three (JW), four (PvO) and
seven (HR) discussed the patient recommendations. Based on group
consensus, the following elements were incorporated into the treat-
– The patients raised concerns about the amount of homework. Based
on this, online exercises are split into ‘mandatory’ and optional
exercises, to ensure that completion of exercises is feasible on ‘bad
days’, preventing unnecessary negative effects of treatment work-
– No changes were made to the web-based treatment delivery system.
– The patients have access to one new session at a time in the web-
based treatment system, instead of all sessions at once.
– At the beginning of each face-to-face session, 15 min are reserved for
patients to discuss personal issues that arose over the past week.
This can be related to the online homework, but can also incorporate
discussions of other challenges that patients faced.
– Face-to-face sessions are provided on a weekly basis, because the pa-
tients thought it was important to see their therapist regularly at the
specialized mental health care clinic. This matched the therapist
2.2. Initial evaluation of blended cognitive behavioural treatment (bCBT)
The prototype of the bCBT protocol was offered to a small group of
patients, in order to explore the reach and acceptability of the bCBT pro-
tocol. Measurements were taken at baseline (pre-treatment) and post-
treatment (10 weeks). The Medical Ethics Review Committee of the VU
University Medical Centre in Amsterdam, the Netherlands approved the
study (REF 2013/381). The study was carried out between January 2014
and May 2014.
Patients (n = 9) were recruited at an outpatient clinic of a spe-
cialized mental health care centre in Amste rdam, the Netherlands.
To participate, patients had to be 18 years or older and be diagnosed
with a current depressive episode, based on the criteria from t he
Diagnostic and Statistical Manual of Mental Disorders, Fourth
Edition, Text Revision (DSM-IV-TR, APA, 2000). Additionally,
patients needed to have adequate proﬁciency in the Dutch language,
and access to the Internet at home via a (tablet) computer. Patients
were exclude d from the study if individual and/or outpatient
CBT for d epression was not indicated, or if they were already receiv-
ing psychotherapy. Concurrent pharmacotherapy was not an
exclusion criterion. Inclusion and exclusion criteria were assessed
at the outpatient clinic during the routine face-t o-face intake
The third author (JW) approached patients after their intake assess-
ment at the outpatient clinic and informed them about the study and
bCBT. Potential participants received an information broch ure via
email. After providing written informed consent, patients ﬁlled in the
online baseline self-report questionnaires. Before the start of blended
reatment, the use of the online treatment platform was demonstrated
to the included patients during an individual 30-min face-to-face ses-
sion at the outpatient clinic. Face-to-face treatment sessions (45 min)
took place at the clinic and patients completed the online sessions at
home. After blended treatment, patients were asked to ﬁll in the online
Once every two weeks a supervision meeting was held with
the participating therapists. At the end of the study, when all patients
had completed the blended protocol, a ﬁnal evaluation meeting
was held in order to prov ide therapists with the opportunity to
evaluate the bCBT protocol. Minutes were taken during all supervision
sessions. The ﬁ nal evaluation meeting was audio-rec orded and
In order to gain insight into the feasibility (Leon et al., 2011) of bCBT,
we ﬁrst examined the reach of the intervention in terms of the
proportion of patients enrolled at the mental health care cent re that
was eligible for bCBT. Next, the clinical proﬁles and treatment outcomes
of patients that were enrolled in bCBT were explored based on patients'
electronic patient records and clinical self-report measures. Finally,
patients' treatment adherence to bCBT, and patient self-reports
on treatment acceptability, usage and satisfaction were a ssessed.
All self-report measures used in our study are considered to
have good psychometric properties and were administe red via the
22.214.171.124. Clinical proﬁles. Information on baseline functioning was extract-
ed from the electronic patient ﬁles. This information included current
DSM-IV-TR diagnosis (APA, 2000), use of anti-depressant medication
and cur rent Global Assessment of Functioning (GAF) index sc ore
(APA, 2000). Socio-demographic information such as gender, age, na-
tionality, level of education and employment status were collected
with a self-report ques tionnaire at baseline as part of the c urrent
study. In addition, clinical self-report measures were used to gain in-
sight into the clinical patient proﬁles before and im mediately after
treatment. We used questionnaires that were proposed by the mental
health care centre for routine outcome measurement (ROM) before
and immediately after treatment.
The 30-item self-report version of the Inventory of Depressive
) was used to assess the severity of depressive symp-
toms in the past week (Rush et al., 2000). Total scores range from 0 to 84,
with higher scores indicating greater severity of depressive symptoms
(Trivedi et al., 2004). The severity index ranges from 0 to 4 and is built
up as follows; IDS-SR
scores 0 to 13 = 0 (None), scores 14 to 25 =
1 (Mild), scores 26 to 38 = 2 (Moderate), scores 39 to 48 = 3 (Severe),
and scores 49 to 84 = 4 (Very severe).
The Beck Anxiety Inventory (BAI) (Beck and Steer, 1993) was used
to measure anxiety. Total scores range from 0 to 63, with higher scores
indicating a higher level of anxiety (Trivedi et al., 2004). The severity
index ranges from 0 to 3 and is built up as follows; BAI scores 0 to
63L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
9 = 0 (Normal or no anxiety), scores 10 to 18 = 1 (Mild to moderate),
scores 19 to 29 = 2 (Moderate to sever e), and scores 30 to 63 = 3
(Severe anxiety) (Aaron T Beck et al., 1988).
Furthermore, health-related quality of life was measured with the
EuroQol qu estionnaire (EQ-5D-3 L) (EuroQol Group, 1990; Lamers
et al., 2006). The questionnaire is composed of a visual analogue scale
(VAS) ranging from 0 to 100, and ﬁve items with 3 response categories
each. The combination of responses on the ﬁve items is converted into
health states based on utility weights (Lamers et al., 2006). Health
state scores range from 0 to 1, with 1 representing the best possible
state of health (Brooks, 1996).
126.96.36.199. Patient evaluation of bCBT. Treatment expectancy and credibility
were measured before the start of bCBT with the 6-item credibility/ex-
pectancy questionnaire (CEQ)(Devilly and Borkovec, 2000). Total scores
on the credibility and expectancy scales range between 3 and 27. Total
scores for the overall scale range between 6 and 54. Higher scores indi-
cate higher credibility and more positive treatment expectations
(Devilly and Borkovec, 2000).
After bCBT, pati ents rated system usability of the online treat-
ment platf orm on the 10-item System Usability Scale (SUS)
(Bangor et al., 2008; Brooke, 1996). Total SUS scores are convert-
ed to a 0 to 100 scale, wit h higher scores b eing indicati ve of
greater system usabilit y. A S US scor e above 68 is considere d
above average (Sauro, 2011), indicati ng acceptable experienced
Post-treatment satisfaction with bCBT was measured with the
Client Satisfaction Questionnaire-8 (CSQ-8) (Larsen et al., 1979). The
CSQ-8 consists of 8 items. The tota l sc ores range between 8 and 32,
with higher scores indicating better treatment satisfaction (De Brey,
Information on treatment adherence (number of completed face-
to-face sessions) was extracted from the electronic patient ﬁles. Infor-
mation on the use of the number of completed online sessions was ex-
tracted from the web-based treatment platform (Minddistrict; www.
Data on treatment adherence, credibility, usage and satisfaction and
clinical measures are presented on an individual patient level. Simple
summary statistics (M, SD) are presented to provide a clinical descrip-
tion of the included group of participants.
3.1. Blended cognitive behavioural treatment (bCBT)
Based on the information gathered in the development phase, an in-
tegrated bCBT protocol was developed by authors one (LK), two (JR),
three (JW), four (PvO) and seven (HR). Fig. 1 provides an overview of
the treatment content and sequence.
The content of the blended protocol is based on a Dutch protocol for
face-to-face CBT in specialized mental health care (Bockting and
Huibers, 2011) which recommends providing 16 to 20 weekly sessions.
This face-to-face CBT protocol is based on the protocol by Beck (Beck
et al., 1979). The blended treatment is intensiﬁed compared to standard
CBT, delivering one face-to-face session and one online session per
week for te n weeks, instead of one face-to-face per week for
20 weeks. Both face-to-face and online sessions are highly structured,
and are comprised of psycho-education, behavioural activation, cogni-
tive th erapy and relapse prevention (Spijker et al., 2013)withthe
same order and dosage for all patients.
Fig. 1. Overview of the blended treatment protocol Note: F-to-F sessions: face-to-face sessions; Online FB: online feedback.
64 L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
3.1.1. Face-to-face sessions
Treatment starts with a face-to-face session in which 1) the therapist
and patient become acquainted with each other, 2) the general aspects
of bCBT are explained, and 3) the online treatment platform is intro-
duced. Each further face-to-face session begins with a short open reﬂec-
tion on the patient's mood, experiences and homework in the past
week, before addressing that week's topic dictated by the protocol. If pa-
tients do not complete a scheduled online session prior to their visit to
the clinic, therapists guide patients through the online session during
the face-to-face session.
3.1.2. Online sessions
The online element of bCBT consists of two parts: 1) a session
that patients work through by themselves, and 2) p ersonalized
therapist f eedback on the completed homework assignments.
Nine online sessions elaborate on the them es of the face-to-f ace
sessions, and consist of psycho-educatio n ( written information,
and a short video in whi ch the information is explained in lay
terms), and homework e xercises, which are illustrated by vi-
gnettes of two ﬁctional patients. In order to facilitate the use of
the online platform, an additional online session that provides sp e-
ciﬁc information o n how to work with the plat form is available at
the start of treatment.
Patients are encouraged to access the online platform on a daily
basis, in order to keep a mood diary and complete homework exer-
cises, such as daily activity monitoring. The online sessions are
delivered through a web-based treatment platform (Minddistrict;
www.minddistrict.com), which patients and therapists access se-
curely with a personal login account. The “back of ﬁce” of the plat-
form enables professional users such as therapists and supervisors
to monitor patients and/or therapists. A messaging system enables
therapists to communicate with each other and with their patients
on the platform. When patients do not complete a scheduled online
session on time, therapis ts use the messaging system to motivate pa-
tients to complete the online session before the upcoming face-to-
3.2. Initial evaluation of bCBT
3.2.1. Recruitment and allocation of patients
Fig. 2 describes the ﬂow of patients in the study. During the two-
month recruitment period, ﬁfty-two patients with depressive symp-
toms were referred to the mental health care centre. The intake staff
at the mental health care centre indicated treatments other than indi-
vidual CBT for twenty-seven patients (52%). Most often this concerned
an indication for 1) other types of psychotherapy (n = 12) such as inter-
personal therapy (IPT), psychoanalytic treatment or cognitive behav-
ioural analysis system of psychotherapy (CBASP) or 2) inpatient or
day-treatment (n = 11).
Twenty-ﬁve patients (48%) were indicated for cognitive behav-
ioural treatment (CBT). Out of these patients, ten were excluded be-
cause the pr imary diagnosis was n ot depre ssion, and one patient had
Fig. 2. Patient Flow Diagram.
65L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
insufﬁcie nt command o f the Dutch language. F ourteen patients
(27%) t hus met inclusion criteria, of whom t hree decided not to re-
ceive treatment at the specialized mental heal t h care c ent re. There-
fore, eleven patients could be approached for study participation,
of wh om nine agreed to participate in the study and ﬁlled in the
baseline quest ionnaires.
3.2.2. Study and treatment adherence
Complete follow-up data is available from seven out of nine
patients (see Table 1). The two patient s who did not complete the
follow-up measures also did not start bCBT (patients 4 and 7).
Patient 4 chose to receive face-to-face CBT instead of bCBT, because
of a malfunctioning computer at home. Patient 7 coul d no longer
travel to the outpatient clinic for the face-to-face sessions, due to ﬁ-
nan cial problems.
Out of the seven patients who started bCBT, ﬁve completed 90%
or mor e of the blended treatment protocol, receivi ng the full face-
to-face protocol and completing seven (n = 1) or all nine ( n = 4)
online sessions. In this group the bCBT protocol was delivered
in 10 to 13 w eeks (mean = 12, SD = 1.3). Reasons for the two
other pat ients to discontinue bCBT after the start o f treatment
were a full remission of the depressive disorder due to the use of
anti-depressant medication (patient 1) and a shift in the main
focus of treatment from depr ession to attention de ﬁcit disorder
3.2.3. Patient characteristics and clinical proﬁles
Characteristics of the nine participants (ﬁve females, four males)
are d isplayed in Table 1. Six patients reported severe to very severe
depressive symptoms on the IDS-SR
and seven w ere diagnose d
with a co-morbid disorder such as anxiety, post-traumatic stress
disorder or an au tism spectrum dis order. Four patient s re ported
that t he current depressive e pisode was their ﬁ rst episode. For two
patients the current episode was their second e pisode, and the
remaining three patients reported having two or more prior
At follow-up a decrease in depression severity was reported by
seven out of eight patients. Severity of depressive s ymptoms in
this group ranged from no severity (n = 3), to mild severity (n = 2)
to mod erate severity (n = 3). Patient 7 reported an increase of two
points on de IDS-SR
compared to baseline, indicating very severe
depressive symptoms. Anxiety scores at follow-up decreased in six
out of seven patients. Severity of anxiety symptoms in this group
ranged from normal or no anxiety at follow-up (n = 2), to mild to mod-
erate symptoms (n = 2), to moderate to severe anxiety symptoms
(n = 2). Patient 3 reported having the same mild to moderate anxiety
level before and after treatment.
Health-related quality of life increa sed in four patients. Patient 5
reported the same level of health-relate d quality of life at follow-up
and patient 6 reported a decrease in health-related quality of life after
treatment, due to an increase in physical pain.
3.2.4. Patient evaluation of bCBT
188.8.131.52. Treatment credibility and expectancy. Table 1 displays pre-
treatment credibility and expectancy (CEQ) scores for all nine patients.
Total scores ranged from 31 to 42 (mean = 33.9, SD = 3.6), with six out
of nine patients reporting a neutral attitude concerning overall treat-
ment credibility and expectancy and three patients having a somewhat
to moderately positive attitude.
Patient characteristics pre-intervention and post-treatment at individual patient level and group level.
Time Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 9 Mean (SD)
Sex T0MMF MF F F MF –
Age T0 45 43 27 30 39 29 50 45 33 37.98 (8.36)
Education T0 High Mod. High Low High High Low Mod. High –
Employed T0 Y Y Y Y Y Y Y N N –
Current GAF T0 50 45 50 – 55 45 50 65 50 51.25 (6.41)
Anti-depressant use T0 Y N Y N Y Y N N N –
Co-morbid disorder T0 N Y Y Y Y Y N Y Y
# Face-to-face sessions T1 3 10 11 6 9 10 1 10 4 7.11 (3.69)
# Online sessions T1 3 9 9 0 9 9 0 7 2 5.33 (4.03)
Treatment duration (weeks) T1 3 10 13 7 13 10 1 12 3 7.92 (4.57)
Evaluation of bCBT
CEQ total T0 31 37 42 32 33 34 33 33 31 33.88 (3.61)
CEQ credibility T0 19 18 21 17 17 22 18 17 18 18.56 (1.81)
CEQ expectancy T0 12 19 21 15 16 12 15 16 13 15.33 (3.23)
SUS T1 78 68 70 – 85 75 – 63 77 73.21 (7.32)
CSQ-8 T1 22 21 23 – 32 21 – 16 24 22.71 (4.82)
T0 21 (1) 48 (3) 28 (2) 58 (4) 53 (4) 42 (3) 48 (3) 40 (3) 26 (2) 40.44 (12.87)
T1 5 (0) 24 (1) 11 (0) – 26 (2) 23 (1) 50 (4) 36 (2) 11 (0) 23.25 (14.73)
T0 10 (1) 29 (3) 10 (1) 38 (3) 43 (3) 21 (2) 22 (2) 10 (1) 15 (1) 22 (12.39)
T1 2 (0) 22 (2) 10 (1) – 19 (2) 18 (1) – 9 (0) 10 (1) 12.86 (7.03)
T0 44 50 50 19 46 40 50 33 70 44.67 (13.9)
T1 90 75 75 – 76 59 – 66 82 74.71 (10.09)
T0 0.810 0.377 0.604 0.190 0.686 0.427 0.251 0.337 0.686 0.481 (0.221)
T1 1 0.686 0.774 – 0.686 0.209 – 0.772 0.896 0.718 (0.251)
#: Number of; BAI: Beck Anxiety Inventory; CEQ: Credibility/Expectancy Questionnaire; CSQ-8: Client Satisfaction Questionnaire-8; IDS-SR: Inventory of Depressive Symptomatology, Self-
Report. Index: IDS-SR Severity index; Education level: Lower: primary school education; Mod. (Moderate): High school or vocational education; Higher: college degree or upwards; EQ-
VAS: EuroQol-5D-3 L VAS scale; EQ-5D: EuroQol-5D-3 L; GAF index: Global Assessment of Functioning. Mastery: Mastery Scale; Mod.: Moderate; Pt: Patient number;
SD: standard deviation; SUS: system usability scale. Note: given sample limitations, means and standard deviations in the last column should be interpreted as a descriptive summary of
the clinical proﬁle of the group of participants only.
66 L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
When treatment credibility and expectancy are explored separately,
the range of patients' treatment credibility scores was 17 to 22
(mean = 18.6, SD = 1.8,), with all patients rating bCBT as somewhat
(n = 7) t o moderately credible (n = 2). Treatment expectancy
ranged from 12 to 21 (mean = 15.3, SD = 3.2), ranging from
slightly nega tive (n = 3), to neutral (n = 4), to m oderately positive
(n = 2). Patients' rational e xpect ations appeared to be higher than
their emotional expectations (range 40 to 84, mean = 57.7, SD =
17.3 for thinking versus range 10 to 70, mean 46.4, SD = 19.8, for
3.2.5. System usability
At follow-up, seven patients completed the system usability scale
(SUS), evaluating syst em usability of the web-bas ed treatment plat-
form. Table 1 displays individual patient scores. Six out of seven patients
scored 68 or hig her, which is indicative of an above ave rage score
(Sauro, 2011). Total scores ranged from 63 to 85 (mean = 73.2, SD =
7.3), which can be translated into system evaluation adjectives ranging
from ‘OK’ to ‘good’ (n = 2) to ‘good’ to ‘excellent’ (n = 5) (Bangor et al.,
3.2.6. Treatment satisfaction
At follow-up, seven patients completed the Client Satisfaction
Questionnaire-8 (CSQ-8), evaluating blended treatment satisfaction.
Table 1 displays individual patient scores. The range in treatment
satisfaction scores was 16 to 32 (mean = 22.7, SD = 4.8). One
participant (patient 8) was somewhat di ssatisﬁ ed with the
content of CBT treatment, suggesting that ano ther type of psycho-
therapy (such as CBASP) might have been a better ﬁt. The other
six participants were mostly (n = 5) to very satisﬁed (n = 1) with
3.2.7. Exploration of written patient evaluations
As discusse d in the Methods secti on, patie nts could provi de a
written evaluation a t the end of each online sessio n. The open-
ended question, phrased “what was your experience with this
lesson”, was added to enable patients to 1) to comment on
(evaluate) the o nline sessions, and/or 2) t o promote r eﬂective
thinking on the meaning and impact of the homework excersises.
Relative to the number of completed online sessions per patient, pa-
tients provided 33% to 100% of possible responses (mean = 68.14,
SD = 21.43).
An explorative evaluation was performed, comparing the con-
tent of responses to the intended categories 1) session evaluation
and 2 ) r eﬂection. A su mmary of al l responses ( n = 35) can be
found in Appendix A. The majority of responses appeared to be
of a reﬂective nature (n = 25), with patients elaborating on the
emotional reaction t hey had to the session and/or evaluating
their progress. Examples of such re sponses are: “I fou nd it hard
to write everything down. I recognize a lot of myself in the text.
I want to get started in order to rega in control ” and “Because I ru-
min ate a lot, things have not been going well for me. I want to do
ings r ight, but I at the same time I do realize that just doing
things is more import ant. I became aware of the fact that I need
Patients provided information on how they evaluated the ses-
sion on six occasions. For example, two patients commented on
the way cognitive dysfunctions were illustrate d. This was done
by presenting reactions of two ﬁctional patient s to vario us scenar-
ios, such as losing your job, with one patient providing negative
interpretations and thoughts and the other patient pro viding
more positive alternative views. An example of an evaluative re-
sponse was: “I did not like the wa y the dysfunctions were present-
ed. It reminde d me of high sch ool. Nevertheless I answered all
We also noticed that patients used the open-ended question to pro-
vide information on context (n = 11), such as circum stances under
which the online session was completed: “I feel stressed because the
weekend has started and I forgot to work on this session. Yoga was
not as relaxing as I hoped and at the moment I am having two of my
friends over”, or past-week experi ences: “I went to see colleagues. It
felt good, but now it is hard to unwind again. That still is an important
theme for me”.
In addition to the responses on the open-ended question, two pa-
tients provided their therapist with an evaluation of bCBT in general.
Patient 5 stated that: “the online part of treatment really helped me a
lot because you can always access it and by d oing so you can put
everything into practise more easily”. Patient 6 had a different expe-
rience, and stated that she felt like “after e-health the ‘real treatment’
could start”, explaining further that the treatment protocol felt
restricting because it did not address the full spectrum of he r
The aim of this study was to develop an integrated high-intensive
bCBT for depression in outpatient specialized mental health care and
to conduct an initial evaluation of the treatment protocol. Our results in-
dicate that bCBT has the potential to be a suitable intervention for de-
pression in specialized mental health care.
4.1. Reach of bCBT
There was a high willingness to receive blended treatment. Out
of the eleven patients that could be approached for study partici-
pation during the two-month recruitment period, nine agreed to
receive bCB T. The difference bet ween the total number of referre d
patients and the number of patients scre ened for bCBT is largely
explained by the fac t that only a limited proportion of patients
was indicated for CBT for depression (15 out of 52; 29%). This
was mainl y due to the fact that patients were often indicate d
for other treat ments than CBT b y the intake st aff, such as IPT,
CBASP or intensive group treatment (day treatment). In order to
accommodate more pati ents within sp ecialized mental he alth
car e, it could therefore be valuable to explore the option of blend-
ing on line and face-to-face sessions for these kind s of treatment as
Despite making up a relatively small proportion of referred patients,
our patient group did appear to be representative for the patient popu-
lation that is expected in outpatient specialized mental health care,
which includes patients presenting with co-morbid disorders, moderate
to (very) severe depressive symptoms and serious impairments in gen-
eral functioning (Piek et al., 2011).
In the current study, four out of nine patients received the full
bCBT protocol and one patient completed 90% the protocol (i.e. all
10 face-to-face sessions and 7 out of 9 online sessions). We observed
slight d ifferences in the number of face-to-face sessions needed to
ensure delivery of the full content of the bCBT protocol, with one pa-
tient receiving nine face-to-face sessions, three patients receiving all
ten sessions and one patient receiving eleven sessions. Treatment
duration among com pleters ranged from 10 to 13 weeks, in dic ating
that bCBT i ndeed has the pote ntial to s horten treatment duration
compared to the minimu m of 16 to 20 weeks ne eded in face-to-
Reasons for discontinuing bCBT once treatment had started ap-
peared to be unrelated to the blended nature of treatment. Perhaps
unsurprisingly, we found that having Internet access and a function-
al computer at home are key elements for patients in order to be able
67L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
to receive bCBT. Therapists also not ed this during supervision ses-
sions, adding that ha ving access to up-to-da te hard- and sof tware
the online treatment environment can be accessed during the face-
consider the adherence rates to be promising. Future research
needs to establish to what extent these rates are representative for
the patient group in general and how they compare to standard
face-to-face CBT. For example, a recent meta-analysis found
that overall, patient s completed 84% of sessions in face-to-face CBT
and 80% of sessions in guided online CBT (VanBallegooijenetal.,
4.3. Patient evaluation of bCBT
A notable ﬁnding is that while patients' pre-treatment expectations
were mainly neutral, most patients appeared to have positive attitudes
towards bCBT after they received treatment. Patients' responses to
the open-ended question at the end the ﬁrst online session appear
to mirror this somewhat sceptical baseline attitud e, with three
patients mentioning that it was hard to start with this online session
(see Appendix A).
In future research it would be interesting to further investigate
patients' attitudes towards bCBT and to study the consequences
for treat ment adhe rence. For example, the study by Wilh elmsen
and colleagues (Wilhelmsen et al., 2013)foundthatasenseof
relatedness in terms of feeling connected to the therapis t and being
able to identify with the online CBT modules a ppears to be an impor-
tant element for patients (n = 14) to persist with bCBT in primary
4.4. Study observations
During the supervision sessions, we observed that therapists
evaluated the highly structured protocol as easy to use and to im-
plement in their daily practise. When compared to standard fac e-
to-face t herapy, therapists expressed it was con venient and
timesaving to have all homework forms and diaries available on-
line, instead of using paper-and-pencil versions. In addition, we
observed that the bCBT protocol appeared to help therapists to ad-
here to an evidence-based treatment manual, since patients have
insight in the content of treatment. Based on t his, we believe
that bCBT can potentially reduce therapis t drift from the treatment
However, due to the complex and co-morbid nature of this particu-
lar patient group, a highly structured protocol that predominantly fo-
cuses on depression will not always provide en ough trea tment for
some patients to reach remission, and continuation of CBT or a referral
to another treatment will be necessary. This can also be seen in the cur-
rent study. Nevertheless, looking at their clinical proﬁles after treat-
ment, most patients did appear to beneﬁt from b CBT. By treating
depression ﬁrst with an evidence-based treatment such as CBT, we be-
lieve better decisions can be made concerning the next steps in
4.5. Study limitations
This study should be seen as a ﬁrst step in the development and eval-
uation of intensive bCBT for depressed patients in specialized mental
health care and our observations provide some insight in the potential
use of bCBT. However no conclusions can be drawn based on the current
sample and our ﬁndings cannot be generalized beyond the included
group of patients.
Further, although the system usability measure (SUS) gives a
general indication of how patients e valuated the web-based
treatment platform, we advise future studies to examine treat-
ment satisfaction and system usability more closely, for example
by assessing actual use of all web-based treatment elements
via logﬁles (Van Gemert-Pijne n et al., 2014), or by observing
patients as they work through one or more of the online
Finally, extensive collection and evaluation of qualitative data
was beyond the scope of the current paper. Therefore, no formal
methods were used to quantify the needs and recommendations
of end-users. We would advise future studies to examine qualita-
tive evaluations more t horoughly by using fo rmal methods.
Examples of such evaluations c an be found in the studies by
Wilhelmsen et al. (2013)), Van der Vaart et al. (2014)
t al. (2015).
Our observations suggest that blending face-to-face and online CBT
sessions has the potential to be a valuable treatment option for patients
with severe depression at specialized mental health care settings. This
ﬁnding needs to be interpreted with caut ion, as more extensive re-
search is required to establish whether our initial observations can be
generalized beyond the current study.
By combining a personalized approach with the stan dardized
structure of evidence-based treatment protocols, it seems possible to re-
tain and combine the beneﬁts associated with stand-alone online and
face-to-face treatments. Further, bCBT can potentially reduce the num-
ber of face- to-fac e sessions and overall length of therapy. This could
beneﬁt accessibility of care and might lower the costs of mental health
Exploration of the effectiveness of blended depression treatment by
means of a randomized controlled trial is warranted to conﬁrm this.
Therefore, such a study is currently conducted by our group (Kooistra
et al., 2014). In addition to CBT, it might be valuable to explore blended
formats for other psychotherapies, such as IPT and CBASP, in order to
extent the reach of blended treatment.
The authors declare that they do not have competing interests.
HR (PI) and JvGP obtained funding for this study. All authors
contributed to the design of the study and LK, JE, JW, HR and PvO
contributed to development of the intervention. LK and JW coordinated
the recruitment of patients and the data collection. JW was responsible
for the supervis ion of therapists during the study. LK wrote the
manuscript. All authors read, contributed and approved the ﬁ nal
Conﬂict of interest
The authors declare that the y do not have any actual or potential
conﬂict of interest including any ﬁnancial, personal or other relation-
ships with other people or organizations within three years of begin-
ning the submitted work that could inapprop riately inﬂuence, or be
perceived to inﬂuence, their work.
This study was funded by Innovatiefonds Zorgverzekeraars
(Healthcare Insurers' Innovation Fund), project number B-12-059, dos-
sier number 2444 and ZonMw (the Netherlands Organization for Health
Research and Development), project number 837001007.
68 L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
Appendix A. Patients' written responses to the online sessions.
American Psychiatric Association, 2000. Diagnostic and Statistical Manual of Mental Disor-
ders, Fourth Edition, Text Revision.Author,Washington,DC.
Andersson, G., 2010. The promise and pitfalls of the internet for cognitive behavioral ther-
apy. BMC Med. 8 (1), 82. http://dx.doi.org/10.1186/1741-7015-8-82.
Andersson, G., Cuijpers, P., 2009. Internet-based and other computerized psychological
trea tments for adult depression: a meta-analysis. Cogn. Beh av. Ther. 38 (4),
Patient Session Label
Online session 1: psycho-education
Patient 1 It was good to look back at what happened in the fall. How can it be that you are down one moment and you feel so much better in the next? I now
recognize negativity in others and see how that blocks you. As soon as I start thinking too much about myself, I try to meditate. This really works for me.
Patient 2 I found it hard to write everything down. I recognize a lot of myself in the text. I want to get started in order to regain control. R
Patient 3 This lesson did not provide me with a lot of new information. I am not quite sure how I feel about our example patients. I get that the content of their
depression is not really important within this context, but my experience is very different from theirs.
Patient 5 Difﬁcult. I would rather not think about it. R
Patient 8 It was difﬁcult to start with this session. I either postponed it, or let myself be distracted by other things. To be honest I would rather not think about it. R
Patient 9 It was really good to think about everything. The story of the female example patient made me really emotional. I recognize feeling overwhelmed by
all the things that need to be done. The lesson took me to complete than I anticipated, but this does not surprise me: -)
Online session 2: motivation and goal-setting
Patient 1 Reading everything I wrote, I know who I am but I also see that I play several characters in my life. I would like to make this less confusing and more
homogenous. I know that meditation is a great way to achieve this.
Patient 3 I feel stressed because the weekend has started and I forgot to work on this session. Yoga was not as relaxing as I hoped and at the moment I am
having two of my friends over.
Patient 5 I think this was a difﬁcult session. It took me a few days to complete it. I know how I want things to be, but even now I am not sure whether this will
Patient 8 I had great difﬁculty with completing this session and I postponed working on it for a long time. At ﬁrst glance I could not think of any goal or
possible change. Then I understood that setting (positive and feasible) goals is the ﬁrst step towards improvement, and that it is therefore important.
Online session 3: activity monitoring
Patient 1 It made me aware of the amount of activities that I engage in during the day. I realize now that I haven't been living in a structured way. I do have to
pay attention not to engage in everything that crosses my path [...] I can see that I have a social life again and that feels good.
Patient 2 I found it hard to keep track of my activities and to grade how they made me feel. Yesterday I felt really insecure [shares personal information about
his home life] It was an emotional, but also a good experience to talk to different people.
Patient 3 I did not have a good week; my life sort of fell apart. Now I just want to lie on my couch. C
Patient 5 This was not a good week [shares personal details about her home life] It is difﬁcult to read everything that I wrote. I would have liked to experience
this week in a more positive way.
Online session 4: positive activities
Patient 2 I quickly came to the conclusion that I need to change. I almost immediately started working on this. R
Patient 5 It was a helpful session. I should pay more attention to the things I like doing and just schedule them. If I start planning things, this will provide structure.
Knowing what my day will look like in the morning might prevent me from panicking when I think about all the things that I need to do during the day.
Patient 6 I went to see colleagues. It felt good, but now it is hard to unwind again. That still is an important theme for me. C/R
Patient 8 I copied the list with things that can make you feel better. Maybe it is a good idea to print this and hang it on the wall. I think I will do the same with
my list of goals, so that I will continue to remember them.
Online session 5: structure and planning
Patient 2 Difﬁcult. Because I ruminate a lot, things have not been going well for me. I want to do things right, but I at the same time I do realize that just doing
things is more important. I became aware of the fact that I need structure.
Patient 3 The to-do lists are going well. I do ﬁnd it difﬁcult to ﬁt them in a speciﬁc time schedule. R
Patient 5 I have noticed that it is very helpful for me to plan ahead. This Wednesday I planned to take a walk. When the moment to go out came, I really did not
feel like going. I went anyway, because that was how I planned it. Afterwards I felt a lot better and regretted not doing this sooner. If I would not have
planned the walk, I would have stayed in and aid on the couch.
Patient 8 I ﬁnd it difﬁcult to write down all the things that I do during the day, let alone grade how these activities made me feel. This is why I stopped doing it. E
Online session 6: cognitive dysfunctions
Patient 2 I recognized a lot of myself in the text. Negative thinking just creeps up on you and it is hard to keep ﬁghting this. R
Patient 3 The way the different dysfunctions are presented with the two ﬁctional women was too simpliﬁed
for my taste. I thought this was insulting. E
Patient 5 I started writing down several thoughts. While doing this, I realized that these thoughts all stem from the same cognitive dysfunction. It was difﬁcult
to think about this, because normally I would just avoid thinking about it.
Patient 6 I did not like the way the dysfunctions were presented. It reminded me of high school. Nevertheless I answered all questions. E
Patient 8 I am a bit at a loss. It is 8 o'clock at night and I do not feel like doing anything. I might just go to bed. C
Online session 7: recognizing dysfunctional thoughts
Patient 2 I think this is hard to do. E/R
Patient 3 I did not have a good week. C
Patient 6 I believe I try to suppress certain fears and thoughts, but then certain events still remind me of them. So for me I think events remind me of my
thoughts, rather than causing them to occur. [Provides an example of a past week experience during which this process occurred].
Online session 8: changing dysfunctional thoughts
Patient 2 It all seems easy to do, but in practise it is harder to accomplish. E/R
Patient 5 This was a good session for me, because I quickly became aware of what I was doing wrong. R
Patient 6 [Provides personal information on past week experiences with negative thinking] C
Online session 9: relapse prevention
Patient 2 I know this works in private situations, but I am not sure what will happen at work. R
Patient 5 This was not easy for me, but it felt good to think about it. I do not feel quite stable yet. Two weeks ago I felt on top of the world and then last week I
felt less happy again. Nevertheless, I could come out of this again! This gives me hope for the future.
E: Evaluation of online session; C: Context; R: Reﬂection.
69L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., Hedman, E., 2014. Internet-based vs.
face-to-face cognitive behaviour therapy for psychiatric and somatic disorders: a sys-
tematic review and meta-analysis. World Psychiatr. 13 (3), 288–295. http://dx.doi.
Andrews, G., Williams, A.D., 2014. Up-scaling clinician assisted internet cognitive
behavioural therapy (iCBT) for depression: a model for dissemination into pri-
mary care. Clin. Psychol. Rev. 41 (11). http://dx.doi.org/10.101 6/j.cpr.2014.05.
Andrews, G., Cuijpers, P., Craske, M.G., McEvoy, P., Titov, N., 2010. Computer therapy for
the anxiety and depressive disorders is effective, acceptable and practical health
care: a meta-analysis. PLoS ONE 5 (10), e13196. http://dx.doi.org/10.1371/journal.
Bangor, A., Kortum, P.T., Miller, J.T., 2008. An empirical evaluation of the system usability
scale. Int. J. Hum. Comput. Interact. 24 (6), 574–594. http://dx.doi.org/10.1080/
Bangor, A., Kortum, P., Miller, J., 2009. Determinin g w hat individual SUS scores
mean: adding an ad jective rati ng scale. J. U sability S tud. 4 (3), 1 14–123 doi:
Beck, A.T., Steer, R.A., 1993. Beck Anxiety Inventory: Manual. The Psychological Corpora-
tion, San Antonio, TX.
Beck, A.T., Rush, A.J., Shaw, B.F., Emery, G., 1979. Cognitive Therapy Of Depression.
Guilford, New York.
Beck, A.T., Epstein, N., Brown, G., Steer, R.A., 1988. An inventory for measuring clinical
anxiety: psychometric properties. J. Consult. Clin. Psychol. 56 (6), 893–897.
Becker, E.M., Jensen-Doss, A., 2013. Computer -assisted therapies: examination of
therapist-level barriers to their use. Behav. Ther. 44 (4), 614–624. http://dx.doi.org/
Bockting, C.L.H., Huibers, M.J.H., 2011. Treatment protocol for patients with a depressive
disorder—protocollaire behandeling van patienten met een depressieve stoornis. In:
Keijsers, G.P.J., van Minnen, A., Hoogduin, C.A.L. (Eds.), Protocollaire behandelingen
voor volw assenen met psychische klachten 1. Uitgeverij Boom, Amsterdam,
Bower, P., Gilbody, S., 2005. Stepped care in psychological therapies: access, effectiveness
and efﬁciency. Narrative literature review. Br. J. Psychiatry 186, 11–17. http://dx.doi.
Bremmer, F., van Es, M., 2013. An Analysis Of The Expected Costs And Beneﬁts Of Ehealth
— Blended Treatment And Guidance — Een Analyse van Verwachte Kosten en Baten
van eHealth — Blended Behandelen en Begeleiden. GGZ Nederland, Amersfoort.
Brooke, J., 1996. SUS-a quick and dirty usability scale. In: Jordan, P.W., Tho mas, B.,
Weerdmeester, B.A., McClelland, I.L. (Eds.), Usability Evaluation In Industry 189. Tay-
lor & Francis Ltd., London, pp. 189–194.
Brooks, R., 1996. EuroQol: the current state of play. Health Policy (Amst., Neth.) 37 (1),
53–72 Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/10158943.
Butler, A.C., Chapman, J.E., Forman, E.M., Beck, A.T., 2006. The empirical status of cogni-
tive–behavioral therapy: a review of meta-analyses. Clin. Psychol. Rev. 26 (1),
Cuijpers, P., Riper, H., 2014. Internet interventions for depressive disorders : an overview.
Rev. Psicopatología Y Psicol. Clín. 19 (2), 209–216http://dx.doi.org/10.5944/rppc.vol.
Cuijpers, P., Berking, M., Andersson, G., Quigley, L., Kleirboer, A., Dobson, K.S., 2013a. A
meta-analysis of cognitive-behavioural therapy for adult depression, alone and in
comparison with other treatments. Can. J. Psychiatr. 58 (7), 376–385.
Cuijpers, P., Huibers, M., Ebert, D.D., Koole, S.L., Andersson, G., 2013b. How much psycho-
therapy is needed to treat depression? A metaregression analysis. J. Affect. Disord.
149 (1–3), 1–13. http://dx.doi.org/10.1016/j.jad.2013.02.030.
De Brey, H., 1983. A cross-national validation of the client satisfaction questionnaire: the
Dutch experience. Eval. Program Plann. 6 (3–4), 395–400. http://dx.doi.org/10.1016/
Demyttenaere, K., Bruffaerts, R., Posada-Villa, J.,Gasquet,I.,Kovess,V.,Lepine,J.P.,...WHO
World Mental Health Survey Consortium, 2004. Prevalence, severity, and unmet need
for treatment of mental disorders in the World Health Organization World Mental
Health Surveys. JAMA 291 (21), 2581–2590. http://dx.doi.org/10.1001/jama.291.21.
Devilly, G.J., Borkovec, T.D., 2000. Psychometric properties of the credibility/expectancy
questionnaire. J. Behav. Ther. Exp. Psychiatry 31 (2), 73–86. http://dx.doi.org/10.
Group, EuroQol, 1990. EuroQol - a new facility for the measurement of health-related
quality of life. Health Policy 16 (3), 199–208 Retrieved from. http://www.ncbi.nlm.
Gyani, A., Shafran, R., Myles, P., Rose, S., 2014. The gap between science and practice: how
therapists make their clinical decisions. Behav. Ther. 45 (2), 199–211. http://dx.doi.
Harvey, A.G., Gumport, N.B., 2015. Evidence-based psychological treatments for mental
disorders: modiﬁable barriers to access and possible solutions. Behav. Res. Ther. 68
(5), 1–12. http://dx.doi.org/10.1016/j.brat.2015.02.004.
Hedman, E., Ljótsson, B., Kaldo, V., Hesser, H., El Alaoui, S., Kraepelien, M., ... Lindefors, N.,
2014. Effectiveness of Internet-based cognitive behaviour therapy for depression in
routine psychiatric care. J. Affect. Disord. 155 (1), 49–58. http://dx.doi.org/10.1016/
Hickie, I.B., Davenport, T.A., Luscombe, G.M., Moore, M., Grifﬁths, K.M., Christensen, H.,
2010. Practitioner-supported delivery of internet-based cognitive behaviour therapy:
evaluation of the feasibility of conducting a cluster randomised trial. Med. J. Aust. 192
(11 Suppl), S31–S35 Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/
Høifødt, R.S., Lillevoll, K.R., Grifﬁths, K.M., Wilsgaard, T., Eisemann, M., Waterloo, K.,
Kolstrup, N., 2013. The clinical effectiveness of web-based cognitive b ehavioral
therapy with face-to-face therapist support for depressed primary care patients: ran-
domized controlled trial. J. Med. Internet Res. 15 (8), e153. http://dx.doi.org/10.2196/
Kelders, S.M., Bohlmeijer, E.T., Pots, W.T.M., van Gemert-Pijnen, J.E.W.C., 2015. Comparing
man and automated support for depression: fractional factorial randomized con-
trolled trial. Behav. Res. Ther. 72 (9), 72–80. http://dx.doi.org/10.1016/j.brat.2015.
Kenter, R.M.F., van Straten, A., Hobbel, S.H., Smit, F., Bosmans, J., Beekman, A., Cuijpers, P.,
2013. Effectiveness and cost effectiveness of guided online treatment for patients
with major depressive disorder on a waiting list for psychotherapy: study protocol
of a randomized controlled trial. Trials 14 (1), 412. http://dx.doi.org/10.1186/1745-
Kenter, R.M.F., van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R.S., ... van
Straten, A., 2015. Costs and effects of Internet cognitive behavioral treatment blended
with face-to-face treatment: results from a naturalistic study. Internet Interv. 2 (1),
Kooistra, L.C., Wiersma, J.E., Ruwaard, J., van Oppen, P., Smit, F., Lokkerbol, J., Cuijpers, P.,
Riper, H., 2014. Blended vs. face-to-face cognitive behavioural treatment for major
depression in specialized mental health care: study protocol of a randomized con-
trolled co st-effectiveness tr ial. BMC Psychiatr. 14 (1), 290. http ://d x.doi.org/10.
Lamers, L.M., McDonnell, J., Stal meier, P.F. M., Krabbe, P.F.M., Buss chbach , J.J. V., 2006.
The D utch tariff: results a nd arguments for an effect ive de sign fo r nation al EQ-
5D valuation studies. Health Econ. 15 (10), 1121–1132. http://dx.doi.org/10.
Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., Nguyen, T.D., 1979. Assessment of client/pa-
tient satisfaction: development of a general scal e. Eval. Program Plann. 2 (3),
Leon, A.C., Davis, L.L., Kraemer, H.C., 2011. The role and interpretation of pilot studies in
clinical researc h. J . Psychiatr. Res. 45 (5), 626–629. http://dx.doi.org/10.1016/j.
Lovell, K., Richards, D., 2000. Multiple Access Points & Levels of Entry (MAPLE): en-
suring choice, accessibility & equity for CBT services. Behav. Cogn. Psychother.
28 (4), 379–391 Retrieved from ht tp://linkinghub.els evier.com/retrieve/pii/
Ly, K.H., Janni, E., Wrede, R., Sedem, M., Donker, T., Carlbring, P., Andersson, G., 2015. Ex-
periences of a guided smartphone-based behavioral activation therapy for depres-
sion: A qualitative study. Internet Interv. 2 (1), 60–68. http://dx.doi.org/10.1016/j.
Månsson, K.N.T., Skagius Ruiz, E., Gervind, E., Dahlin, M., Andersson, G., 2013. Develop-
ment and initial evaluation of an Internet-based support system for face-to-face cog-
nitive behavior therapy: a proof of concept study. J. Med. Internet Res. 15 (12), e280.
Piek, E., van der Meer, K., Penni nx, B.W.J.H., V erhaak, P.F.M., Nolen, W.a., 2011. Refer-
ral of patients w ith dep ression to mental health care by Dutch ge neral pr acti-
tioners: an observational study. BMC Fam. Pract. 12 (1), 41. http://dx.doi.org/
Richards, D., Richardson, T., 2012. Computer-based psychological treatments for depres-
sion: a systematic review and meta-analysis. Clin. Psychol. Rev. 32 (4), 329–342.
Riper, H., van Ballegooijen, W., Kooistra, L., de Wit, J., Donker, T., 2013. Prevention &
eMental-health — Preventie & eMental-health. Vrije Universiteit comm isioned by
Robertson, L., Smith, M., Castle, D ., Tannenbaum, D., 2006. Using the Internet to en-
hance the treatment o f depression. Australas. Psychiatry : Bull. R. Aust. N. Z.
Coll. Psychiatr. 14 (4), 413
Rush, A.J., Carmody, T., Reimitz, P., 2000. The Inventory of Depressive Symptomatol-
ogy (IDS): Clinician (IDS-C) and Self-Report (IDS-SR) ratings o f depressive
symptoms.Int.J.MethodsPsychiatr.Res.9(2),45–59. http: //dx.doi.org /10.
Ruwaard, J., Lange, A., Schrieken, B., Dolan, C.V., Emmelkamp, P., 2012. The effectiveness of
online cognitive behavioral treatment in routine clinical practice. PLoS One 7 (7).
Sauro, J., 2011. Measuring usability With The System Usability Scale (sus). Retrieved from
Spijker, J., Bockting, C.L.H., Meeuwissen, J.A.C., van Vlie t, I.M., Emmelkamp, P.M.G.,
Hermens, M.L.M., van Balkom, A. , 2013. Multidisciplinary Guideline for Depression
(Third Revision). Guideline for Diagnostics, Treatment and Guidance of Adult
Patients with a Depressive Disorder — Multidisciplinaire Richtlijn Depressie
(Derde Revisie). Richtlijn Voor de Diagnostiek, Behandeling en e n begeleiding van
volwassen patiënten met een depressieve stoornis. Trimbos-instituut, Utrecht Retrieved
from http ://www.ggzrichtli jnen.nl/index.php?pagina=/richtlijn/item/pag ina.
Trivedi, M.H., Rush, A.J., Ibrahim, H.M., Carmody, T.J., Biggs, M.M., Suppes, T., ... Kashner,
T.M., 2004. The Inventory of Depressive Symptomatology, Clinician Rating (IDS-C)
and Self-Report (IDS-SR), and the Quick Inventory of Depressive Symptomatology,
Clinician Rating (QIDS-C) and Self-Report (QIDS-SR) in public sector patients with
mood disorders: a psych. Psychol. Med. 34 (1), 73–82. http://dx.doi.org/10.1017/
Van Ballegooijen, W., Cuijpers, P., van Straten, A., Karyotaki, E., Andersson, G., Smit, J.H.,
Riper, H., 2014. Adherence to Internet-based and face-to-face cognitive behavioural
therapy for depression: a Meta-Analysis. PLoS One 9 (7), e100674. http://dx. doi.
Van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E.T., van Gemert-Pijnen,
L.J.E.W.C., 2014. Blending online therapy into regular face-to-face therapy for
70 L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71
depression: content, ratio and preconditions according to patients and therapists using a
Delphi study. BMC Psychiatr. 14 (1), 355. http://dx.doi.org/10.1186/s12888-014-0355-z.
Van Gemert-Pijnen, J.E., Kelders, S.M., Bohlmeijer, E.T., 2014. Understanding the usage of
content in a mental health intervention for depression: an analysis of log data. J. Med.
Internet Res. 16 (1), e27. http://dx.doi.org/10.2196/jmir.2991.
Waller, G., 2009. Evidence-based treatment and therapist drift. Behav. Res. Ther. 47 (2),
Wilh elmsen, M., Lillevoll, K., Risør, M.B., Høifødt, R., Johansen, M.-L., Waterloo, K., . ..
Kolstrup, N., 2013. Motivation to persist with internet-based cognitive behavioural
treatment using blended care: a qualitative study. BMC Psychiatr. 13 (1), 296–304.
Williams, A.D., Andrews, G., 2013. The Effectiveness of Internet Cognitive Behavioural
Therapy (iCBT) for depression in primary care: a quality assurance study. PLoS One
71L.C. Kooistra et al. / Internet Interventions 4 (2016) 61–71