Computerized continuing care support for alcohol and drug dependence:
A preliminary analysis of usage and outcomes
Audrey A. Klein, (Ph.D.)a,⁎, Valerie J. Slaymaker, (Ph.D.)a,
Karen L. Dugosh, (Ph.D.)b, James R. McKay, (Ph.D.)c
aButler Center for Research, Hazelden Foundation, Center City, MN, USA
bTreatment Research Institute, Philadelphia, PA, USA
cDepartment of Psychiatry, University of Pennsylvania, and Philadelphia Veterans Affairs Medical Center, Philadelphia, PA, USA
Received 30 August 2010; received in revised form 1 July 2011; accepted 5 July 2011
The central aim of this administrative data analysis was to examine usage of a Web-based disease management program designed to
provide continuing recovery support to patients discharged from residential drug and alcohol treatment. Tailored clinical content was
delivered in a multimedia format over the course of 18 months posttreatment. The program also included access to a recovery coach across
the 18 months. Consistent with other disease management programs, program usage decreased over time. A small subsample of patients
accessed a large number of program modules in the year following treatment; these patients had significantly higher abstinence rates and
consumed less alcohol than patients accessing few or no modules. Regression analyses revealed a significant relationship between the
number of modules accessed and substance use outcomes in the year following treatment when controlling for motivation, self-efficacy, and
pretreatment substance use. Limiting the analyses to only the more compliant patients did not reduce the magnitude of these effects. These
preliminary results suggest that computerized support programs may be beneficial to patients recently treated for drug and alcohol issues.
Methods to increase program engagement need additional study. © 2011 Elsevier Inc. All rights reserved.
Keywords: Alcohol dependence; Drug dependence; Continuing care; Computer based; Residential treatment
The use of computerized disease management programs
and other e-health interventions to treat and manage chronic
health conditions has increased over recent years. Several
research studies have examined participant usage of these
programs and the extent to which usage is related to positive
health behavior outcomes across a number of domains,
including cancer (Gustafson et al., 2005), diabetes (Glasgow
et al., 2005; Williams, Lynch, & Glasgow, 2007), heart
disease (Verheijden et al., 2004), and mood disorders
(Farvolden, Denisoff, Selby, Bagby, & Rudy, 2005).
Overall, these studies suggest that computerized disease
management programs can be effective, although rates of
engagement are often low.
A number of computerized programs have been devel-
oped to address substance use and addictive disorders, with
many focusing on alcohol abuse. Most of these programs
provide screening and brief intervention, wherein partici-
pants provide information about alcohol and drug use and
then receive clinical information, advice, and guidance on
how to reduce use. A number of studies suggest that these
programs are effective. For example, Hester, Squires, and
Delaney (2005) developed the Drinker's Check-Up (DCU), a
brief motivational intervention delivered over the computer
in a series of modules. Modules focus on assessing drinking
and alcohol-related problems, providing personalized feed-
back, and helping the participant resolve ambivalence about
changing drinking. A recent study found that participants
receiving DCU substantially reduced their alcohol use and
other alcohol-related problems over a 12-month follow-up
Journal of Substance Abuse Treatment xx (2011) xxx–xxx
⁎Corresponding author. Butler Center for Research, Hazelden, P. O.
Box 11 (BC-4), Center City, MN 55012-0011. Tel.: +1 651 213 4106; fax:
+1 651 213 4536.
E-mail address: firstname.lastname@example.org (A.A. Klein).
0740-5472/11/$ – see front matter © 2011 Elsevier Inc. All rights reserved.
period (Hester et al., 2005). Studies of other computerized
screening and brief interventions have reported similar
results (Cunningham, Humphreys, Kypri, & van Mierlo,
2006; Kypri, Langley, Saunders, Cashell-Smith, &
Other studies have shown that computerized versions of
popular psychosocial treatment approaches are effective
when administered to individuals having a substance use
disorder. For example, Carroll et al. (2008) developed
CBT4CBT, which supplemented therapist-delivered treat-
ment with 8 weeks of biweekly access to a multimedia
computer-based version of cognitive–behavior therapy
(CBT). The first randomized controlled trial of this program
focused on individuals with substance dependence attending
outpatient treatment. Substance use outcomes were com-
pared for two groups: one group received outpatient
treatment plus the computerized version of CBT, and the
other group received outpatient treatment as usual. The
group receiving computerized CBT reported a greater
number of abstinent days across the 8-week treatment period
and had fewer positive drug screens compared with the
treatment-as-usual group (Carroll et al., 2008). A second
report from this study examined substance use over a longer
follow-up period (6 months) and produced similar findings
(see Carroll et al., 2009).
In a randomized controlled trial of opioid-dependent
outpatients, Bickel, Buchhalter, Marsch, and Badger (2008)
examined the efficacy of an interactive computerized
behavioral treatment intervention based on contingency
management and community reinforcement approaches
(CRA). Patients receiving the computerized CRA treatment
along with vouchers showed roughly the same number of
weeks of continuous cocaine and opioid abstinence as
patients receiving therapist-delivered CRA with vouchers,
and both groups showed significantly longer periods of
abstinence than a treatment-as-usual group. In a recent
review of studies using computer-based treatment with
individuals dependent on one or more illicit drugs, Moore,
Fazzino, Garnet, Cutter, and Barry (2011) reported that most
studies found better drug use outcomes for patients receiving
computerized treatment than for control group patients not
receiving the intervention. Patients also reported being
highly satisfied with the computerized interventions and
showed levels of treatment engagement and retention that
were similar to therapist-provided treatments.
Although research generally supports the effectiveness of
computerized prevention programs, such as screening and
brief intervention, and computer-delivered treatment, very
little is known about whether computer technology can be
used to deliver continuing care and recovery support to
individuals who have already received formalized treatment
for a substance use disorder. To our knowledge, in contrast
to other disorders such as heart disease and diabetes, no
computerized continuing care programs have been devel-
oped for long-term management of substance use disorders
despite a large body of research suggesting that traditional
(noncomputerized) continuing care programs are associated
with better posttreatment outcomes (McKay, 2009a). A
recent review of controlled continuing care studies found
that interventions that were at least 12 months long or that
included more active efforts to deliver the intervention
components to patients were more likely to produce positive
effects (McKay, 2009b).
Most continuing care programs for substance use
disorders, such as those discussed above, require participants
to attend structured support groups and/or engage with
counselors and other clinical staff over the telephone. It is
currently unclear to what extent computerized programs can
be used to deliver continuing care and recovery support to
individuals treated for alcohol/drug dependence. Several
years ago, Hazelden staff created My Ongoing Recovery
Experience (MORE), a computerized recovery support
program developed for patients receiving residential treat-
ment for alcohol/drug dependence. Unlike other continuing
care programs, the program delivers assessments, clinical
content, resources, and activities over the computer. The
program also involves periodic telephone and e-mail contact
between patients and recovery coaches. This study had two
goals: to examine usage of the MORE program after
discharge from residential treatment and to assess the
relationship between program usage and substance use
outcomes during the year following treatment.
2. Materials and methods
2.1. The Hazelden MORE program
In 2006, Hazelden, a private nonprofit addiction treatment
center, launched an innovative, first-of-its-kind computer-
ized recovery support program called MORE. MORE was
developed by licensed alcohol and drug counselors and
mental health professionals for use with adult patients
completing residential treatment for alcohol or drug
dependence at Hazelden sites. Prior to implementation,
patient input regarding program development was obtained
through a number of focus groups. Feasibility of the program
was tested by administering a sample of the program to
several groups of patients who volunteered to be end users.
Patients provided a number of recommendations regarding
clinical content and program functionality that were
incorporated into the final version of the program.
The program incorporates concepts and findings from a
number of evidence-based treatment approaches, such as 12-
step facilitation, motivational interviewing, and CBT.
Through the program, patients have access to recovery-
related information and services for a period of 18 months
following discharge from treatment. MORE is an iteratively
tailored, Web-based, mixed-media program that provides
interactive recovery-related activities, videos, an extensive
electronic library of content, opportunities for fellowship
with other recovering individuals, and other sources of
2A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
support. The videos cover a wide range of content, including
patient testimonials and information on topics such as how to
manage a co-occurring disorder, how to recognize factors
that may lead to relapse, how to protect oneself from
developing an addiction to prescription medications, and
how to effectively manage stress. Other computerized
resources include a patient journal, workbook activities for
practicing and applying recently learned information, and an
electronic library of articles pertaining to issues such as
relapse prevention, dealing with cravings, and maintaining
emotional health. Participants also have access to a network
of Hazelden alumni and a number of resources for staying in
contact with other individuals in recovery, such as recovery
blogs, forums, podcasts, and online 12 step meetings.
The program is delivered in seven recovery “modules,”
each of which begins with an electronically administered
assessment where patients provide detailed information
about substance use and other aspects of life functioning.
Based on the responses, the module then delivers content
tailored to the individual needs of the patient. Patients
receive modules in a sequential manner over an 18-month
period, and modules have to be completed in order. In other
words, content from later modules is only accessible if the
participant accessed prior modules. In addition, the
availability of the modules is timed across the 18-month
period. The assessment for Module 1 is completed while
patients are still in residential treatment, usually 1 week
prior to discharge.
Each module focuses on specific issues that are likely to
be experienced at that particular time in recovery. Table 1
provides a summary of each module and its key activities.
Modules vary somewhat in length and range from 19
pages of content (Module 7) to 36 pages of content
(Module 2). The total number of pages across all seven
modules is 201.
In addition to the recovery modules and other comput-
erized program components, another aspect of MORE
pertains to recovery coaching. Each participant has a
personal recovery coach, a licensed drug and alcohol
counselor who provides ongoing support and guidance for
a period of 18 months following treatment. Coaches
maintain periodic contact with the patient both electronically
and by telephone through a series of system “flags” or alert
messages that indicate the patient has reached a milestone or
is experiencing problems. When coaches receive a flag, they
call the patient and discuss his or her current concerns.
Patients who cannot be reached by telephone are sent an e-
mail message and asked to contact their coach. The
frequency of coach-to-patient contact varies considerably,
ranging from no contact at all to several times per month.
We examined a sample of residential patients who
enrolled in the MORE program shortly before discharge;
patients were discharged in the period from May 2006 to
December 2007 (N = 1,124). Only participants who met
criteria for dependence on at least one substance were
included in the sample; those who were classified as having
abuse but not dependence were excluded. No other criteria
were used to exclude patients from the sample. Table 2
summarizes demographic and baseline clinical characteris-
tics of the sample. The average length of stay was 25.72 days
(SD = 5.77), with 92% (n = 1,034) successfully completing
treatment. Treatment completion was defined as attending
the residential program for the period prescribed by clinical
staff. The sample was 45% female, and the average age was
42.12 years (SD = 11.70 years). Forty six percent (n = 516)
of the sample was married at the time of treatment admission,
and 42% (n = 470) was employed full time. Ninety-six
percent (n = 1,079) of patients were Caucasian. Please note
that in this section and in later sections, the “n” represents the
number of participants with a positive value for the relevant
variable, not the total number in the sample.
Substance dependence and mental health diagnoses were
made in the course of routine clinical operations through a
comprehensive baseline assessment conducted by an inter-
disciplinary team of licensed alcohol and drug counselors,
psychiatrists, and doctoral-level clinical psychologists. All
diagnoses were based on the Diagnostic and Statistical
MORE module content
Module Module focusSample of key activities
Module 1Mind, body, and spirit Identifying people, places, and things that are triggers to relapse and developing coping strategies to deal
with them. Creating a safe home environment free of triggers and cues. Engaging in AA, finding groups,
obtaining a home group, finding a sponsor, and using the 12 steps.
Developing a supportive network of healthy people, finding friends in recovery, reconnecting with family,
and asking for help.
Addressing workplace issues and establishing sober fun. Achieving lifestyle balance.
Coping with normal feelings in recovery including anger, anxiety, sadness, and loneliness.
Addressing unhealthy thought processes or compulsive behaviors that may emerge.
Developing healthy intimate relationships, communicating effectively with others, and establishing trust.
Establishing an identity as a sober individual, identifying and abandoning old thought patterns and beliefs,
and exploring ways to continuously improve emotional and physical health.
Reexamining career, education, and life goals for long-term recovery.
Module 2Getting connected
Balancing work and play
Healthy and emotional living
Module 7 Life goals
3 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
Manual of Mental Disorders, Fourth Edition (DSM-IV)
criteria (American Psychiatric Association [DSM-IV-TR],
2000). Eighty-seven percent (n = 976) of the sample received
a diagnosis of alcohol dependence. The percentage diag-
nosed as dependent on both alcohol and at least one other
drug was 35% (n = 392). Sixty-five percent (n = 730) had a
co-occurring mental disorder in addition to alcohol and/or
drug dependence, with 35% (n = 393) having a depressive
disorder. The average overall score on the Behavior and
Symptom Identification Scale (BASIS-32) indicated a
moderate level of impairment (1.45; SD = 0.73).
In addition to baseline data, patients completed
follow-up surveys administered at roughly 6 and 12
months after discharge. Seventy percent of the sample
(n = 786) completed the 6-month survey, 54% (n = 611)
completed the 12-month survey, and 48% (n = 541)
completed both surveys.
2.3.1. Measures collected at baseline
In the course of routine clinical operations, several
surveys were administered to patients by counseling staff
within several days of treatment admission. These surveys
captured demographic information, frequency of substance
use before treatment, and mental health characteristics.
22.214.171.124. Pretreatment alcohol and drug use. Several
questions from the modified Form 90 (Project MATCH
Research Group, 1993) were used to assess alcohol and drug
use during the 90 days prior to treatment admission. Alcohol
use items included the number of days the patient had at least
one alcoholic drink (drinking days), the average number of
drinks consumed on drinking days (drinks per drinking day),
and the number of days the patient used each of several
recreational drugs, as a function of drug type (e.g.,
marijuana, cocaine, heroin). The Form 90 has been used
with adolescents and adults with substance use disorders and
has demonstrated good validity and test–retest reliability
(Tonigan, Miller, & Brown, 1997).
126.96.36.199. Pretreatment 12-step involvement. Several ques-
tions from the Alcoholics Anonymous Affiliation Scale
(AAAS; Humphreys, Kaskutas, & Weisner, 1998) measured
12-step group involvement during the year prior to treatment
admission. Patients were asked how many 12-step meetings
they had attended during the period and how many of the
steps they had “worked.” They were also asked the following
yes/no questions: whether they had attended AA (or a related
group), whether they considered themselves to be a member
of AA, whether they had celebrated a sobriety birthday,
whether they had a sponsor, whether they had ever been a
sponsor, and whether they had ever experienced a spiritual
awakening. The number of yes responses to these questions
was summed to comprise the “AA composite” score, which
served as a quantitative measure of AA involvement separate
from meeting attendance (see Witbrodt & Kaskutas, 2005).
The AAAS has been shown to have high reliability and
validity (Humphreys et al., 1998).
188.8.131.52. Behavior and Symptom Identification Scale
(BASIS-32). The BASIS-32 is a global measure of psycho-
logical functioning and was administered within the first
week of treatment. It consists of 32 questions that assess five
aspects of mental health: relation to self and others, daily
living and role functioning, depression and anxiety, impul-
siveness, and psychosis. Participants are given 32 items that
list various domains of functioning (e.g., “adjusting to major
life stresses”) and are asked to rate how much difficulty they
experienced in that area during the past week. Higher scores
represent greater problem severity. The BASIS-32 has been
used with adolescents and adults and has demonstrated
acceptable levels of internal consistency, test–retest reliabil-
ity, and concurrent validity (Eisen, Dill, & Grob, 1994).
Demographic and clinical characteristics at baseline
Variable% or M (SD)
Did not complete high school
High school diploma/GED
Single, never married
4 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
2.3.2. Administrative measures related to the treatment
Several variables, including demographic information,
were obtained from the administrative data set. Length of
stay was captured as the number of days each patient
attended residential treatment.
2.3.3. Measures collected within the MORE program
Several questions were administered within the assess-
ment for Module 1 of the MORE program. This assessment
was completed by patients over the computer roughly 1 week
before treatment discharge.
184.108.40.206. Number of people in the support network. Module 1
asks a question regarding the patient's support network:
“Currently, who are the main people you rely on for
support?” It then lists a number of possible responses (e.g.,
wife, husband, father, daughter). A variable was created for
each patient by summing together the total number of
220.127.116.11. Self-efficacy and motivation. The question for self-
efficacy asked, “How confident do you feel in your ability to
work an active recovery program at this time?” (response
scale: 1 = not confident at all to 7 = extremely confident).
The question for motivation asked, “How motivated do you
feel in your ability to work an active recovery program at this
time?” (response scale: 1 = not motivated at all to 7 = ex-
2.3.4. Measures collected at follow-up
A follow-up survey, used in the course of routine health
care operations, contained questions about behaviors and
events that occurred in the period since discharge; only those
questions germane to the present evaluation will be discussed
here. Patients completed the follow-up survey at three
different times postdischarge: 1 month (25–44 days), 6
months (173–210 days), and 12 months (358–395 days).
Questions referred to the period since treatment discharge or
sincethe last follow-upsurveydate ifonewas completed.Itis
up data because these data showed extremely low variability;
with more than 85% of patients reporting no substance use
the 1-month follow-up was accounted for in later follow-ups.
The substance use questions came from the modified
Form 90 (Project MATCH Research Group, 1993) and were
the same as those asked at baseline (discussed above). These
questions and all others were asked to the patient over the
telephone by research department outcomes evaluation staff
extensively trained to conduct follow-up interviews. The 12-
step involvement questions were also the same as those
asked at baseline except they referred to behaviors during the
follow-up period. Another question asked the patient if he or
she had attended any other formal alcohol/drug treatment
during the follow-up period.
Each follow-up also consisted of several questions designed
to measure satisfaction with the MORE program. These
questions were asked to each patient who indicated that he or
she had used MORE at least once since discharge from
fair, and poor). Other questions asked patients to indicate their
perception of how useful the program was to their recovery, for
because of MORE” (both questions used a scale of strongly
agree, agree, neutral, disagree, and strongly disagree).
2.3.5. Outcomes measures
18.104.22.168. MORE usage: Number of modules accessed. Each
of the seven modules within MORE has a tailor date or the
date that a participant first accesses the module assessment.
The total number of module tailor dates for each participant
was summed together to create a variable called “number of
modules accessed.” This variable served as the primary
measure of program use. In addition to serving as a
secondary outcomes measure, number of modules touched
was also used as a predictor in regression analyses for
substance use outcomes.
22.214.171.124. Continuous abstinence. A patient was counted as
continuously abstinent at each follow-up if he or she reported
no alcohol or illicit drug use during the period since
discharge. The percentage of patients continuously abstinent
was calculated for the 6- and 12-month follow-ups.
126.96.36.199. Percent days abstinent from alcohol. At each
follow-up assessment, patients indicated how many days
they consumed alcohol during the period referenced by the
assessment. This number was then subtracted from the total
number of days in the assessment period to get the number of
abstinent days from alcohol. This number was then divided
by the total number of days in the period and multiplied by
100 to obtain the percent days abstinent (PDA) from alcohol.
PDA from alcohol at the 6- and 12-month follow-ups took
into account the number of abstinent days reported at
previously completed evaluations. Although it was possible
to calculate PDA for each illicit drug, we could not calculate
overall PDA from both alcohol and drugs because use days
for different substances often overlapped. Therefore, given
that 87% of patients were alcohol dependent, we focused our
analyses on PDA from alcohol.
2.4. Analysis strategy
The first set of analyses examined module usage within
the MORE program, as indicated by the percentage of
patients accessing a given module at least one time. We also
created a variable that represented the overall level of module
use for each participant.
5 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
In the second series of analyses, we classified participants
into one of two groups based on module usage. The first
group was called the “highly adherent” group and consisted
of participants who accessed at least six of the seven program
modules at least once during the year following discharge.
The second group was called the “less adherent” group and
consisted of participants who accessed five or fewer modules
in the year following discharge. A cutoff of five modules was
chosen for several reasons. Dividing the sample using seven
modules as the criterion resulted in a very small number of
highly adherent users (less than 6% of patients). However, it
was still deemed clinically important to use a fairly stringent
definition of highly adherent use, as the MORE program was
designed to deliver six of the seven modules within 1 year of
leaving treatment. Highly adherent and less-adherent module
users were compared on substance use outcomes at 6 and 12
months postreatment via a series of bivariate tests.
Generalized estimating equations (GEE) and mixed-
model regression analyses were then conducted to further
examine the relationship between module use and substance
use outcomes at 6 and 12 months postdischarge. GEE and
mixed effects models have advantages over conventional
repeated-measures methods in that they allow for missing
observations, accommodate measurements made at different
time points, provide greater flexibility in modeling the
variance–covariance matrix, and permit the estimation of
both group and random subject-specific effects (Diggle,
Heagerty, Liang, & Zeger, 2002). The current analysis
assumes a compound symmetry structure, as we only have
two levels of the repeated-measures factor (i.e., Month 6 and
Month 12). The compound symmetry assumption can only
be violated when the repeated-measures factor has more than
two levels. A separate model was produced for each of two
outcomes: the percentage of patients who were continuously
abstinent from alcohol and drugs at follow-up and the
percentage of days abstinent from alcohol at follow-up. The
number of modules accessed and the time of follow-up (6 vs.
12 months) were entered as predictors. In addition, all other
variables listed in the Measures section were included in the
models as covariates. Although there was a substantial
number of missing assessments, we ran the models on the
existing data and did not impute missing data. Continuous
abstinence status was only calculated if the appropriate
substance use data were available; we did not count
participants who were lost to follow-up as relapsed.
As a last step, the regression analyses outlined above were
conducted only on patients who were deemed to be in
compliance with continuing care recommendations. These
number of MORE modules may also be engaging in other
prescribedrecoverybehaviors,suchasfollowing through with
their continuing care plans and engaging with 12-step
fellowships. These patients would be expected to have better
substance use outcomes, therefore making it unclear whether
outcomes are due to compliance in general rather than to
module use specifically. To address this possibility, we
identified a subsample of compliant patients and reran the
regression models on this group. A patient was labeled
compliant if he or she met all of the following criteria at the
time of the 1-month follow-up: (a) reported following most,
at least two AA or 12-step meetings a week.
3.1. Module engagement
Table 3 shows the percentage of patients in the sample
who accessed each of the seven program modules. As
expected, engagement was highest for Module 1, which was
accessed while patients were in treatment. Eighty-four
percent (n = 943) of discharges accessed this module at
least once. It is important to note that although Module 1 was
available to all patients and they were encouraged by clinical
staff to log onto the module, some patients opted to not log
on. All patients received an introduction to MORE and were
briefly trained on how to use the program, but logging onto
Module 1 was not required as a part of training. It is
unknown why some patients opted to not log onto Module 1
while in treatment. Another key finding was that module
engagement steadily decreased over time (postdischarge),
with 25% (n = 285) accessing Module 3 and 5% (n = 58)
accessing Module 7 (the final module).
Another way to examine module engagement is to sum up
the number of modules accessed by each patient and
determine how many patients accessed a given number of
modules. The mean number of modules accessed was 1.75
(SD = 2.09). Forty-four percent (n = 489) of the sample
accessed Module 1 only (i.e., they did not access any
modules beyond Module 1), and 5% (n = 58) of patients
accessed all seven modules.
3.2. Substance use outcomes of highly adherent versus
less-adherent module users: Results of bivariate tests
Regarding continuous abstinence rates, highly adherent
users had significantly higher rates than less-adherent users
Percentage of participants accessing each MORE program module at least
No. of days
% of May 2006 to
December 2007 discharges
6 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
at both 6 months (81% vs. 54%, p b .001) and 12 months
(78% vs. 49%, p b .001). Regarding PDA from alcohol,
highly adherent users had significantly higher PDA than
less-adherent users at both 6 months (99% [n = 102] vs. 95%
[n = 747], p b .01) and 12 months (99% [n = 94] vs. 95% [n =
584], p b .01).
Another way to examine the relationship between module
use and outcomes is to identify patients who accessed one or
more modules after treatment discharge and compare their
outcomes to patients who do not access any modules after
treatment. Because Module 1 was the only module accessed
during the treatment stay; posttreatment module use was
defined as access of Module 2 or beyond. Forty percent (n =
454) of the sample had posttreatment module use; the
remaining 60% (n = 668) only accessed Module 1 (or no
modules at all).
Regarding continuous abstinence at 6 months, the
posttreatment module use group had a significantly higher
abstinence rate (62%, n = 239) than the group that did not
access the modules after treatment (53%, n = 248), p b .01.
For continuous abstinence at 12 months, the difference
between the two groups approached significance (56% vs.
49%, p = .08). For PDA from alcohol at 6 months, the
posttreatment use group was significantly higher (97%, n =
381) than the other group (94%, n = 468), p b .01. The
difference between the two groups for PDA at 12 months
was also significant (97% vs. 95%, p b .05).
3.3. Mixed-model regression analyses
Mixed-model regression analyses examined the rela-
tionship between number of modules accessed and two
outcome variables: (a) continuous abstinence (binary) and
(b) PDA from alcohol (continuous). The outcome
measures were assessed at Month 6 and Month 12. GEE
(Diggle et al., 2002) analyses were used for the binary
outcome, and linear mixed-effects models (Littell, Milli-
ken, Stroup, & Wolfinger, 1996) were used for the
continuous outcome. The models included terms for total
number of modules accessed, time of follow-up (6 vs. 12
months), and the Module × Time interaction and specified
an exchangeable (GEE) or compound symmetry (linear
mixed effects) covariance structure. In addition, the models
included the following covariates: self-efficacy and
motivation at the Module 1 assessment, attended formal
alcohol/drug treatment during the 1-month follow-up, total
number of people in the support network, number of
pretreatment drinking days, the AA composite score at the
1-month follow-up, and presence of a co-occurring
disorder. A total of 1,122 participants were included in
3.3.1. Results of GEE for continuous abstinence
The GEE analysis revealed a significant effect of the
number of modules accessed, χ2(1) = 18.17, p b .001, with
the likelihood of abstinence increasing as the number of
modules increased (odds ratio [OR] = 1.18, 95% confidence
interval [CI] = 1.10–1.28). In addition, there was a
significant effect of time, χ2(1) = 17.07, p b .001, with
higher rates of abstinence in Month 6 than Month 12 (OR =
1.50, 95% CI = 1.25–1.80). The interaction did not approach
statistical significance (p = .24). Significant covariates in the
model included self-efficacy, χ2(1) = 6.44, p = .01, and co-
occurring disorder, χ2(1) = 5.81, p = .01, with a higher
likelihood of abstinence associated with increases in self-
efficacy (OR = 1.31, 95% CI = 1.07–1.61) and a higher
likelihood of not having a co-occurring disorder (OR = 1.52,
95% CI = 1.08–2.13).
3.3.2. Results of mixed effect models for PDA from alcohol
The results from the mixed-effects model for PDA
indicated that the number of modules accessed was a
significant predictor of PDA, F(1, 602) = 10.25, p = .001.
Time and the Module × Time interaction were not
statistically significant (p b .70 and p b .35, respectively).
Significant covariates in the model included self-efficacy, F
(1, 602) = 4.02, p = .045, number of people in the support
network, F(1, 602) = 4.44, p = .035, and number of
pretreatment drinking days, F(1, 602) = 18.32, p b .001.
Self-efficacy and number in the support network were
positively related to PDA from alcohol, and number of
pretreatment drinking days was negatively related to PDA
3.4. Regression analyses of compliant patients
The same regression models reported above were
conducted on 483 patients who met the definition of being
compliant (as defined in the Materials and methods section)
with one modification: If a predictor in the original model
was one of the variables used to determine compliance, then
it was dropped from the compliant patient model.
These analyses were conducted to determine if the
number of modules accessed continued to predict 6- and
12-month outcomes in the compliant patient models. The
GEE for continuous abstinence revealed a significant effect
of modules accessed, χ2(1) = 6.38, p = .01, with the
likelihood of abstinence increasing with the number of
modules (OR = 1.16, 95% CI = 1.04–1.29). There was also a
significant effect of time, χ2(1) = 10.09, p = .001, with a
greater likelihood of abstinence at Month 6 than at Month 12
(OR = 1.57, 95% CI = 1.21–2.03). The Module × Time
interaction was not statistically significant (p b .27). The
mixed-effects model for PDA from alcohol indicated that the
number of modules accessed was a significant predictor of
PDA, F(1, 324) = 4.39, p = .04. There was also a significant
effect of time, F(1, 211) = 4.61, p = .03, and the Module ×
Time interaction approached significance, F(1, 211) = 3.56,
p = .06. In addition, the covariate of pretreatment drinking
days was a significant predictor, F(1, 324) = 8.61, p = .004,
with a lower number of pretreatment drinking days
associated with a higher PDA.
7 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
3.5. Satisfaction with the MORE program
On each follow-up survey, several questions asked
patients who indicated using the MORE program after
treatment discharge to rate their satisfaction with various
aspects of the program. Of the 454 patients (40% of the
sample) who were using the program during the follow-up
period, more than 70% rated the program as very good or
excellent. More than 80% stated that they agreed or strongly
agreed with the statement “The content of the MORE
program has been relevant to my recovery,” and 64% stated
that they agreed or strongly agreed with the statement “I have
begun working a recovery program because of MORE.”
The present analysis examined usage of the MORE
program, an innovative, computerized recovery support
program for patients who recently completed residential
treatment for alcohol or drug dependence. The program is
available during the first 18 months after discharge from
treatment. The first component of the program involves a
series of modules and other resources and activities delivered
in a mixed-media format over the computer. In addition, each
patient has contact with a recovery coach, a licensed drug
and alcohol counselor. The results of the present analysis
suggest a positive relationship between use of the program
modules and posttreatment outcomes. Patients who accessed
a large number of modules during the 12-month follow-up
period were significantly more likely to be abstinent and had
a higher percentage of days abstinent from alcohol than
patients who accessed few or no modules. In addition,
regression analyses revealed that number of modules
accessed was a significant predictor of substance use
outcomes during the 12 months following treatment, even
when accounting for other covariates such as motivation,
self-efficacy, and pretreatment substance use.
Although a relationship was found between MORE use
and outcomes, it is unclear whether the relationship is causal.
It is possible that patients who are very compliant with all of
their continuing care recommendations (only one of which is
using MORE) have better outcomes than noncompliant
patients. Overall, the regression analyses conducted on
“compliant” patients revealed an effect for MORE program
use that was about the same magnitude as that observed in
the full sample, but findings should be considered prelim-
inary until they are replicated with a stronger design such as a
randomized controlled trial. It is also important to note that
this analysis was based on administrative data collected in
the course of standard clinical operations. A randomized
controlled trial comparing outcomes of MORE participants
with outcomes of participants not using the program is the
only way to determine program efficacy.
This analysis also revealed that a substantial number of
patients were not using the program as recommended.
More than 80% of patients accessed Module 1 at least
once, but this is not surprising given that patients access
this module while still in treatment. In contrast, less than
half of patients accessed Module 2, which is available
roughly 1 month following discharge, with access decreasing
steadily across subsequent modules. However, it is
important to note that many patients did use MORE after
leaving treatment, and a small number of these patients
accessed at least five of the seven modules, indicating some
long-term use of the program during the year following
treatment. Regarding satisfaction with the MORE program,
most of the patients rated the overall quality of the program
as excellent, although 30% of patients did not give it a high
rating. This result combined with the relatively low levels
of engagement suggests that modifications to implementa-
tion efforts or to the program itself may be in order. Future
research should focus on increasing the acceptability and
feasibility of the program. Methods to increase engage-
ment, such as providing patients with incentives, incorpo-
rating module use directly into the residential treatment
plan, or increasing the frequency and duration of contacts
between each patient and his or her recovery coach, should
be further explored.
It is important to note that low levels of patient
engagement are not unique to MORE, and several studies
have reported low engagement with other Web-based
addiction programs. Strecher et al. (2008) examined a
behavioral Web-based smoking cessation program in two
samples of HMO members who were attempting to quit
smoking. Similar to MORE, the program consisted of
distinct sections available over a period of time that
delivered tailored messaging and feedback to participants
based on their responses. On average, participants accessed
2.5 sections during the period. These numbers are similar
to the mean number of program modules accessed in this
study, which was 1.75. Another study examined Quit
Smoking Now, a Web-based program providing smoking
cessation information and guidance on behavior change
strategies (McKay, Danaher, Seeley, Lichtenstein & Gau,
2008). The average number of program visits during a
6-month period was 2.14, and more than 90% of
participants had completely stopped using the program
by 6 months postenrollment. In summary, low levels of
patient engagement with Web-based, computerized health
behavior change programs continue to present clinical and
Although no published studies to date have examined
computerized continuing care programs, a large number of
studies have examined patient engagement with more
traditional programs. Similar to the engagement results
with MORE reported here, these studies indicate that client
engagement and retention are an issue with traditional
continuing care programs. Cacciola et al. (2008) conducted a
study of a telephone-based continuing care protocol recently
implemented at the Betty Ford Center. This program, called
Focused Continuing Care, is offered to all patients who have
8 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
completed residential treatment and involves several brief
telephone-counseling sessions between patients and clinical
staff. These calls are scheduled to take place at predeter-
mined periods after discharge from treatment. A key finding
was that although 71% of patients in the sample completed
the first call, only 28% completed the last call (which took
place roughly a year after discharge). By 6 months
postdischarge, less than 40% of patients completed any
given call, and less than 1% completed all 14 calls of the
program. The 1% call completion rate is similar to the
module completion rate found for the MORE program,
which was 5%. Lash et al. (2007) conducted a randomized
clinical trial to determine the effectiveness of a continuing
care program for patients attending residential treatment
through the VA. The program consisted of three processes:
(1) clients signed a written contract prior to leaving treatment
whereby they agreed to attend continuing care sessions, (2)
clients received attendance prompts prior to each scheduled
session, and (3) clients were positively reinforced for
attendance. Even when patients were explicitly reminded
to attend their continuing care sessions and were positively
reinforced for doing so, attendance decreased substantially
over time, with only 40% attending sessions on a monthly
basis throughout the follow-up period. These studies as a
whole suggest a number of challenges in keeping patients
engaged in continuing care programs over time.
A large number of studies have also examined the extent
to which traditional continuing care programs enhance
treatment outcomes. Similar to our finding that patients
who were highly engaged with MORE had better substance
use outcomes, several studies have found continuing care
programs to be effective for patients who engage with these
programs. It is important to note that these studies provide
much stronger support than this study because many were
randomized controlled trials that compared patients receiv-
ing the program with a control group of patients who did not
receive the program. In the study by Lash et al. (2007)
described above, clients who participated in the continuing
care program were significantly more likely to be abstinent 1
year following treatment than clients in the treatment-as-
usual condition. McKay (2005) examined clients attending a
4-week intensive outpatient program through either the VA
or their community. Clients in the continuing care condition
received brief telephone counseling sessions combined with
several group support sessions; these patients were compared
with those who received relapse prevention or 12-step-
oriented group counseling. Continuing care clients had
significantly higher abstinence rates in the 2 years following
treatment than the other two groups. Several other studies
have reported similar findings (for a review, see McKay,
Several limitations should be considered when interpret-
ing the administrative data reported here. Although the
follow-up rate at 6 months was 70%, only 54% of the sample
was contacted at12 months. The low end point follow-up rate
raises concerns about potential sample bias. Another issue
pertaining to generalizability concerns findings for PDA
from alcohol during the 6- and 12-month follow-up periods.
PDA was extremely high (i.e., greater than 93%) even at 12
months. These data suggest that patients who are relapsing to
alcohol use are not doing so for a long period. Potential
explanations for this include underreporting of substance use
or that a disproportionate number of patients with highly
favorable outcomes were reached at follow-up. In any case,
the relative lack of variability in this outcome reduced the
likelihood of finding significant relationships between
module use and frequency of abstinent days.
The analyses reported here focused on one aspect of
MORE usage, which was the total number of modules
accessed by each participant. This measure was the only one
available at the time, so we could not report other measures
such as number of program logins or number of visits to
components other than the modules. On a further method-
ological note, the measure of self-efficacy used here differed
somewhat from measures used in other studies. Self-efficacy
is typically referred to as confidence in one's “ability to stay
sober”; in the MORE program, it is phrased in terms of
confidence in one's “ability to work a recovery program.”
Despite limitations inherent with administrative data sets,
these analyses have several strengths. This is the first
evaluation we are aware of to examine usage of a unique and
innovative Web-based continuing care program for patients
with alcohol or drug dependence. The results provide
support for a positive relationship between use of the
MORE program and substance use outcomes after treatment
and suggest that computerized programs may hold promise
as continuing care interventions. This study also used both
bivariate tests and mixed-model regression analyses to
explore the relationship between program use and outcomes
and conducted analyses on compliant patients to account for
the possibility that the relationship between MORE use and
outcomes merely reflected a relationship between outcomes
and compliant behavior in general. Future evaluations will
further examine how usage impacts outcomes and how
program engagement can be increased.
American Psychiatric Association. (2000). Diagnostic and statistical
manual of mental disorders, (Revised 4th ed.). Washington, DC: Author.
Bickel, W. K., Buchhalter, A. R., Marsch, L. A., & Badger, G. J. (2008).
Computerized behavior therapy for opioid-dependent outpatients: A
Cacciola, J. S., Camilleri, A. C., Carise, D., Rikoon, S. H., McKay, J. R.,
McLellan, A. T., et al. (2008). Extending residential care through
telephone counseling: Initial results from the Betty Ford Center Focused
Continuing Care protocol. Addictive Behaviors, 33, 1208–1216.
Carroll, K. M., Ball, S. A., Martino, S., Nich, C., Babuscio, T., Gordon,
M. A., et al. (2008). Computer-assisted cognitive–behavioral therapy
for addiction. A randomized clinical trial of “CBT4CBT”. American
Journal of Psychiatry, 165, 881–888.
Carroll, K. M., Ball, S. A., Martino, S., Nich, C., Babuscio, T. A., &
Rounsaville, B. J. (2009). Enduring effects of a computer-assisted
9 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx
training program for cognitive behavioral therapy: A six-month follow- Download full-text
up of CBT4CBT. Drug and Alcohol Dependence, 100, 178–182.
Cunningham, J. A., Humphreys, K., Kypri, K., & van Mierlo, T. (2006).
Formative evaluation and three-month follow-up of an online person-
alized assessment feedback intervention for problem drinkers. Journal
of Medical Internet Research, 8, e5.
Diggle, P. J., Heagerty, P., Liang, K. Y., & Zeger, S. L. (2002). The analysis
of longitudinal data, (2nd ed.). Oxford, England: Oxford University
Eisen, S. V., Dill, D. L., & Grob, M. C. (1994). Reliability and validity of a
brief patient report instrument for psychiatric outcome evaluation.
Hospital and Community Psychiatry, 45, 242–247.
Farvolden, P., Denisoff, E., Selby, P., Bagby, R. M., & Rudy, L. (2005).
Usage and longitudinal effectiveness of a Web-based self-help cognitive
behavioral therapy program for panic disorder. Journal of Medical
Internet Research, 7, e7.
B., et al. (2005). Randomized effectiveness trial of a computer-assisted
intervention to improve diabetes care. Diabetes Care, 28, 33–39.
Gustafson, D. H., McTavish, F. M., Stengle, W., Ballard, D., Hawkins, R.,
Shaw, B. R., et al. (2005). Use and impact of eHealth system by low-
income women with breast cancer. Journal of Health Communication,
10(Suppl 1), 195–218.
Hester, R. K., Squires, D. D., & Delaney, H. D. (2005). The Drinker's
Check-Up: 12-Month outcomes of a controlled clinical trial of a stand-
alone software program for problem drinkers. Journal of Substance
Abuse Treatment, 28, 159–169.
Humphreys, K., Kaskutas, L. A., & Weisner, C. (1998). The Alcoholics
Anonymous Affiliation Scale: Development, reliability, and norms for
diverse treated and untreated populations. Alcoholism: Clinical and
Experimental Research, 22, 974–978.
Kypri, K.,Langley, J.D., Saunders, J.B., Cashell-Smith, M. L., & Herbison,
P. (2008). Randomized controlled trial of Web-based alcohol screening
and brief intervention in primary care. Archives of Internal Medicine,
Lash, S. J., Burden, J. L., DeMarce, J. M., Jones, M. E., Lozano, B. E.,
Fearer, S. A., et al. (2007). Contracting, prompting, and reinforcing
substance use disorder continuing care: A randomized clinical trial.
Psychology of Addictive Behaviors, 21, 387–397.
Littell, R. C., Milliken, G. A., Stroup, W. W., & Wolfinger, R. D. (1996).
SAS system for mixed models. Cary, NC: SAS Institute, Inc.
McKay, H. G., Danaher, B. G., Seeley, J. R., Lichtenstein, E., & Gau, J. M.
(2008). Comparing two Web-based smoking cessation programs:
Randomized controlled trial. Journal of Medical Internet Research, 10
McKay, J. R. (2005). Is there a case for extended interventions for alcohol
and drug use disorders? Addiction, 100, 1594–1610.
McKay, J. R. (2009a). Treating substance use disorders with adaptive
continuing care. Washington, DC: American Psychological Association
McKay, J. R. (2009b). Continuing care research: What we've learned and
where we're going. Journal of Substance Abuse Treatment, 36,
Moore, B. A., Fazzino, T., Garnet, B., Cutter, C. J., & Barry, D. T. (2011).
Computer-based interventions for drug use disorders: A systematic
review. Journal of Substance Abuse Treatment, 40, 215–223.
Project MATCH Research Group. (1993). Project MATCH (Matching
Alcoholism Treatment to Client Heterogeneity): Rationale and methods
for a multisite clinical trial matching patients to alcoholism treatment.
Alcoholism: Clinical and Experimental Research, 17, 1130–1145.
Strecher, V. J., McClure, J., Alexander, G., Chakraborty, B., Nair, V.,
Konkel, J., et al. (2008). The role of engagement in a tailored Web-based
smoking cessation program: Randomized controlled trial. Journal of
Medical Internet Research, 10, e36.
Tonigan, J. S., Miller, W. R., & Brown, J. M. (1997). The reliability of the
Form 90: An instrument for assessing alcohol treatment outcome.
Journal of Studies on Alcohol, 58, 358–364.
Verheijden, M., Bakx, J. C., Akkermans, R., van den Hoogen, H., Godwin,
N. M., Rosser, W., et al. (2004). Web-based targeted nutrition
counseling and social support for patients at increased cardiovascular
risk in general practice: Randomized controlled trial. Journal of Medical
Internet Research, e44, 6.
Williams, G. C., Lynch, M., & Glasgow, R. E. (2007). Computer-assisted
intervention improves patient-centered diabetes care by increasing
autonomy support. Health Psychology, 26, 728–734.
Witbrodt, J., & Kaskutas, L. A. (2005). Does diagnosis matter? Differential
effects of 12 step participation and social networks on abstinence.
American Journal of Drug and Alcohol Abuse, 31, 685–707.
10 A.A. Klein et al. / Journal of Substance Abuse Treatment xx (2011) xxx–xxx