Content uploaded by Jatinder Takhar
Author content
All content in this area was uploaded by Jatinder Takhar on Jan 25, 2018
Content may be subject to copyright.
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6
883366
The provision of psychiatric
services to rural and geo-
graphically isolated regions
challenges most health systems (1).
Countries with shortages of psychia-
trists find it inefficient for psychia-
trists to travel long distances to see
few patients in remote communities.
Conversely, it is expensive and often
not feasible for patients from remote
regions to travel to urban centers for
psychiatric care. Interactive video-
conferencing, often called telepsy-
chiatry, is a potential solution to this
problem (2). However, the efficacy
of telepsychiatry to provide clinical
psychiatric services to distant com-
munities has yet to be definitively
established (3).
Researchers have established that
telepsychiatry can provide a reliable
diagnosis of common psychiatric dis-
orders (4,5) and accurate assessments
of cognitive (6), depressive (7), anxi-
ety (7), and psychotic symptoms (8).
However, the assessment of symp-
toms, such as emotional affect, that
require visual observation of behavior
appear less reliable (9), especially
when using bandwidths of 128 kilo-
bits per second or less (8,10).
The ultimate test of telepsychiatry
is whether it can produce clinical out-
comes that are at least equivalent to
those achieved through face-to-face
service. Two small studies examining
clinical outcomes showed no statisti-
cally significant differences in out-
comes between patients seen via
Is Telepsychiatry Equivalent to Face-to-Face
Psychiatry? Results From a Randomized
Controlled Equivalence Trial
RRiicchhaarrdd OO’’RReeiillllyy,, MM..BB..,, FF..RR..CC..PP..CC..
JJooaann BBiisshhoopp,, MM..DD..,, FF..RR..CC..PP..CC..
KKaarreenn MMaaddddooxx,, RR..NN..,, MM..AA..
LLooiiss HHuuttcch
hiinnssoonn,, MM..DD..,, FF..RR..CC..PP..CC..
MMiicchhaaeell FFiissmmaann,, MM..BB..,, FF..RR..CC..PP..CC..
JJaattiinnddeerr TTaakkhhaarr,, MM..DD..,, FF..RR..CC..PP..CC..
Dr. O’Reilly, Dr. Fisman, and Dr. Takhar are affiliated with the Department of Psychia-
try, Regional Mental Health Care, St. Joseph’s Health Care, P.O. Box 5532, Station B, 850
Highbury Ave. N., London, Ontario N6A 4H1, Canada (e-mail: richard.o’reilly@sjhc.lon
don.on.ca). They are also with the Department of Psychiatry, University of Western On-
tario, London, Ontario. Dr. Bishop is with the Department of Psychiatry, University of
British Columbia, Vancouver, British Columbia. Ms. Maddox and Dr. Hutchinson are af-
filiated with the Department of Psychiatry, Thunder Bay Regional Health Sciences Cen-
tre, Thunder Bay, Ontario.
Objective: The use of interactive videoconferencing to provide psychi-
atric services to geographically remote regions, often referred to as
telepsychiatry, has gained wide acceptance. However, it is not known
whether clinical outcomes of telepsychiatry are as good as those
achieved through face-to-face contact. This study compared a variety
of clinical outcomes after psychiatric consultation and, where needed,
brief follow-up for outpatients referred to a psychiatric clinic in Cana-
da who were randomly assigned to be examined face to face or by
telepsychiatry. Methods: A total of 495 patients in Ontario, Canada,
referred by their family physician for psychiatric consultation were
randomly assigned to be examined face to face (N=254) or by telepsy-
chiatry (N=241). The treating psychiatrists had the option of providing
monthly follow-up appointments for up to four months. The study test-
ed the equivalence of the two forms of service delivery on a variety of
outcome measures. Results: Psychiatric consultation and follow-up de-
livered by telepsychiatry produced clinical outcomes that were equiv-
alent to those achieved when the service was provided face to face. Pa-
tients in the two groups expressed similar levels of satisfaction with
service. An analysis limited to the cost of providing the clinical service
indicated that telepsychiatry was at least 10% less expensive per pa-
tient than service provided face to face. Conclusions: Psychiatric con-
sultation and short-term follow-up can be as effective when delivered
by telepsychiatry as when provided face to face. These findings do not
necessarily mean that other types of mental health services, for exam-
ple, various types of psychotherapy, are as effective when provided by
telepsychiatry. (Psychiatric Services 58:836–843, 2007)
ore.qxd 5/21/2007 10:49 AM Page 836
telepsychiatry and those seen in per-
son (11,12). In the only reported ran-
domized controlled trial of telepsy-
chiatry for adults, 119 U.S. veterans
with depression were randomly as-
signed to six months of outpatient
treatment in person or by telepsychi-
atry (13). They received medications,
psychoeducation, and supportive
counseling. No between-group dif-
ferences were observed in depressive
symptoms, adherence to appoint-
ments, adherence to medication regi-
mens, dropout rates, or satisfaction
levels. A second small (N=28) ran-
domized controlled trial of telepsy-
chiatry evaluating an eight-week
course of cognitive-behavioral thera-
py for children reported “decreasing
symptoms of childhood depression
over videoconferencing at rates com-
parable to face-to-face” (14).
All published outcome studies, in-
cluding the randomized controlled
trial of adults with depression (13),
used comparative methods rather
than equivalence methods for the de-
termination of sample size, analysis of
data, and interpretation of results.
Comparative studies often incorrectly
conclude that a failure to detect a sta-
tistically significant difference in out-
come implies “equivalence.” Howev-
er, the lack of a statistically significant
difference does not mean equiva-
lence (15,16). “While non-significant
P values from tests of equality indi-
cate that trial results have not conclu-
sively established the superiority of
an experimental regimen, these tests
do not address the issue of equiva-
lence which requires the use of confi-
dence intervals” (17) and predeter-
mined equivalence margins (15).
We overcame the limitation of pre-
vious telepsychiatry research using
methods specially designed to test
whether two interventions are equiv-
alent. We describe a variety of clinical
outcomes among outpatients with
mixed psychiatric diagnoses referred
by their family physician and random-
ly assigned to receive psychiatric con-
sultation and short-term follow-up ei-
ther in person or by telepsychiatry.
We predicted that patients referred
by their family physician for a psychi-
atric consultation and, if needed,
short-term follow-up would have
equivalent clinical outcomes regard-
less of whether they were seen via
telepsychiatry or in person. We also
predicted that telepsychiatry would
be less expensive than in-person care.
Methods
The research ethics committees of
the University of Western Ontario
and the Thunder Bay Regional Hos-
pital reviewed and approved this
study. All participants were provided
with written and verbal information
about the nature of the study and
consented to take part.
Study design
Participants were sampled from re-
ferrals to the psychiatric consultation
clinic of the Thunder Bay Regional
Hospital. We used a sample size cal-
culation and analytical methods that
are designed for equivalence trials
(15). In this process, we start by pre-
specifying delta (∆), the absolute val-
ue of the difference that could be
found between telepsychiatry and
face-to-face care and still conclude
that the two interventions are equiva-
lent. This is called the equivalence
margin and –∆to +∆is the range
within which ∆can vary and still be of
no clinical importance (15). Because
our main interest is that telepsychia-
try is not inferior to face-to-face care,
we are concerned with the lower lim-
it of this range (–∆), and in our analy-
sis, we check whether the confidence
interval (CI) for the difference be-
tween the groups on various outcome
measures is less than –∆.
Sample size was determined for
our primary outcome, the difference
in proportions of patients moving
from dysfunctional (case) to function-
al (noncase) status on the Brief Symp-
tom Inventory (BSI) (18) four months
after initial psychiatric consultation.
We used expert clinical judgment
(15) to choose the lower limit of the
difference in proportions, which
would still be consistent with clinical
equivalence (–∆=–.15) (17). If the
difference in improvement between
intervention and control groups is less
than this predetermined equivalence
margin, the treatments would be con-
sidered equally effective or equiva-
lent, even though one can never actu-
ally “prove” equivalence (19).
With alpha error of .05 (one sided)
and statistical power of 80%, we used
the sample size formula from Jones
and colleagues (15) for the one-sided
case for comparison of proportions in
equivalence trials, indicating a re-
quirement of 138 patients per group,
or 276 patients total.
We experienced a high loss to fol-
low-up that required recruitment of
more participants to achieve the full
sample of completers. The first 42 pa-
tients were allocated to groups by flip
of a coin, and block randomization
(that is, using random numbers gen-
erated by computer algorithm in
blocks of eight) was subsequently em-
ployed to control for any change of
referral pattern over the 30-month
duration of the study.
Inclusion criteria
Patients were eligible if they were
aged 18 to 65, from the Thunder Bay
region (officially designated as an un-
derserviced area for psychiatry), and
referred by a family doctor to the psy-
chiatric outpatient department of the
local general hospital. Patients were
excluded if their family doctor consid-
ered them incapable of consenting to
the study or if the referral was prima-
rily for a medico-legal or insurance
assessment. All eligible patients re-
ceived a letter explaining the nature
of the study. Shortly afterward they
were contacted by telephone by a re-
search assistant, who is an experi-
enced registered nurse. The research
assistant answered questions about
the study and, if the patient was will-
ing to proceed, completed the BSI
over the telephone. Only patients
who had an initial BSI score in the
dysfunctional range were randomly
assigned to one of the study groups.
Equipment
The interactive videoconference equip-
ment consisted of a Polycom 512
View Station and a Sony Trinitron
68.5-centimeter diagonal screen. The
connection was made by using three
ISDN lines delivering a bandwidth of
384 kilobits per second.
Clinical services
Participants in each arm of the study
attended the Thunder Bay Regional
Hospital for service. Four psychia-
trists from London, Ontario, located
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6 883377
ore.qxd 5/21/2007 10:49 AM Page 837
approximately 1,000 kilometers from
Thunder Bay, provided the service.
They traveled by air to Thunder Bay
to provide in-person service and con-
nected via interactive videoconfer-
ence from London for the telepsychi-
atry service. Each psychiatrist was as-
signed an equal number of partici-
pants in each arm of the study.
The clinical service provided in the
study was modeled on the outpatient
service already in use in Thunder Bay.
The psychiatrists sent a handwritten
form with recommendations to the
family physician by facsimile within
48 hours of the initial assessment, fol-
lowed by a full typed report. The psy-
chiatrists decided, on the basis of
clinical need, whether patients should
return for follow-up visits. Patients
could be seen within the protocol for
up to four months after the initial as-
sessment. When needed, follow-up
visits were scheduled at monthly in-
tervals for each group.
Treatment recommendations in-
cluded medication management, psy-
choeducation, supportive counseling,
and triage to other local services. In
the vast majority of cases the pre-
scription of medication, recommend-
ed by the psychiatrists, was undertak-
en by the family physician. This is typ-
ical of practice in underserviced areas
where telepsychiatry would most of-
ten be used. However, the psychia-
trists could prescribe directly when
the need arose. The psychiatrists fre-
quently referred patients to a short-
term psychotherapy program, com-
munity-based case-management ser-
vices, various self-help groups, and
recreation programs. The psychia-
trists were instructed to provide serv-
ices in the same manner to partici-
pants in each group. As a check on the
similarity of services provided to the
two groups, the research assistant re-
viewed the handwritten forms filled
out by the psychiatrist and collected
data on whether medications or refer-
rals to other community services were
recommended.
Research scales
The BSI is a 53-item self-report psy-
chological symptom inventory with
lower scores indicating fewer symp-
toms (18). The Global Severity Index
(GSI) subscale of the BSI is calculat-
ed from the raw scores and is the
most sensitive indicator of distress
from psychiatric symptoms. Raw
scores are converted into standard-
ized T scores. The operational rule
for classifying a patient as a “case” is if
the respondent has a GSI score
greater than or equal to a T score of
63 or if any two of the nine primary
dimensions are greater than or equal
to a T score of 63. When the individ-
ual is classified as a case he or she is in
the dysfunctional range (20). The rec-
ommended brief standardized in-
structions were provided to the par-
ticipant on the telephone for the
screening contact and later in the
mail for the four-month follow-up by
the same trained research assistant.
On the initial assessment visit partic-
ipants completed the Medical Out-
comes Study Short Form (SF-36), a
self-report health survey with 36 ques-
tions, which is suitable for self-admin-
istration or administration by a trained
interviewer in person or by telephone
(21). It yields an eight-scale profile of
scores. For this study, the full SF-36
was administered even though we
planned to use only the five-item men-
tal health subscale, because this is the
usual manner of scale completion.
Scores on the mental health subscale
range from 0 to 100, with higher scores
indicating higher functioning. Because
this study was done in Canada, we
used the Canadian norm-based scores,
which are standardized to 50 by using
Canadian weights (22).
The Client Satisfaction Question-
naire (CSQ-8) is an eight-item self-
report scale with brief written in-
structions. Scores range from 8 to 32,
with higher scores indicating higher
satisfaction (23–26).
At four months, the BSI, SF-36,
and CSQ-8 were mailed to all partici-
pants with a stamped addressed re-
turn envelope. All scales included
standardized instructions for self-re-
port. Two further mailings were sent
to noncompleters. In a pilot project
for measuring outcomes for telepsy-
chiatry, we found that the procedures
for administration of the BSI and SF-
36 on the telephone, in person, and in
the mail yielded baseline and four-
month scores with expected values
and direction of change (27). Thus it
was deemed to be valid to administer
the baseline BSI on the telephone
and the four-month BSI via mail for
this full study.
Admissions
Psychiatric admissions to the Thun-
der Bay general hospital and the re-
gional psychiatric hospital were noted
for all participants. It was not possible
to determine admissions to psychi-
atric units outside of the Thunder Bay
region. However, the geographical lo-
cation, 480 kilometers from the near-
est alternative psychiatric unit, makes
it unlikely that many patients were
admitted elsewhere. There is no rea-
son to believe that participants in ei-
ther group would be more likely to
leave the region.
Costs
Details of fees paid to the psychia-
trists and of reimbursement of ex-
penses the psychiatrists incurred (on
the basis of receipts) when traveling
to and staying in Thunder Bay were
collected throughout the study. ISDN
line rental charges and per minute
connection costs were available from
telephone company monthly billing
invoices. The capital cost of the
videoconference equipment was de-
preciated over a five-year period. All
participants attended the same office
in the outpatient department of the
distal site for clinical service. Howev-
er, an additional office was used at the
proximal site to house the videocon-
ference equipment, and market rate
rental for this office was added to the
costs of telepsychiatry.
Outcome variables
The primary outcome is the propor-
tion of participants whose BSI score
moved from dysfunctional to func-
tional range (that is, the patient
moved from being classified as a case
to being classified as a noncase). The
BSI was chosen because it measures
overall distress from psychiatric symp-
toms, and distress is an important rea-
son that patients seek help. Secondary
outcome variables were the propor-
tion of participants with any psychi-
atric admission during the 12 months
after the initial assessment, change in
scores on the GSI subscale of the BSI,
change in scores on the mental health
subscale of the SF-36 (21) standard-
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6
883388
ore.qxd 5/21/2007 10:49 AM Page 838
ized on a Canadian population (22),
score on the CSQ-8 at four months,
and cost of providing psychiatric as-
sessment and follow-up.
Analysis
The two groups were compared on
major baseline variables to check if
the randomization worked. We used
equivalence methods (15) to analyze
outcome data for scores on the BSI,
hospital admissions, scores on the
GSI subscale of BSI, scores on the
mental health subscale of SF-36, and
scores on the CSQ-8.
BSI. We constructed a CI cen-
tered about the observed difference
in proportions of participants mov-
ing from dysfunctional to functional
status on the BSI from baseline to
four months. Because the objective
was to ensure that telepsychiatry is
not inferior to in-person service, a
lower one-sided 95% CI was con-
structed by using the method of
Pagano and colleagues (28).
Admissions. For comparing propor-
tions of participants in each group with
at least one admission during the 12
months after initial consultation, we
used methods suitable for testing
equivalence of two proportions with
the predetermined lower limit of the
equivalence margin (–∆=–.10). Clini-
cal consensus was that a difference of
10% or less was a conservative esti-
mate, as decided by research team
members in consultation with psychia-
trist colleagues working in both clinical
and research practice. We calculated
cumulative hospital days for each
group to help understand the results.
We did not use mean number of ad-
missions or days in the hospital, as we
anticipated (and found) skewed data.
GSI. We also tested for equivalence
of mean improvement in scores on
the GSI, the BSI subscale, which is
the most sensitive single indicator of
distress (20). We used the lower limit
of the predetermined equivalence
margin (–∆=–5) to be conservative,
because a difference in scores on the
GSI of 7 is considered to be clinically
significant (18,29).
Mental health subscale of SF-36. A
5-point variation in the score on the
mental health subscale is considered
the smallest clinically significant dif-
ference. To detect this, we needed
132 participants per group (30),
which is compatible with the number
for testing the primary outcome. We
set the lower limit of the predeter-
mined equivalence margin (–∆) at –5
and used SPSS to obtain the lower
one-sided 95% CI centered around
the observed difference in the mean
improvement scores on the mental
health subscale.
CSQ-8. We constructed a lower one-
sided CI centered around observed
differences in mean four-month scores
using SPSS. The lower limit of the
equivalence margin (–∆) was set at –2
by using clinical expertise to extrapo-
late from literature, such as the study
by Gill and colleagues (31), who found
a main effect when CSQ-8 scores var-
ied from 3 to 4 points (22.4–26.4).
Costs. We calculated the average
cost per patient by simply dividing
the total cost per group by the num-
ber of patients randomly assigned to
that group. Because it was not possi-
ble to attribute costs to individual
patients, standard deviations are not
available. Because participants went
to the same hospital for telepsychia-
try and face-to-face services, they in-
curred no differences in expenses in
either arm of the study.
Results
Disposition and
characteristics of patients
The study was conducted between
2001 and 2004. Figure 1 depicts the
enrollment, random assignment, and
follow-up of study patients. Table 1
shows the baseline demographic and
clinical characteristics of the two
groups. There were no significant dif-
ferences between the groups on base-
line measures or on the measures of
services provided. Completers and
noncompleters in the two groups
were similar, except that more com-
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6 883399
FFiigguurree 11
Enrollment and follow-up patterns of patients referred for psychiatric consultation
834
met initial
screening criteria
339 excluded
125 could not contact
38 refused any service
31 refused participation in research
71 refused telepsychiatry
47 already attending another service
27 did not meet Brief Symptom
Inventory criteria
495
underwent randomization
254
face to face
241
telepsychiatry
246
had a
psychiatric visit
8
no show
148
complete
follow-up
assessments
98
incomplete
follow-up
assessments
224
had a
psychiatric visit
17
no show
138
complete
follow-up
assessments
86
incomplete
follow-up
assessments
ore.qxd 5/21/2007 10:49 AM Page 839
pleters in both groups were married
and noncompleters in the face-to-
face group used more hospital days
than completers, whereas in the
telepsychiatry group, the opposite
was true. These differences would
not challenge our hypothesis.
Clinical outcomes
and hospital use
Tables 2 and 3 show the results of
equivalence testing for the primary
and secondary outcome measures, us-
ing the predetermined equivalence
margins. All results support the hy-
pothesis that telepsychiatry produces
equivalent outcomes to face-to-face
care.
As expected, scores on the BSI
(GSI subscale) and SF-36 (mental
health subscale) showed that patients
reported less distress from symptoms
and improved mental health after the
clinical intervention in both groups.
The levels of improvement were con-
sistent with those considered to be
clinically and socially relevant by the
authors of these scales (18,29,30) and
in the literature in which the BSI has
been used to measure outcomes for
patients similar to those in our trial
(32). The CSQ-8 indicated a moder-
ate degree of satisfaction (33). We
conducted an intent-to-treat analysis
for the proportion of participants hos-
pitalized within 12 months from ini-
tial consultation, and this analysis
showed equivalence between the two
groups.
Costs
Table 4 shows that face-to-face servic-
es required travel and accommoda-
tion expenses for the psychiatrists that
were unnecessary when using telepsy-
chiatry. Face-to-face services also re-
quired larger fees for psychiatrists to
compensate for travel time. These
costs were greater than the technical
costs of telepsychiatry. The average
cost of telepsychiatry was 10% less per
patient (16% less per visit) than the
cost of in-person service.
Discussion
Using equivalence methods, we
demonstrated that psychiatric consul-
tation and short-term follow-up pro-
vided by telepsychiatry can produce
clinical outcomes that are equivalent
to those achieved when patients are
assessed and followed in-person. On
the primary outcome, approximately
20% of each group moved from a dys-
functional to functional rating. This is
a modest proportion because we used
a stringent test of effectiveness: the
change from a positive psychiatric di-
agnosis to functional status, or a pa-
tient’s moving from being a case to a
noncase (20). The GSI baseline, four-
month, and improvement scores were
similar in magnitude to those found
in a study of different types of psy-
chotherapy for major depressive dis-
order (32). In addition, our finding of
clinically significant improvements
and equivalence in the primary out-
come is supported by the analysis of
the other outcomes, as measured by
hospitalization and mean improve-
ment in the GSI and mental health
subscales.
The clinical service provided via
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6
884400
TTaabbllee 11
Demographic and clinical characteristics of 495 patients referred for psychiatric
consultation, by method of consultation
Face to face Telepsychiatry
(N=254) (N=241)
Characteristic N % N %
Age (years)
18–24 47 19 30 12
25–34 55 22 65 27
35–44 88 35 84 35
45–54 48 19 38 16
55–65 16 6 24 10
Sex
Male 94 37 89 37
Female 160 63 152 63
Marital status
Married 88 35 81 34
Widow 1 <1 2 1
Single 96 38 77 32
Common law 16 6 21 9
Separated or divorced 48 19 48 20
Unknowna5 2 12 5
Primary diagnosis
Depression 138 54 138 57
Bipolar disorder 24 9 22 9
Adjustment disorder 21 8 12 5
Other anxiety disorder 20 8 16 7
Psychosis 11 4 4 2
Alcohol or drug abuse or dependence 8 3 7 3
Posttraumatic stress disorder 7 3 4 2
Adult attention deficit disorder 4 2 7 3
Phobic disorder 3 1 3 1
Personality disorder 3 1 1 <1
Eating disorder 2 1 1 <1
Rule out somatic disorder 2 1 2 1
Obsessive compulsive disorder 1 <1 3 1
Relationship problems 1 <1 1 <1
Other disorders 1 <1 3 1
Missinga8 3 17 7
Services providedb
Medications recommended 230 93 210 94
Community referral made 119 48 119 53
Baseline score on the Global Severity
Index of the Brief Symptom
Inventory (M±SD)c57.5±9.7 57.6±10.1
aWhen patients did not attend the initial assessment, diagnosis for all and marital status for most
could not be assigned.
bData available for 246 patients in the face-to-face group and 224 patients in the telepsychiatry
group
cPossible scores range from 30 to 80, with higher scores indicating more distress.
ore.qxd 5/21/2007 10:49 AM Page 840
telepsychiatry was less expensive than
when it was provided in person. This
finding coupled with the equivalent
clinical outcomes suggests that
telepsychiatry can be a cost-effective
method for delivering psychiatric
services. Our study provided services
to a remote community, which re-
quired air travel and overnight stays.
As noted elsewhere (34), the relative
cost of telepsychiatry and in-person
care is influenced by several factors,
such as the distance traveled, volume
of patients, and the type of technolo-
gy. Therefore, the cost savings to the
service provider in this study, may not
be realized in other settings. Further-
more, the costs in the study present-
ed here were assessed solely from the
perspective of the provider. In this
study, the patients traveled to the
Thunder Bay Regional Hospital irre-
spective of whether they received
service in-person or via telepsychia-
try, and therefore there is no reason
that patients’ travel expenses and
time taken from work would differ.
Other research has suggested that
telepsychiatry, used under certain
conditions, can reduce cost to service
users (35).
A major strength of our study was
that it was conducted in a remote, un-
derserviced area and thus replicated
the conditions in which telepsychiatry
is most likely to be used. We mini-
mized exclusion criteria to ensure the
inclusion of a broad range of patients,
similar to the usual referrals from pri-
mary care physicians to psychiatrists.
However, the naturalistic nature of
the service also produced limitations.
Because of the broad inclusion crite-
ria, we did not limit psychiatric care
protocols to a carefully defined, diag-
nosis-specific, therapeutic interven-
tion. Psychiatrists were instructed to
provide the same type and level of
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6 884411
TTaabbllee 22
Follow-up results of rates of hospitalization and return to functional status among patients referred for psychiatric
consultation, by method of consultation
Lower one-sided Predetermined
Difference in 95% CI for equivalence
Measure N Proportion proportion difference margin (–∆)a
Return to functional score on the
Brief Symptom Inventory by four months
Face to face (N=148) 29 .20 .02 –.10 –.15
Telepsychiatry (N=138) 30 .22
Any hospitalization in a psychiatric
unit in year after initial assessment
Face to face (N=246) 18 .073 .01 –.03 –.10
Telepsychiatry (N=224) 15 .066
a∆represents the absolute value of the difference that could be found between telepsychiatry and face-to-face care and still conclude that the two in-
terventions are equivalent. Because we are only interested in whether telepsychiatry is not inferior to face-to-face care, we used the lower limit (–∆).
If the difference in improvement between intervention and control groups is less than this predetermined equivalence margin, the treatments would
be considered equally effective or equivalent.
TTaabbllee 33
Follow-up results on various measures among patients referred for psychiatric consultation, by method of consultation
Four-month
Baseline score score Improvement Lower one- Predetermined
Difference sided 95% CI equivalence
Measure Mean SD Mean SD Mean SD in means for difference margin (–∆)a
GSIb
Face to face (N=148) 56.5 10.1 49.7 13.3 6.9 9.1 –.3 –2.6 –5.0
Telepsychiatry (N=138) 56.9 10.2 49.7 12.6 7.2 9.8
MHc
Face to face (N=148) 24 12.4 30.9 15.7 6.9 13.9 –1.0 –4.7 –5.0
Telepsychiatry (N=138) 23.8 12.0 31.7 14.2 7.9 13.0
CSQd
Face to face (N=129) — — 23.0 5.7 — — .3 –1.2 –2.0
Telepsychiatry (N=125) — — 22.7 6 — —
a∆represents the absolute value of the difference that could be found between telepsychiatry and face-to-face care and still conclude that the two in-
terventions are equivalent. Because we are only interested in whether telepsychiatry is not inferior to face-to-face care, we used the lower limit (–∆).
If the difference in improvement between intervention and control groups is less than this predetermined equivalence margin, the treatments would
be considered equally effective or equivalent.
bGlobal Severity Index of the Brief Symptom Inventory. Possible scores range from 30 to 80, with higher scores indicating more distress.
cMental health subscale of the 36-item Medical Outcomes Study Short Form. Possible scores range from 0 to 100, with higher scores indicating high-
er functioning.
dClient Satisfaction Questionnaire. Possible scores range from 8 to 32, with higher scores indicating higher satisfaction.
ore.qxd 5/21/2007 10:49 AM Page 841
service to patients seen in person and
by telepsychiatry. Furthermore, data
on services provided (Table 1) sug-
gests that similar care was actually
provided to both groups. However, it
is still possible that there may have
been subtle differences in the way pa-
tients in each group were managed.
A second limitation was the high
rate of noncompletion of the four-
month research scales. Although we
continued to recruit patients until we
had the required number of partici-
pants, only 58% of participants ini-
tially randomized to the groups com-
pleted these scales. As a result we
were able to do only a per-protocol
analysis on these outcomes. Howev-
er, a per-protocol analysis is consid-
ered by many to be more important
than an intent-to-treat analysis for
equivalence trials (15). It is impor-
tant to note that we were able to per-
form an intent-to-treat analysis on
the risk of hospitalization in the year
after consultation, for which we had
full data, and this analysis also
showed equivalence.
The low completion rate was prob-
ably influenced by the fact that con-
sultation was available more quickly
through the study than through reg-
ular local services and that most par-
ticipants had ended their clinical
contact a number of months before
they were required to complete the
final research scales. These factors
likely contributed to the recruitment
of a cohort of participants with low
motivation to complete the research
component of the intervention.
We did not measure satisfaction
with the technical components of
telepsychiatry as has been done in
other studies. This would have been
possible only for the telepsychiatry
group. Rather, we used the opportu-
nity provided by the randomized
controlled trial to compare satisfac-
tion with the clinical service provid-
ed to the two groups using a standard
questionnaire, the CSQ-8. The re-
sults demonstrated equivalent levels
of satisfaction in both face-to-face
and telepsychiatry groups.
Ten percent of patients who were
initially contacted refused participa-
tion in the study because of an un-
willingness to use telepsychiatry.
This figure is lower than the 33% of
residents of a rural area of Iowa who
said that they would be unwilling to
use telepsychiatry if they needed
mental health services (36). The
higher rate in the Iowa study may be
because the researchers surveyed a
general community population
rather than individuals referred for
psychiatric assessment. Neverthe-
less, there appears to be a group of
individuals who are averse to the use
of telepsychiatry. Administrators de-
veloping telepsychiatry programs
may need to maintain some parallel
face-to-face service to meet the
needs of this group.
Conclusions
Our findings indicate that psychi-
atric consultation and short-term fol-
low up provided by telepsychiatry
can produce clinical outcomes that
are equivalent to those achievable
when patients are seen face to face.
In our setting telepsychiatry was less
expensive than face-to-face service,
although the relative cost of the two
modes of service delivery is likely to
be influenced by factors such as the
distance between sites and service
volume. It is important to recognize
that we examined a single psychiatric
service: psychiatric consultation and
short-term follow-up. It is possible
that telepsychiatry may not produce
equivalent outcomes when used to
deliver other mental health services,
such as psychotherapy, which is
more dependent on the therapist-pa-
tient relationship. Nevertheless, the
findings are likely to encourage
those who advocate a more wide-
spread adoption of telepsychiatry to
counter the shortage of psychiatrists
in remote regions.
Acknowledgments and disclosures
This study was supported by grant R2354-A01
from the Ontario Mental Health Foundation
and by NORTH Network. The authors thank
Emmanuel Persad, M.B., for his helpful advice
and assistance and Allan Donner, Ph.D., for his
advice on statistics.
The authors report no competing interests.
References
1. El-Guebaly N, Kingstone E, Rae-Grant Q,
et al: The geographical distribution of psy-
chiatrists in Canada: unmet needs and re-
medial strategies. Canadian Journal of Psy-
chiatry 38:212–216, 1993
2. Brown FW: Rural telepsychiatry. Psychi-
atric Services 49:963–964, 1998
3. Monnier J, Knapp RG, Frueh BC: Recent
advances in telepsychiatry: an updated re-
view. Psychiatric Services 54:1604–1609,
2003
4. Ruskin PE, Reed S, Kumar R, et al: Relia-
bility and acceptability of psychiatric diag-
nosis via telecommunication and audiovisu-
al technology. Psychiatric Services 49:
1086–1088, 1998
5. Elford R, White H, Bowering R, et al: A
randomized, controlled trial of child psy-
chiatric assessments conducted using
videoconferencing. Journal of Telemedi-
cine and Telecare 6:73–82, 2000
6. Ball CJ, Scott N, McLaren PM, et al: Pre-
liminary evaluation of a low-cost video con-
ferencing (LCVC) system for remote cogni-
tive testing of adult psychiatric patients.
British Journal of Clinical Psychology
32:303–307, 1993
7. Baer L, Cukor P, Jenike MA, et al: Pilot
studies of telemedicine for patients with
obsessive-compulsive disorder. American
Journal of Psychiatry 152:1383–1385, 1995
8. Zarate CA Jr, Weinstock L, Cukor P, et al:
Applicability of telemedicine for assessing
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6
884422
TTaabbllee 44
Number of patients and visits and costs in U.S. dollars for psychiatric consultation
among patients referred for psychiatric consultation, by method of consultation
Variable Face to face Telepsychiatry
Payments to psychiatrists $73,636 $45,580
Travel and accommodation for psychiatrists $34,913 $0
Telephone connection costs for telepsychiatry $0 $22,665
Equipment depreciation $0 $10,036
Room rental cost for telepsychiatry in
proximal site $0 $10,030
Total costs $108,549 $88,311
Number of patients 246 224
Number of visits 344 333
Average cost per patient $439 $394
Average cost per visit $315 $265
ore.qxd 5/21/2007 10:49 AM Page 842
PSYCHIATRIC SERVICES ♦ps.psychiatryonline.org ♦June 2007 Vol. 58 No. 6 884433
patients with schizophrenia: acceptance
and reliability. Journal of Clinical Psychia-
try 58:22–25, 1997
9. Jones BN 3rd, Johnston D, Reboussin B, et
al: Reliability of telepsychiatry assessments:
subjective versus observational ratings.
Journal of Geriatric Psychiatry and Neurol-
ogy 14:66–71, 2001
10. Yoshino A, Shigemura J, Kobayashi Y, et al:
Telepsychiatry: assessment of televideo
psychiatric interview reliability with pres-
ent- and next-generation internet infra-
structures. Acta Psychiatrica Scandinavica
104:223–226, 2001
11. Zaylor C: Clinical outcomes in telepsychia-
try. Journal of Telemedicine and Telecare
5(suppl 1):S59–S60, 1999
12. Kennedy C, Yellowlees P: The effectiveness
of telepsychiatry measured using the
Health of the Nation Outcome Scale and
the Mental Health Inventory. Journal of
Telemedicine and Telecare 9:12–16, 2003
13. Ruskin PE, Silver-Aylaian M, Kling MA, et
al: Treatment outcomes in depression:
comparison of remote treatment through
telepsychiatry to in-person treatment.
American Journal of Psychiatry 161:1471–
1476, 2004
14. Nelson EL, Barnard M, Cain S: Treating
childhood depression over videoconferenc-
ing. Telemedicine Journal and E-Health:
The Official Journal of the American
Telemedicine Association 9:49–55, 2003
15. Jones B, Jarvis P, Lewis JA, et al: Trials to
assess equivalence: the importance of rigor-
ous methods. British Medical Journal
313:36–39, 1996
16. Garrett AD: Therapeutic equivalence: fal-
lacies and falsification. Statistics in Medi-
cine 22:741–762, 2003
17. Fleming TR: Design and interpretation of
equivalence trials. American Heart Journal
139:S171–S176, 2000
18. Piersma HL, Reaume WM, Boes JL: The
Brief Symptom Inventory (BSI) as an out-
come measure for adult psychiatric inpa-
tients. Journal of Clinical Psychology
50:555–563, 1994
19. Hauck WW, Anderson S: Some issues in
the design and analysis of equivalence tri-
als. Drug Information Journal 33:109–118,
1999
20. Derogatis LR: BSI Brief Symptom Inven-
tory: Administration, Scoring, and Proce-
dures Manual, 4th ed. Minneapolis, Na-
tional Computer Systems, 1993
21. Ware JE Jr, Gandek B: Overview of the SF-
36 Health Survey and the International
Quality of Life Assessment (IQOLA) Pro-
ject. Journal of Clinical Epidemiology
51:903–912, 1998
22. Hopman WM, Towheed T, Anastassiades T,
et al: Canadian normative data for the SF-
36 Health Survey: Canadian Multicentre
Osteoporosis Study Research Group. Cana-
dian Medical Association Journal 163:265–
271, 2000
23. Larsen DL, Attkisson CC, Hargreaves WA,
et al: Assessment of client/patient satisfac-
tion: development of a general scale. Eval-
uation and Program Planning 2:197–207,
1979
24. Attkisson CC, Zwick R: The client satisfac-
tion questionnaire: psychometric proper-
ties and correlations with service utilization
and psychotherapy outcome. Evaluation
and Program Planning 5:233–237, 1982
25. Attkisson CC, Greenfield TK: The UCSF
Client Satisfaction Scales: I. the Client Sat-
isfaction Questionnaire-8, in The Use of
Psychological Testing for Treatment Plan-
ning and Outcomes Assessment, 2nd ed.
Edited by Maruish MA. Mahwah, NJ,
Lawrence Erlbaum Associates, 1999
26. Nguyen TD, Attkisson CC, Stegner BL:
Assessment of patient satisfaction: develop-
ment and refinement of a service evalua-
tion questionnaire. Evaluation and Pro-
gram Planning 6:299–313, 1983
27. Bishop JE, O’Reilly RL, Maddox K, et al:
Client satisfaction in a feasibility study
comparing face-to-face interviews with
telepsychiatry. Journal of Telemedicine and
Telecare 8:217–221, 2002
28. Pagano M, Gavreau K: Principles of Biosta-
tistics. Belmont, Calif, Duxbury Press, 1993
29. Carscaddon DM, George M, Wells G: Rur-
al community mental health consumer sat-
isfaction and psychiatric symptoms. Com-
munity Mental Health Journal 26:309–318,
1990
30. Ware JE: SF-36 Health Survey Manual
and Interpretation Guide. Boston, Health
Institute, New England Medical Centre,
1993
31. Gill KJ, Pratt CW, Librera LA: The effects
of consumer vs staff administration on the
measurement of consumer satisfaction with
psychiatric rehabilitation. Psychiatric Reha-
bilitation Journal 21:365–370, 1998
32. Beutler LE, Engle D, Mohr D, et al: Pre-
dictors of differential response to cognitive,
experiential and self directed psychothera-
peutic procedures. Journal of Consulting
and Clinical Psychology 59:333–340, 1991
33. Greenwood N, Key A, Burns T, et al: Satis-
faction with in-patient psychiatric services:
relationship to patient and treatment fac-
tors. British Journal of Psychiatry 174:159–
163, 1999
34. Hyler SE, Gangure DP: A review of the
costs of telepsychiatry. Psychiatric Services
54:976–980, 2003
35. Simpson J, Doze S, Urness D, et al:
Telepsychiatry as a routine service—the
perspective of the patient. Journal of
Telemedicine and Telecare 7:155–160,
2001
36. Rohland BM, Saleh SS, Rohrer JE, et al:
Acceptability of telepsychiatry to a rural
population. Psychiatric Services 51:672–
674, 2000
ore.qxd 5/21/2007 10:49 AM Page 843
