The effect of computer-based reminders on the management of hospitalized patients with worsening renal function.
ABSTRACT We performed a prospective time-series study to determine whether computerized reminders to physicians about rising creatinine levels in hospitalized patients receiving nephrotoxic and renally excreted medications led to more rapid adjustment or discontinuation of those medications, and to evaluate physician acceptance of computerized reminders. Laboratory data were followed on 10,076 patients over 13,703 admissions generating 1104 events of rising creatinine levels during treatment with nephrotoxic or renally excreted medications. During the intervention period, medications were adjusted or discontinued an average of 21.1 hours sooner (p less than 0.0001) after such an event occurred when compared with the control period. This effect of the reminders was strongest for patients receiving renally excreted and mildly nephrotoxic medications. Of physicians who responded to a computerized survey, 53% said that the reminders had been helpful in the care of their patients, while 31% felt that the reminders were annoying. Seventy-three percent wished to continue receiving computerized reminders. We conclude that computerized reminders are well-accepted in our hospital and have a strong effect on physician behavior.
- SourceAvailable from: PubMed Central[Show abstract] [Hide abstract]
ABSTRACT: Healthcare costs are increasing rapidly and at an unsustainable rate in many countries, and inpatient hospitalizations are a significant driver of these costs. Clinical decision support (CDS) represents a promising approach to not only improve care but to reduce costs in the inpatient setting. The purpose of this study was to systematically review trials of CDS interventions with the potential to reduce inpatient costs, so as to identify promising interventions for more widespread implementation and to inform future research in this area. To identify relevant studies, MEDLINE was searched up to July 2013. CDS intervention studies with the potential to reduce inpatient healthcare costs were identified through titles and abstracts, and full text articles were reviewed to make a final determination on inclusion. Relevant characteristics of the studies were extracted and summarized. Following a screening of 7,663 articles, 78 manuscripts were included. 78.2% of studies were controlled before-after studies, and 15.4% were randomized controlled trials. 53.8% of the studies were focused on pharmacotherapy. The majority of manuscripts were published during or after 2008. 70.5% of the studies resulted in statistically and clinically significant improvements in an explicit financial measure or a proxy financial measure. Only 12.8% of the studies directly measured the financial impact of an intervention, whereas the financial impact was inferred in the remainder of studies. Data on cost effectiveness was available for only one study. Significantly more research is required on the impact of clinical decision support on inpatient costs. In particular, there is a remarkable gap in the availability of cost effectiveness studies required by policy makers and decision makers in healthcare systems.BMC Medical Informatics and Decision Making 12/2013; 13(1):135. · 1.50 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: The evaluation of clinical information systems is essential as they are increasingly used in clinical routine and may even influence patient outcome on the basis of reminder functions and decision support. Therefore we try to answer three questions in this paper: what to evaluate; how to evaluate; how to interpret the results. Those key questions lead to the discussion of goals, methods and results of evaluation studies in a common context. We will compare the objectivist and the subjectivist evaluation approach and illustrate the evaluation process itself in some detail, discussing different phases of software development and potential evaluation techniques in each phase. We use four different practical examples of evaluation studies that were conducted in various settings to demonstrate how defined evaluation goals may be achieved with a limited amount of resources. This also illustrates advantages, limitations and costs of the different evaluation methods and techniques that may be used when evaluating clinical information systems.Journal of Evaluation in Clinical Practice 10/2001; 7(4):373 - 385. · 1.58 Impact Factor
The Effect of Computer-Based Reminders on the Management of
Hospitalized Patients with Worsening Renal Function
David M. Rind, M.D.; Charles Safran, M.D.; Russell S. Phillips, M.D.;
Warner V. Slack, M.D.; David R. Calkins, M.D.,M.P.P.; Thomas L.
Delbanco, M.D.; and Howard L. Bleich, M.D.
From the Center for Clinical Computing and the Divisions of Clinical Computing and General
Medicine, Department of Medicine
Harvard Medical School and Beth Israel Hospital
We performed a prospective time-series study to
determine whether computerized reminders to
physicians about rising creatinine levels in
hospitalized patients receiving nephrotoxic and
renally excreted medications led to more rapid
adjustment or discontinuation of those medications,
and to evaluate physician acceptance of
computerized reminders. Laboratory data were
followed on 10,076 patients over 13,703 admissions
generating 1104 events of rising creatinine levels
during treatment with nephrotoxic or renally
excreted medications. During the intervention
period, medications were adjusted or discontinued
an average of 21.1 hours sooner (p<0.0001) after
such an event occurred when compared with the
control period. This effect of the reminders was
strongest for patients receiving renally excreted and
mildly nephrotoxic medications. Of physicians who
responded to a computerized survey, 53% said that
the reminders had been helpful in the care of their
patients, while 31 % felt that the reminders were
annoying. Seventy-three percent wished to
continue receiving computerized reminders. We
conclude that computerzed reminders are well-
accepted in our hospital and have a strong effect on
McDonald and others have shown that computer-
generated reminders can affect physician behavior
in such areas as preventive care [1-4], repetitive
test ordering , and physician attentiveness to
clinical events [6,7]. These studies were conducted
in the ambulatory setting where the computer can
analyze data the night before a visit and produce a
report for the provider with the patient's visit
Physicians who care for inpatients must process
and deal with large amounts of information from
many sources. A single hospital admission may
generate more patient laboratory data than many
outpatient visits. A hospital computing system can
help physicians by speedin
has the potential to highlight specific information on
inpatients by sending reminders. Unless reminders
provide useful and timely data, however, they may
create distractions that stand between the physician
and desired information. The increased intensity of
care, as well as the large amount of data collected
relative to the outpatient setting, make it uncertain
whether the results of studies on outpatient
reminders can be applied to inpatients.
One study that did look at an inpatient alert
system for potentially life-threatening conditions
detected in laboratory findings suggested positive
effects on both clinician behavior and length of stay
. The investigators felt it necessary to use a
one-group pretest-posttest experimental design and
thus were unable to control for possible underlying
trends in patient care that could have influenced
At our hospital, where clinicians look up patient
data over 40,000 times per week and send more
than 13,000 pieces of electronic mail , the
computer has always provided some feedback on
laboratory data by flagging abnormal values with an
asterisk, critically abnormal values with an
exclamation point, and values that have changed
significantly with a pound sign . The heavy
use of the computer system, including electronic
mail, and the system's integrated database, which
allows real-time analysis of pharmacy and
laboratory information, have made it possible to
better evaluate the effects of computerized
reminders for hospitalized patients.
We have undertaken a time-series controlled trial
of reminders to physicians caring for hospitalized
patients who develop worsening renal function
during treatment with nephrotoxic or renally
excreted medications. We present data from the
first year of the study, looking at effects on
physician behavior, as well as physician acceptance
of these reminders.
The study was performed at a 504-bed major
teaching hospital with a well-established and
heavily used hospital computing system [9-11].
The study had a time-series design, with the first
year of data collected as follows. A first control
period was run for 3 months, an intervention period
was run for 6 months, and a second control period
was run for 3 months.
Patients were assigned to
the control or intervention groups on the basis of
their date of admission.
0195-4210/91/$5.00© 1992 AMIA, Inc.
All patients admitted to the hospital with an
initial creatinine level of 3.0 mg/dl or less and an
age of 18 years or older were eligible for the study.
Definition of Medications
The nephrotoxic and renally excreted medications
referred to in the reminders were those felt to be
clinically important by a senior nephrologist at our
hospital. Nephrotoxic medications included
aminoglycosides, angiotensin converting enzyme
inhibitors, non-steroidal anti-inflammatory drugs,
and other nephrotoxic agents such as amphotencin
B. Renally excreted medications included H2-
blockers, most cephalosporins, and many individual
agents such as digoxin.
An event was defined as a rise in creatinine level
of 0.5 mg/dl or more while a patient was receiving
a nephrotoxic medication or a rise in creatinine
level of 50% or more to 2.0 mgfdl or more while
receiving a renally excreted medication. There
could be at most one event per medication for a
During the control periods, events were recorded
but no action was taken. During the intervention
period, reminders describing the change in the
creatinine level and relevant medications were
automatically sent through the computer mailbox to
physicians caring for the patient whenever an event
occurred (Figure 1). Physigians were given the
option to respond by indicating that the reminder
was "Taken care of." The reminders provided no
suggestion as to a course of action. Reminders
were sent to all physicians who had looked up
information on the patient in the 3 days preceding
the event, as well as the patient's attending
physician. If the medication that had provoked the
event was not changed, and the reminder was not
marked as taken care of by one of these recipients,
then the reminder was also mailed to all new
physicians who looked up information on the
patient during the 3 days following the event.
The effect of the reminders on physician
behavior was evaluated by determining the time
between the occurrence of an event and a change in
the triggering medication or its dosage.
Medications not changed or discontinued by the
time of discharge were considered to have been
discontinued at midnight of the day of discharge.
Physician acceptance of the reminders was
assessed with a computer-administered
questionnaire sent by electronic mail to all
physicians who had received one or more
reminders. The questionnaire, which was written
in the authoring language Converse , assessed
whether the reminders had been irritating or
helpful, had provided useful information, and
whether physicians wished to continue receiving
reminders after the study was concluded. Possible
responses were "Yes, definitely," "Yes, probably,"
"Not sure," "Probably not," and "Definitely not."
Univariate comparisons of patient characteristics
during the intervention and control periods were
performed using t-tests and chi-square tests as
appropriate. DRG cost-weighting was used as a
proxy for severity of illness. The distributions of
times to a change in dosage or discontinuation of a
medication were normalized with a log
transformation before t-tests or linear regression
analysis. When the distributions of such times
were treated as nonparametric, Wilcoxon rank sum
tests were performed. All p values reported reflect
two-tailed tests. In analyzing the questionnaires,
responses of "Yes, definitely" and "Yes, probably"
were pooled and considered a positive response,
and responses of "Not sure" were eliminated.
During the year of data collection, laboratory
data were followed on 10,076 patients who had
74,104 creatinine levels recorded over 13,703
admissions. Over the same period, 508 admissions
were excluded because of an initial creatinine level
above 3.0 mg/dl, and 113 admissions were
excluded because of an age at admission below 18
years. There were 607 events in 348 admissions
during the control periods and 497 events in 303
admissions during the intervention period.
Reminders could include information about
concurrent events in a single patient. The 497
events that occurred during the intervention period
generated 369 reminders that were sent to 584
different physicians, with an average of 9.25
recipients per reminder. On average, these
reminders were sent out 1.7 hours after the blood
specimen for creatinine measurement was logged in
by the laboratory.
During the control and intervention periods,
patients were similar with respect to age, sex, and
severity of illness as suggested by DRG category.
Also similar were the types of medications
(nephrotoxic or renally excreted) and creatinine
levels responsible for triggering reminders (Table
The mean time from an event until
discontinuation of a medication or a change in its
dosage (time to change -- TTC) was 93.7 hours
during the control periods and 72.6 hours during
the intervention period, for a difference of 21.1
The difference in the median
TTC was 15.9 hours. A linear regression model
adjusting for age, sex, severity of illness, date of
admission, and creatinine levels responsible for the
event also showed a significant effect of the
The data were analyzed separately for
nephrotoxins and renally excreted medications,
usmg the regression model. For nephrotoxins the
difference in the means of TTC was 6.7 hours
(p=0.067), while for renally excreted medications
the difference was 35.8 hours (p=0.0001) (Table
II). A subgroup analysis was performed on the
nep,hrotoxins(Table III), classifying them as
angiotensin converting enzyme inhibitors,
aminoglycosides, non-steroidal anti-inflanmmatory
drugs (NSAIDs), and others. Of these, only the
time to change NSAIDs was significantly reduced.
The control periods were analyzed separately to
look for trends in the data due either to underlying
changes in behavior or learning over time as a
result of the reminders. The mean TTC was 93.3
hours during the first control period and 94.0 hours
during the second control period (p=0.67).
The data were also analyzed after exclusion of
events involving medications that were continued
until discharge from the hospital. The difference in
the means of TTC for the remaining 410 events
during the control period and 355 events during the
intervention period was 10.9 hours (p=0.0001).
A computer-administered questionnaire about the
reminders was electronically mailed to all
physicians who had received any reminders through
March 1, 1991. Of 622 questionnaires mailed out,
517 were received by a physician. Of those, 397
(76.8%) were completed. Of the 397 recipients
who completed questionnaires, 347 (87.4%)
remembered receiving reminders and were asked
questions about them. The responses, which are
summarized in Table IV, show that 53% of
physicians with an opinion found the reminders
helpful, while 31 % found them annoying. Seventy-
three percent wished to continue receiving
reminders after the study was concluded.
When computer-based reminders are sent to
physicians caring for patients with rising creatinine
levels during treatment with nephrotoxic or renally
excreted medications, the physicians adjust those
medications more rapidly. This is true despite the
close attention that is usually paid to inpatients in a
The effect on physician behavior in this study
was most pronounced when patients were receiving
renally excreted medications or mild nephrotoxic
agents such as NSAIDs. This suggests that
physicians already pay close attention to the effects
of major nephrotoxins (such as the
aminoglycosides) on renal function. Although the
NSAIDs and renally excreted medications might be
viewed as less important, the strong effect of
reminders on physician behavior, as well as the
physicians' response that the medications mentioned
in the reminders are clinically important, suggests
that physicians still believe that the drugs are of
sufficient relevance to warrant discontinuation or a
change in dosage in the presence of deteriorating
renal function. Recent studies have shown that
NSAIDs appear to pose an important risk for
producing renal impairment in selected populations
The time it takes to read reminders must be
balanced against their efficacy if the information
they contain is already known or clinically
unimportant. Only 31 % of the recipients in this
study found the reminders annoying, and more than
70% expressed a desire to continue to receive
The major criticism of the reminders, expressed
in the comments section of the questionnaire, was
that they were sent to too many physicians not
directly involved in the patient's care. On average,
more than nine physicians received each reminder.
We considered this necessary because in our
hospital, where multiple physicians are responsible
for the care of a patient, there is no way to be
certain of who is caring for a patient at any time.
We believe, however, that we can reduce the
number of reminder recipients. Radiologists who
check patient laboratories before performing
intravenous pyelograms, and pathologists who
check them as part of their work in the clinical
laboratories, could be removed from the group of
This study was performed at a single hospital.
is unclear whether the results can be generalized to
other settings. It seems likely, however, that in
hospitals that do not have house staff to monitor
patients closely, reminders might have an even
greater effect on physician behavior. The ease of
delivery of the reminders, and probably their high
rates of acceptance, was made possible by the
presence of an integrated hospital computing system
m which electronic mail, laboratory data, and
medication data are all available within a single
Although the strong effect on behavior implies
that physicians find these reminders useful, this
study does not look at the effect of reminders on
patient outcomes. We intend to collect further data
in an attempt to evaluate whether computer
reminders improve patient outcomes such as length
of stay and peak creatinine level.
Computer-based reminders to physicians about
rising creatinine levels in hospitalized patients
receiving nephrotoxic and renally excreted
medications reduced the time needed to adjust these
medications by more than 21 hours. The reminders
were well accepted by physicians, and most wished
to continue receiving them. The reminders could
be improved by targeting the physicians who are
most directly responsible for the care of the patient
McDonald CJ, Hui SL, Smith DM, Tiemey
WM, Cohen SJ, Weinberger M, McCabe GP.
Reminders to physicians from an introspective
computer medical record. A two-year
randomized trial. Ann Intern Med 1984;
Fordham D, McPhee SJ, Bird JA, Rodnick
JE, Detmer WM. The Cancer Prevention
Reminder System. MD Comput 1990;
Ornstein SM, Garr DR, Jenkins RG, Rust PF,
Arnon A. Computer-generated physician and
patient reminders. Tools to improve population
adherence to selected preventive services. J
Fam Pract 1991; 32:82-90.
Frame PS. Can computerized reminder
systems have an impact on preventive
services in practice? J Gen Intern Med 1990;
Tierney WM, McDonald CJ, Martin DK, Hui
SL, Rogers MP. Computerized display of
past test results.
Ann Intern Med 1987; 107:569-74.
McDonald CJ. Protocol-based computer
reminders, the quality of care and the non-
perfectibility of man. N Engl J Med 1976;
McDonald CJ, Wilson GA, McCabe GP.
Physician response to computer reminders.
JAMA 1980; 244:1579-81.
Tate KE, Gardner RE, Weaver LK. A
computerized laboratory alerting system. MD
Comput 1990; 7:296-301.
Safran C, Bleich HL, Slack WV. Role of
computing in patient care in two hospitals.
MD Comput 1989; 6(3):141-8.
Bleich HL, Beckley RF, Horowitz GL, et al.
Clinical computing in a teaching hospital. N
Engl J Med 1985; 312:756-64.
Effect on outpatient testing.
Bleich HL, Safran C, Slack WV.
Departmental and laboratory computing in two
hospitals. MD Comput 1989; 6(3):149-55.
12. Bloom SM, White RJ, Beckley RF, Slack
WV. Converse: A means to write, edit,
administer, and summarize computer-based
dialogue. Comput Biomed Res 1978; 11:167-
13. Whelton A, Stout RL, Spilman PS, Klassen
DK. Renal effects of ibuprofen, piroxicam,
and sulindac in patients with asymptomatic
renal failure. Ann Intern Med 1990; 112:568-
14. Murray MD, Brater DC, Tierney WM, Hui
SL, McDonald CJ. Ibuprofen-associated renal
impairment in a large general internal
medicine practice. Am J Med Sci 1990;
Table I: Comparison of the control and intervention periods
Patient Age, years (mean)
Patient Sex (% Male)
Nephrotoxin Events (%)
Baseline Creatinine, mg/dl (mean)
Event Creatinine, mg/dl (mean)
DRG CostWeight (mean)
TableII:Mean time tochangemedication(TTC),in hours(s.d.), bymedication classification
I Number of events
difference in TTC significant at p=0.0001
difference in TTCsignificantatp<0.0001
J 93.7 (145.6)
Table III: Mean time tochange medication (TTC), in hours (s.d.), by class of nephrotoxin
Number of events
difference in TTC significant at p<0.05
Table IV: Responses to the questionnaire
Have the computer reminders been annoying?
Have the computer reminders been helpful in the care of
Have the reminders given you clinically useful
information before you otherwise would have known it?
Have the reminders referred to changes in creatinine that
are too small to beclinically imprtant?
Are the medications mentioned in the reminders ones that
you would want to think about in a patient whose renal
function iS worsening?
Would you like to continue receiving reminders after the
study is over?
We would like to thank Robert Brown, M.D. for
help in choosing the nephrotoxic and renally
excreted medications used to trigger the reminders,
and Charles Rury and Elaine Bianco for assistance
with the development of the computer programs.
This study was supported, in part, by a grant from
the Hartford Foundation.