Application of Information Technologyj
Translating Research into Practice: Organizational Issues in
Implementing Automated Decision Support for Hypertension in
Three Medical Centers
MARY K. GOLDSTEIN, MD, MS, ROBERT W. COLEMAN, MS, SAMSON W. TU, MS, RAVI D. SHANKAR, MS,
MARTIN J. O’CONNOR, MSC, MARK A. MUSEN, MD, PHD, SUSANA B. MARTINS, MD, MSC,
PHILIP W. LAVORI, PHD, MICHAEL G. SHLIPAK, MD, MPH, EUGENE ODDONE, MD, MHSC,
ANEEL A. ADVANI, MD, PARISA GHOLAMI, MPH, BRIAN B. HOFFMAN, MD
A b s t r a c t
settings. Technology can address the quality gap in health care by providing automated decision support to clinicians
that integrates guideline knowledge with electronic patient data to present real-time, patient-specific recommendations.
However, technical success in implementing decision support systems may not translate directly into system use by
clinicians. Successful technology integration into clinical work settings requires explicit attention to the organizational
context. We describe the application of a ‘‘sociotechnical’’ approach to integration of ATHENA DSS, a decision support
system for the treatment of hypertension, into geographically dispersed primary care clinics. We applied an
iterative technical design in response to organizational input and obtained ongoing endorsements of the project by the
organization’s administrative and clinical leadership. Conscious attention to organizational context at the
time of development, deployment, and maintenance of the system was associated with extensive clinician use
of the system.
Information technology can support the implementation of clinical research findings in practice
j J Am Med Inform Assoc. 2004;11:368–376. DOI 10.1197/jamia.M1534.
Information technology holds great promise as a means to
support clinical practice guidelines; however, many promis-
ing information systems have encountered substantial diffi-
culties in implementation.1–4In some cases, clinicians have
low rates of using the system, for example, not interacting
directly with a guideline by using the computer5or not
accessing a clinical decision support system.6We devel-
opeda decision support
ATHENA DSS, that automates evidence-based guidelines
for management of primary hypertension.7We recognized
that development was only the first step in implementing
a decision support system in a health care system: suc-
cessful implementation requires full integration into the
clinical setting.8–14We evaluated what was known about
barriers and facilitators for guideline implementation, in-
cluding experience with successful and unsuccessful imple-
mentations of information technologies in clinical settings,
and designed an approach that benefited from these in-
system for hypertension,
Our implementation has gone well. The system has run for
more than 15 months in clinic sites in nine different cities that
are part of three administratively separate medical centers.
The system has displayed detailed individualized advisories
for more than 10,000 patients. Clinicians are interacting with
the advisories at substantial rates. We describe the steps that
we took in implementing this system. A key feature of our ap-
proach is its interdisciplinary nature: We address technical-
informatics aspects and social-organizational aspects in an
integrated manner.11–15Rather than describe the technical as-
pects of the application in isolation from its clinical and orga-
nizational context, we focus on the interrelationships of the
technical design with the clinical and organizational context
Affiliations of the authors: Geriatrics Research Education and
Clinical Center, VA Palo Alto Health Care System, Palo Alto, CA
(MKG, RWC, SBM, PG): Center for Primary Care and Outcomes
Research, Stanford University School of Medicine, Stanford, CA
(MKG); Stanford Medical Informatics, Stanford University School of
Medicine, Stanford, CA (MKG, SWT, RDS, MJO, MAM, AAA):
Cooperative Studies Coordinating Center, VA Palo Alto Health Care
System, Palo Alto, CA (PWL); San Francisco VA Medical Center, San
Francisco, CA (MGS); Department of Medicine, University of
California San Francisco, San Francisco, CA (MGS); Durham VA
Medical Center, Durham, NC (EO); Department of Medicine, Duke
University Medical Center, Durham, NC (EO); VA Boston Health
Care System – West Roxbury Division, Boston, MA (BBH); Harvard
Medical School, Boston, MA (BBH).
Supported in part by VA HSR&D CPI-99-275 and RCD-96-301 and
NIH LM05708 and LM06245.
The authors thank the Director and staff, in particular IRMS
and primary care leadership, at VA Palo Alto Health Care System,
Durham VA Medical Center, and San Francisco VA Medical Center.
The authors also thank the clinicians who have participated in
monitoring and using ATHENA DSS.
The views expressed in this article are those of the authors and do
not necessarily represent the views of the Department of Veterans
Correspondence and reprints: Mary K. Goldstein, MD, MS, GRECC
182 B, VA Palo Alto Health Care System, 3801 Miranda Avenue, Palo
Alto, CA 94304; e-mail: <firstname.lastname@example.org>.
Received for publication: 01/13/04; accepted for publication:
GOLDSTEIN ET AL., Translating Research into Practice
‘‘In the current health care system, scientific knowledge about
best care is not applied systematically orexpeditiously to clin-
ical practice. An average of about 17 years is required for new
knowledge generated by randomized controlled trials to be
incorporated into practice.’’16,17The gap between scientific
evidence and clinical practice can be bridged only by influ-
encing clinician behavior to translate research findings into
routine clinical practice.18,19Several major programinitiatives
address this need, including the Department of Veterans
Affairs Quality Enhancement Research Initiative (QUERI).
QUERI includes methodologies for translation research, with
an emphasis on organizational structures and process consid-
erations.20Translation projects ideally pay explicit attention
to identification of effective strategies for organizational im-
plementation, so that translation can be systematized for
installation at other medical centers. As described in the over-
view paper for this series, the QUERI program defines six
standard steps of the QUERI process; we describe here our
work on step 4, which is ‘‘Identify and Implement
Interventions to Promote Best Practices.’’21
In Crossing the Quality Chasm, the Institute of Medicine16calls
for a ‘‘New Health System for the 21st Century’’ that recogni-
zes the predominance of chronic disease, creates an infra-
structure to support evidence-based care, and facilitates the
use of information technology to translate research findings
into practice. The Institute of Medicine has identified ‘‘im-
provability gaps’’ in health care, i.e., areas in which clinical
practice falls substantially short of evidence-based best prac-
tices. Information technology has substantial potential to sup-
port translation projects that address improvability gaps.
However, attempts to integrate new informatics technology
into clinical work settings are not always successful,22as illus-
trated dramatically by the termination of the electronic med-
ical record at Cedars-Sinai Medical Center in Los Angeles.23
Integration of new technology into an organization is a ‘‘polit-
ically textured process of organizational change’’ that must
accord primacy to the needs of the users and the organiza-
tion.2Technology use depends on the ‘‘meticulous interrela-
tion of the system’s functioning’’ with the work of the
health care professionals.2The implementation must be sup-
ported by administrative leadership and by future users.1A
sociotechnical approach includes fundamental incorporation
of organizational factors that include both iterative technical
design in response to organizational input and explicit atten-
tion to the political context of technology implementation.2
We describe the integration of a guideline-based decision sup-
port system into geographically dispersed primary care clin-
ics in a largehealth care system. We describe both the technical
features of the implementation developed by responding iter-
atively to organizational input and the interrelated process of
attending to the organizational context by obtaining and
maintaining endorsement of the project by the organization’s
administrative and clinical leadership.
The Information System Deployed
We developed a system for implementing clinical practice
guidelines designed to translate hypertension research find-
ings into practice in primary care clinics. The approach pro-
vides automated decision
clinicians. The decision support system ATHENA DSS was
designed as a platform-independent system for integration
with an existing electronic medical record (EMR) system.
ATHENA DSS uses a guideline interpreter to combine patient
information from the EMR with knowledge of hypertension
to generate patient-specific recommendations, explanations,
and evidence-based education, which are then delivered to
clinicians in a pop-up window at the time of outpatient pri-
mary care clinic visits.7,24ATHENA DSS was constructed us-
ing the EON architecture developed at Stanford Medical
Informatics for guideline-based decision support systems.25
Figure 1 shows the basic system architecture. It includes
a knowledge base (KB) that models hypertension knowledge,
a guideline interpreter, a temporal database mediator, and
a custom graphical user interface. ATHENA DSS, developed
in Prote ´ge ´, separates the KB from the interpreter rules and the
patient database, so that clinician-managers can easily
browse and update the KB.
Our aim was to integrate ATHENA DSS into the primary care
clinics at three VA medical centers—VA Palo Alto Health
Care System (VAPAHCS), San Francisco VA Medical Center,
and Durham VA Medical Center—to implement national
guidelines for the treatment of hypertension. We are evaluat-
ing the impact of this guideline implementation on patient
care in a randomized, controlled trial (RCT) with patients?
blood pressure control and guideline–drug concordance as
the primary outcome measures (results not yet available).
We implemented the system first at VAPAHCS for a limited
number of clinicians who would not be enrolled in the study.
After wehad gained experience with the implementation pro-
cess, we installed the system at San Francisco and Durham,
starting with the physician-investigators and later moving
ATHENA DSS is a platform-independent system designed
VAPAHCS uses the national Department of Veterans
Affairs? medical record system27VistA (Veterans Health
Information Systems and Technology Architecture) (largely
based on M, formerly known as the Massachusetts General
Hospital Utility Multiprogramming System or MUMPS),
F i g u r e 1. Model of ATHENA DSS architecture.
Journal of the American Medical Informatics AssociationVolume 11Number 5 Sep / Oct 2004
and its user interface, Computerized Patient Record System–
Graphical User Interface (CPRS-GUI).
deployment at VAPAHCS was that the system be consistent
with the VAPAHCS’s goals for clinical practice guideline
implementation and that it enhance clinician acceptance of
guideline-based recommendations. Our design requirements
included achievement of VAPAHCS’s administrative goals,
acceptability to clinician-users with simple user interface,
and consistency with the RCT protocol. As a result, our inte-
gration objectives for ATHENA DSS were as follows:
overallorganizational requirementfor the initial
Integrate Smoothly into Individual Clinician’s Workflow
d For recommendations to appear at the time of clinical
decision making, without requiring further action by the
clinician, advisories must appear to the clinician in the
CPRS-GUI coversheet as a pop-up windowwhen an appro-
priate patient record is accessed.
dGeneration of the recommendations using the patient data
must not require that the clinician retype data known to the
dEntry of updated information at the option of the clinician
must return a nearly instantaneous update of recommenda-
dThe system must not slow down workstation performance
in the clinics.
d Users must be able to bypass the pop-up window easily if
they do not want to use it.
dThe system must run on the existing operating system.
Meet Additional Institutional Goals
dFor patient privacy protection, the entire system must oper-
ate on the VA Intranet behind the VA firewall and present
advisories only to users who have logged on to the CPRS.
dThe advisories must appear only on the computer screens
in the primary care clinics and not in areas of the hospital
in which hypertension management is typically different
from that in primary care clinics, for example, in the inten-
sive care units and acute medical units.
d The advisories must appear only for primary care clinicians
enrolled by the developers and not for other clinicians
viewing the patients? records.
d The system must be available at all primary care clinics of
VAPAHCS, including those more than 100 miles from the
d The recommendations must be available on the day of
nosis of hypertension and on the working day preceding
each visit so that clinicians who do visit planning can see
Maintain the System Accuracy over Time
d The system must send alerts to the developers for drug
names that do not match a drug already recognized by
the system (for example, when a new drug is added to
d Updating the knowledge base must be possible without re-
installation on the clinic computers.
dThe system must monitor each clinic computer for activity.
d The system must allow clinician-users to provide free-text
feedback that can be monitored for early identification of
Allow for a Clinical Trial Randomized by Clinicians
dThe system must permit display of two different versions of
a pop-up window to designated clinicians to allow for
a control group and an intervention group in the RCT.
dThe system must allow clinician-users to provide feedback
on their deviations from the system’s recommendations.
d The system must capture data for evaluation, including its
recommendations, and clinician usage data.
To accomplish these goals, we involved representatives of the
organization in the technical design in an iterative process,
selecting technical features to enhance acceptability of the
system to clinicians and administrators: a sociotechnical
We included three VA medical centers: VAPAHCS, San
Francisco VA Medical Center, and Durham VA Medical
Center. VAPAHCS, the initial site, is a large integrated health
care system in mid-coastal and central California, spanning
more than 13,000 square miles, with a tertiary care hospital
at the main campus and a network of sites with subacute
units, long-term care, and outpatient clinics. We included
clinic sites of VAPAHCS located at Palo Alto (PAD), Menlo
Park (MPD), San Jose (SJC), Monterey (MON), Capitola
(CAP), Livermore (LD), Stockton (STC), and Modesto
(MOD). Driving time from the main campus at Palo Alto to
the outer sites is greater than two hours even in optimal traffic
conditions. Primary care clinicians with drug prescribing
privileges include approximately 55 attending physicians,
40 resident physicians, and seven nurse practitioners and
physician assistants. Clinic sites include many shared-
use areas in which primary care clinicians and specialists
use the same computers. In the primary care areas of
VAPAHCS, there were 146 computers from various manufac-
turers, running either Microsoft Windows NT or Windows
2000. All are networked to the central VistA computers lo-
cated in Palo Alto.
San Francisco and Durham VA Medical Centers’ primary care
clinics included in the study are both located in a single build-
ing, with a smaller numberofcomputers (approximately 25 at
each site). Clinicians at all sites possess a wide range of com-
puter experience, including some who are adept and others
who are beginners.
Technical Features to Support Organizational
We describe features of the technical deployment designed to
support the organization’s requirements for system opera-
tion, focusing on the aspects of the technical design that ad-
dressed clinical and organizational aims.
Precomputed Advisories from Patient Data
We developed a method for transferring existing patient data
to ATHENA DSS. An M program extracts patient data from
VistA each night, based on the following selection criteria: pa-
tient has a diagnosis of hypertension; patient has a scheduled
appointment in a primary care clinic (general medical clinic,
geriatrics clinic, or women’s health clinic) within a five-day
window (encompassing Friday as the prior workday for
Monday clinics). For each patient identified, the following
GOLDSTEIN ET AL., Translating Research into Practice
data are extracted: diagnoses; result(s) of 14 selected labora-
tory tests together with the date of the test; all prescribed
drugs in the pharmacy database with date, dose, and number
of pills dispensed; and all blood pressure, pulse, height, and
weight measurements with date. The extracted patient data
file is sent by ftp to the ATHENA server. We precompute ad-
visories on all patients whose data are extracted each night.
Minimization of Code Installation on Client Computers
Our system minimizes the code installed on each client com-
puter in the clinics. We use a client-server architecture where
the guideline interpreter and temporal database mediator run
as server processes on the ATHENA server. We install the
ATHENA Client software that manages the pop-up window
centrally on the ATHENA server. We install on client ma-
chines only stable commercial software that requires no up-
date during the course of the clinical trial and a small client
program that monitors messages from the CPRS. When the
ATHENA knowledge base is updated, for example, to reflect
refinements in the wording of messages, the new knowledge
base is installed only on the server.
Display of Advisories to Clinician Users
When the CPRS notifies ATHENA that a clinician has opened
a patient record, ATHENA Client requests an advisory for
that patient from the guideline interpreter. When ATHENA
receives the advisory, it generates a pop-up window within
the CPRS-GUI window containing recommendations based
on the available patient data. The pop-up window includes
tabs and buttons that allow clinicians to access additional
screens and enter new data. Clinicians can easily bypass the
window by clicking outside it or by closing it. A sample
advisory is shown in Figure 2. When the clinician enters
additional patient data, for example, blood pressure measure-
ments ordiagnostic information not previously available, and
requests an updated advisory, the information is transmitted
to the server and the pop-up window is refreshed with the
updated advisory. Clinicians may enter feedback about the
program either by clicking on a checklist for each drug re-
commendation or by entering free-text in a comment box.
The BP-Prescription Graphs tab (Figure 3) displays all the
blood pressures and antihypertensive agents for this pa-
tient on the same time line, with a scroll bar to view older
data. Graphical displays of clinical information about hyper-
tension can summarize and clarify complex interrelation-
Data logging aids user acceptability, by capturing feedback
that can be used to refine the system, and supports the goals
of the clinical trial. The patient data and the precomputed ad-
visories for all patients are logged, providing a record of the
recommendations that would have been generated for both
control and intervention groups, regardless of whether they
F i g u r e 2. Sample ATHENA advisory recommendations.
Journal of the American Medical Informatics AssociationVolume 11Number 5 Sep / Oct 2004
were displayed to the clinician. The system captures patient
data entered by the clinician-user, the updated recommenda-
tions displayed, and clicks indicating that the user viewed ad-
Addressing Organizational Challenges
Translation projects typically unite people from disparate cul-
tures and activities into what may be, at least initially, an un-
wieldy team. Teams need time to develop skills.29Good
teamwork requires that all the members are trying to succeed
at the same game and have a common understanding of the
rules and the language for communication. Academic infor-
maticians, health services researchers, physician domain ex-
perts, and hospital information system specialists typically
have disparate disciplinary perspectives
Consequently, uniting them on the same team with common
overall objectives requires effort and commitment to develop
a shared vocabulary and patterns for handing-off tasks. We
approached this challenge by starting with small tasks and
using bridge personnel, i.e., individuals who had familiarity
with at least two of the four disciplines. As our team gained
experience in working together, we developed patterns of
communication and identified the most effective role assign-
ments. One of the key roles that emerged was the database/
pharmacist who ensured effective communication between
the Health Care System’s VistA experts and the medical infor-
Diagnosing Barriers to Implementation
Effective implementation requires a realistic initial assess-
ment of local facilitators for and barriers to implementation
and periodic reassessment as the project develops. We de-
scribe these barriers and facilitators in the following sections.
The goal of the project, to improve care of patients with hy-
pertension by implementing the VA guidelines, aligns with
general institutional goals. The initial grant proposal for the
project was discussed with the Chief of the Information
Resource Management Service (IRMS) and the Chief of
Staff, who both wrote letters of support for the project. The
proposal included an implementation phase followed by
a clinical trial phase. During the implementation phase, we
took steps to ensure that the project remained well integrated
with the administrative goals of VAPAHCS. We identified
whose approval and authorization were essential for success-
ful implementation. These included the physician with
(all general internists)
F i g u r e 3. Sample ATHENA BP prescriptions graph: blood pressures and antihypertensive prescriptions on same time line.
Target blood pressure is shown as a straight line.
GOLDSTEIN ET AL., Translating Research into Practice
oversight of VAPAHCS outpatient clinics, of which the pri-
mary care clinics are a subset, and the physician with over-
sight of the large block of satellite sites in the east San
Francisco Bay area, including the LD, STC, and MOD sites.
Several months before the anticipated launch of the clinical
trial, we gave physician-administrators access to ATHENA
DSS and offered them the opportunity to assess (1) the time
involved in viewing the pop-up window and the impact on
clinical work and (2) the clinical content of the knowledge
base to ensure its consistency with VA guidelines. We were
able to provide the system to selected individuals without ac-
tivating it for clinicians who would later be enrolled in the
clinical trial. We encouraged the physician-administrators to
provide feedback to us, which we then addressed in redesign.
For example, one physician-administrator requested that we
display the pop-up window the day before the scheduled
clinic visit as well as on the day of the visit because he encour-
aged outpatient clinic physicians to review their charts the
day before the visits to allow the clinic schedule to run on
time (visit planning). We revised the system accordingly.
Early in the planning stage, we met with IRMS administrators
and networking staff to outline our plans and obtain their in-
put and approval, and we maintained close contact with the
IRMS staff during planning and implementation. The IRMS
staff provided support in several key areas including install-
ing the project server in the IRMS server room so that it could
benefit from backup power supply, air conditioning, and op-
timal network connections; programming the M patient data
extract; and network support.
We presented the project to the Health Care System’s Medical
Informatics Committee to gain feedback from key personnel.
We also discussed the project with the CPRS Implementation
Coordinator at several phases and redesigned in response to
comments. For example, we added a second patient identifier
to the pop-up window when the coordinator requested the
presence of two identifiers to enhance patient safety.
Physician Acceptance of Clinical Content
Successful guideline implementation requires local clinical
opinion leader ‘‘buy in’’ of the clinical content. Clinicians
must be assured that the guideline recommendations are well
founded. The recommendations presented should be based
on sufficient backing.30In our case, national guidelines pro-
vide overall backing for the recommendations. For the recom-
mendations based on compelling indications, we display the
evidence base supporting the recommendations.31However,
automation of guidelines requires translation of guideline
knowledge into computable formats, which involves supple-
mental knowledge and decisions about how to resolve ambi-
guities in the guidelines.32Additional backing is needed for
supplemental knowledge and for choices about resolution
of ambiguities. One of our experts directed the specialized
Hypertension Clinic in the Health Care System and served
as the primary domain expert for hypertension, enhancing
clinician-users? confidence in the recommendations made by
ATHENA DSS. We consulted with medical center experts in
nephrology, cardiology, and rheumatology for some special-
Clinical opinion leaders must also be confident that theguide-
lines apply well to their own patient population. We recruited
several physicians to assist with review of our guideline im-
plementation in ATHENA DSS. In addition to the physi-
cian-administrators described above (one of whom was also
the medical center’s overall guideline implementation
leader), we recruited the supervisor of the general medical
clinics at the Palo Alto site and the primary care chief resident
as physician-monitors. We shared the knowledge rules used
in ATHENA DSS, gave them individual training sessions in
use of the system, activated the system at their clinics, and en-
couraged them to comment directly and to use the feedback
features built into ATHENA DSS as described above.
Implementation Challenges at Other Medical
Implementing the system at additional medical centers, un-
der different administrations, presented a new set of issues.
It is more difficult to achieve and sustain enthusiasm for the
new technology at sites other than where the system was de-
veloped and tested. With funding from the VA Health Service
Research and Development service to study the impact of hy-
pertension guideline implementation using ATHENA, we
were able to implement the ATHENA system at two addi-
tional VA medical centers (San Francisco and Durham). The
San Francisco VA Medical Center is in the same VA regional
group, the Sierra-Pacific Veterans Integrated Service Network
(VISN) 21, as VA Palo Alto Health Care System but has its
own medical center director and administrative structure
completely separate from VAPAHCS. The Durham VA
Medical Center has a separate administration and is located
thousands of miles from Palo Alto. At these medical centers,
we faced and addressed unique implementation issues that
did not occur at Palo Alto.
Site Lead Physician without Informatics Skills
Both San Francisco and Durham had on-site physician-inves-
tigators who were primarily responsible for overseeing im-
plementation, but neither of these physicians had special
training or experience in informatics. We designed the system
to run with as little maintenance as possible required from the
local site. The study funds provided for a half-time research
assistant at each site, who was the only funded support for
all aspects of the project at that site. The additional sites re-
cruited research assistants who had enough computer skills
to work with the staff at Palo Alto to determine that the sys-
tem was up and running and to do initial steps in trouble-
Interfacing with Local Hospital Information System Staff
Local IRMS personnel at the additional sites had less invested
in seeing that ATHENAworked properly than the IRMS per-
sonnel at Palo Alto. A small but crucial amount of support
was required from the IRMS to install the extract program
and to mount the software on the client computers in the clin-
ics. In San Francisco, the VISN chief for IRMS facilitated this
process. It was important for the developers at Palo Alto to
recognize the need for IRMS staff at each site to review the
software to assess its impact on network traffic and security.
Our stance was to encourage and cooperate with every re-
view of the system that the local IRMS staff deemed impor-
tant. In addition, a Palo Alto staff member with extensive
IRMS experience (RC) visited Durham for two days to work
directly with the IRMS staff on installation.
Journal of the American Medical Informatics AssociationVolume 11 Number 5 Sep / Oct 2004
At Durham, ATHENA DSS was run from a local server set up
by the project. Because the ATHENA server was part of a re-
search project, it was outside the main computing infrastruc-
ture of the IRMS at Durham and required an on-site person to
maintain and update changes in that server. This person was
the information officer for the research group participating in
the study that brought ATHENA to the site. For wider imple-
mentation, ownership for this server, its maintenance, and
updating will need to be assumed by the hospital IRMS.
Local Administrative Approvals
Special approval from the facility clinical guidelines commit-
tee was required at the Durham VA Medical Center. The med-
ical center director had to know of the project, and both the
medical center and VISN directors had to approve the imple-
mentation, particularly since other initiatives were in devel-
opment that would have had overlapping goals (e.g.,
a clinical reminder in the CPRS directed at patients whose
blood pressure was higher than the recommended target
Training Clinicians in Use of ATHENA DSS
Since the VAPAHCS study site included clinics located in
seven different cities across a wide geographic area, it was
not feasible to bring the clinicians together for training.
Training at VAPAHCS was accomplished in a short (typically
ten-minute) telephone call with a member of the project staff.
For the Durham and San Francisco sites, a training slide show
was created in Palo Alto and provided to the lead clinician in-
vestigator at each site for local customization. The lead phy-
sician-investigator at each of those two sites then conducted
a group training session for the clinicians, with individual fol-
low-up sessions by either the physician-investigator or the re-
search assistant with clinicians who missed the group session.
As a result, training at the Durham and San Francisco sites
was conducted by people who were not directly familiar with
the development and nuances of ATHENA. As with all clini-
cian training, it was difficult to achieve 100% compliance,
even with both group and individual training sessions.
Sustaining Clinician Interest
Clinicians receive many clinical reminders. One effective
means to sustain clinician interest was to provide quarterly
feedback on guideline–drug concordance for hypertension.
Clinicians are aware of the evidence supporting treatment
of hypertension to lower cardiovascular risk; presenting them
with their medical center’s and their individual rates of ade-
quacy of control of blood pressure appeared to sustain inter-
est. This often stimulated more questions and renewed
interest in achieving stated goals or questions about how pa-
tients or outcomes were chosen. Providing a forum for these
questions was important. At the Durham VA Medical Center,
this occurred in the monthly clinical staff meetings.
Site Lead Physician Confidence in ATHENA DSS
The physician-investigators in San Francisco and Durham
were not the original developers of ATHENA DSS and did
not initially have a sense of ‘‘ownership’’ of the knowledge
base. Lack of familiarity with the program could potentially
interfere with their enthusiastic endorsement of it to their col-
leagues. We addressed this potential barrier by providing op-
portunities for the physician-investigators to ‘‘test drive’’ the
system with theirown patients inrealtime in theirownclinics
and by reviewing panels of patients in their office. We also
discussed knowledge-base issues frequently in telephone
conference calls, developing a group consensus about how
to handle medical questions. Over time, this process built
confidence that the knowledge in the system is state-of-the-
art and correct.
Clinician Usage of ATHENA DSS
In the first 15 months of the clinical trial, 91 primary care clini-
cians from the three sites were assigned to the ATHENA
study arm. The ATHENA system displayed ATHENA advi-
sories for 10,806 distinct patients. Clinicians entered a new
blood pressure and updated the advisory for 34% of patients.
Clinicians interacted with the advisory screen in this or other
ways for 63% of patients. Use of the system remained high
throughout the 15 months. These rates of clinician use con-
trast sharply with reported rates ofotherclinical decisionsup-
ports systems, which are accessed only a very small
percentage of the time that they are available.5Clinicians en-
tered free-text messages into the advisory’s comment field for
747 patients; approximately half included an explanation of
why a particular recommendation was not being followed.
This extensive use of the system by clinicians in practice
speaks to the usability and usefulness of the system.
ATHENA DSS is being installed at additional VA medical
centers. Plans are under way to update the knowledge base
in light of recent changes to hypertension guidelines and to
extend the knowledge base to include diabetes and hyperlip-
idemia, two other important cardiovascular risk factors.
Although ATHENA DSS recommendations use patient data
from VistA and appear in a pop-up window within the
CPRS-GUI cover sheet for the patient selected, the system
does not currently write patient data to VistA. In future work,
we hope to further the integration with VistA so that, for ex-
ample, blood pressure measurements entered into the
ATHENA advisory window to obtain an updated recommen-
dation may also be written to the patient’s medical record,
and drug recommendations made by ATHENA can generate
a physician order-entry screen. We are surveying the clini-
cians for comments on the system. We hope to obtain input
from clinician-users to inform redesign of the user interface.
Information technology can help translate research findings
into practice, but implementations present organizational
challenges. Integration ofa platform-independent system into
a legacy electronic medical record is difficult. In this case, the
EON architecture software developers at Stanford Medical
Informatics were not themselves employees of the institution
(VA) implementing the system. Our approach of building
a collaborative team bridging the necessary institutions and
disciplines, soliciting and addressing the organizations? inter-
ests in the technical design, and maintaining close contact
with the local administration, hospital information systems
group, and clinical opinion leaders has led to a successful im-
plementation in a complex environment. We have computed
GOLDSTEIN ET AL., Translating Research into Practice
and displayed detailed patient-specific advisories to clini-
cians for thousands of patients.
In addition to the success of the deployment as measured by
the display of advisories, we also have good indications that
the advisories have captured the clinicians? interest. The clini-
cians are interacting with the displayed advisories at a very
substantial rate. These time-pressured clinicians, who are
generally fully booked in clinics with complex patients, en-
tered hundreds of free-text comments suggesting their inter-
est in the system. Achieving this degree of implementation
was possible only through the cooperation of a large number
of individuals. We believe that this was achieved in large part
because of the careful attention to sociotechnical integration.
One of the accomplishments of this project is system imple-
mentation at sites that do not have a medical informatics–ori-
ented lead physician. Information technology that can be
implemented only at sites with highly involved informatics-
oriented physicians would be quite limited in dissemination.
We aimed to develop a system that could be implemented in
a manner highly integrated with clinical care without an on-
site physician with strong informatics skills. Our successful
integration at two additional medical centers, both geograph-
ically and administratively separate from the main site,
speaks to the robustness of the technology.
Recent papers have reported a lack ofimpact ofcomputerized
decision support for chronic disease in primary care, in con-
trast to the prior success in preventive medicine and other
areas. Computer-based cardiac care suggestions had no effect
on physicians? adherence to recommended care.33Eccles
et al.6found that a system implementing guidelines for
asthma and angina had no significant effect on processes of
care. They noted that, for much of the study, the median num-
ber of clinician interactions with the system was zero. We do
not yet have our clinical trial results for impact on clinician
guideline adherence, but we have found substantial rates of
clinician interaction with the system. Eccles? group conducted
a parallel interview study to understand factors influencing
system adoption and found that respondents thought the sys-
tem did not fit well within their practice context.34Our higher
rates of clinician use of the system may reflect the attention
given to sociotechnical integration. We anticipate that the in-
formation that we have collected and reported on clinician
use of the system and on organizational factors will illumi-
nate the trial results whether positive or negative.
We used Berg’s inspired term sociotechnical to describe our
approach to integrating social-organizational issues with the
technical-informatics issues.2The work that we describe in
this paper was not a sociological study of an informatics im-
plementation; rather it was a description of both social and
technical aspects of an implementation done deliberately
using approaches that take account of sociotechnical issues
to maximize the likelihood of success.
Berg15has also discussed the ways in which introduction of
new information technology to the medical workplace can
change the way that work is done. Future work could explore
ways in which this may happen at the sites using the
ATHENA DSS or other decision support systems for guide-
line implementation. Researchers could also investigate
changes to the system design and the user interface that could
improve the system further. For example, an advisory is dis-
played for every hypertensive patient who meets the eligibil-
ity criteria for the program, and the advisory includes a large
amount of information on the top level. Ash et al.22noted in
a recent paper on unintended consequences of information
technology in health care that ‘‘decision support systems
could trigger an overdose of reminders, alerts, or warning
messages.’’ In future work, focus groups with clinician-users
could help guide design changes. For example, more selective
triggers for the appearance of the advisory and/or a smaller
pop-up window that alerts the user to the presence of an ad-
visory that is displayed only if the clinician clicks to request it
could be incorporated to lower the risk of decision support
Our approach to implementing new information technology
addresses both social-organizational issues and informatics-
technical issues in an interrelated manner and can be applied
to cross-platform and cross-institution implementations in
1. Berg M. Implementing information systems in health care orga-
nizations: myths and challenges. Int J Med Inf. 2001;64(2–3):
2. Berg M. Patient care information systems and health care work:
a sociotechnical approach. Int J Med Inf. 1999;55(2):87–101.
3. Sicotte C, Denis JL, Lehoux P. The computer based patient re-
cord: a strategic issue in process innovation. J Med Syst. 1998;
4. Sicotte C, Denis JL, Lehoux P, Champagne F. The computer-
based patient record: challenges towards timeless and spaceless
medical practice. J Med Syst. 1998;4:237–56.
5. Maviglia SM, Zielstorff RD, Paterno M, Teich JM, Bates DW,
Kuperman GJ. Automating complex guidelines for chronic dis-
ease: lessons learned. J Am Med Inform Assoc. 2003;10:154–65.
6. Eccles M, McColl E, Steen N, et al. Effect of computerised evi-
dence based guidelines on management of asthma and angina
in adults in primary care: cluster randomised controlled trial.
7. Goldstein MK, Hoffman BB, Coleman RW, et al., Implementing
clinical practice guidelines while taking account of changing ev-
idence: ATHENA DSS, An easily modifiable decision-support
system for managing hypertension in primary care. Proc AMIA
8. Ash JS. Factors affecting the diffusion of the Computer-Based
Patient Record. Proc AMIA Annu Fall Symp. 1997:682–6.
9. Ash JS, Gorman PN, Lavelle M, Lyman J. Multiple perspectives
on physician order entry. Proc AMIA Symp. 2000:27–31.
10. Ash JS, Lyman J, Carpenter J, Fournier L. A diffusion of innova-
tions model of physician order entry. Proc AMIA Symp. 2001:
11. Kaplan B. Addressing organizational issues into the evaluation
of medical systems. J Am Med Inform Assoc. 1997;4:94–101.
12. Kaplan B. Evaluating informatics applications–clinical decision
support systems literature review. Int J Med Inf. 2001;64:15–37.
13. Lorenzi NM, Riley RT, Blyth AJ, Southon G, Dixon BJ. Antece-
dents of the people and organizational aspects of medical infor-
matics: review of the literature. J Am Med Inform Assoc. 1997;4:
14. Anderson JG, Aydin CE. Evaluating the impact of health care in-
formation systems. Int J Technol Assess Health Care. 1997;13:
15. Berg M. Rationalizing medical work: decision-support techni-
ques and medical practices. In: Bijker WE, Carlson WB,
Pinch T (eds). Inside Technology. Cambridge, MA: The MIT
Journal of the American Medical Informatics Association Volume 11 Number 5 Sep / Oct 2004
16. Institute of Medicine Committee on Quality of Health Care in
America. Crossing the Quality Chasm: A New Health System for
the 21st Century. Washington, DC: National Academy Press,
2001, p 364.
17. Balas EA, Boren SA. Managing clinical knowledge for health
care improvement. In: Bemmel J, McCray AT (eds). Yearbook of
Medical Informatics. Bethesda, MD: National Library of Medicine,
2000, pp 65–70.
18. Bero LA, Grilli R, Grimshaw JM, et al. Closing the gap between
research and practice: an overview of systematic reviews of in-
terventions to promote the implementation of research findings.
19. Eisenberg JM, Kamerow DB. The Agency for Healthcare Re-
search and Quality and the U.S. Preventive Services Task Force:
public support for translating evidence into prevention practice
and policy. Am J Prev Med. 2001;20(3 suppl):1–2.
20. Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality En-
hancement Research Initiative (QUERI): a collaboration between
research and clinical practice. Med Care. 2000;38(6 suppl 1):
21. McQueen L, Mittman BS, Demakis JG. Overview of the Veterans
Health Administration (VHA) Quality Enhancement Research
Initiative (QUERI). J Am Med Inform Assoc. 2004;11:339–43.
22. Ash JS, Berg M, Coiera E. Some unintended consequences of in-
formation technology in health care: the nature of patient care
information system-related errors. J Am Med Inform Assoc.
23. Chin T. Doctors pull plug on paperless system. Available at:
http://www.ama-assn.org/amednews. Accessed July 2, 2004.
24. Goldstein MK, Hoffman BB, Coleman RW, et al. Patient safety in
guideline-based decision support for hypertension management:
ATHENA DSS. J Am Med Inform Assoc. 2002;9(6 suppl):S11–6.
25. Musen M, Tu S, Das A, Shahar Y. EON: a component-based
approach to automation of protocol-directed therapy. J Am
Med Inform Assoc. 1996;3:367–88.
26. Advani A, Tu S, O’Connor M, Coleman R, Goldstein MK, Musen
M. Integrating a modern knowledge-based system architecture
with a legacy VA database: the ATHENA and EON projects at
Stanford. Proc AMIA Symp. 1999;653–7.
27. Kolodner RM, ed. Computerizing large integrated health net-
works: the VA success. In: Hannah KJ, Ball MJ (eds). Computers
in Health Care. New York, NY: Springer, 1997.
28. Goldstein MK, Hoffman BB. Graphical displays to improve
guideline-based therapy of hypertension. In: Izzo Jr JL,
Black HR (eds). Hypertension Primer. Baltimore, MD: Williams
& Wilkins, 2003.
29. Goldstein M. Physicians and teams in care of the elderly patient.
In: Ham R (ed). Geriatric Medicine Annual. Oradell, NJ: Medical
Economics Books, 1989.
30. Toulmin S. Uses of Argument. Cambridge, UK: Cambridge Uni-
versity Press, 1958.
31. Shankar RD, Martins SB, Tu SW, Goldstein MK, Musen MA.
Building an explanation function for a hypertension decision-
support system. Medinfo. 2001;10(1):538–42.
32. Tierney WM, Overhage JM, Takesue BY, et al. Computerizing
guidelines to improve care and patient outcomes: the example
of heart failure. J Am Med Inform Assoc. 1995;2:316–22.
33. Tierney WM, Overhage JM, Murray MD, et al. Effects of comput-
erized guidelines for managing heart disease in primary care.
J Gen Intern Med. 2003;18(12):967–76.
34. Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M. Prac-
tice based, longitudinal, qualitative interview study of compu-
terised evidence based guidelines in primary care. BMJ. 2003;
GOLDSTEIN ET AL., Translating Research into Practice