Coping with PACS downtime in digital radiology

Abstract

As radiology departments become increasingly reliant on picture archiving and communication systems, they become more vulnerable to computer downtime that can paralyze a smoothly running department. The experiences and strategies developed during various types of picture archiving and communication system (PACS) downtime in a large radiology department that has completely converted to soft copy interpretation in all modalities except mammography are presented. Because these failures can be minimized but not eliminated, careful planning is necessary to minimize their impact.

Full text

Coping With PACS Downtime in Digital Radiology
Mike McBiles and Anna K. Chacko
As radiology departments become increasingly reliant
on picture archiving and communication systems,
they become more vulnerable to computer downtime
that can paralyze a smoothly running department. The
experiences and strategies developed during various
types of picture archiving and communication system
(PACS)
downtime in a large radiology department
that
has completely converted to soft copy interpretation
in all modalities except mammography are presented.
Because these failures can be minimized but not
eliminated, careful planning is necessary to minimize
their impact.
Copyright 9 2000
by W.B.
Saunders Company
KEY WORDS: PACS, failure, hospital information sys-
tem, server.
B
Y THEIR NATURE, widely distributed pic-
ture archiving and communication systems
(PACS), computerized hospital information sys-
tems (HIS), and radiology information systems
(RIS) alter the interaction of clinicians, radiolo-
gists, technologists, and administrators. The heart
of this alteration is widespread image availability
and rapid access to preliminary and formal reports.
When PACS systems fail, the benefits of rapid
image and report accessibility, reliable archiving,
and quicker image interpretation 1 are erased. The
disadvantages of reliance on a soft copy informa-
tion system are highlighted. During PACS failures,
alternative methods of temporary archiving, image
production, image interpretation, and report dissemi-
nation must be implemented rapidly and efficiently.
In departments such as ours that have heavy
reliance on PACS, this transformation to a nondigi-
tal or partially digital environment represents a
marked and difficult shift in workflow and the usual
methods of doing business. The resulting algo-
rithms to deal with this downtime are a compro-
mise between competing factors of technologists
From the Department of Radiology, Brooke Army Medical
Center, Fort Sam Houston, TX.
The opinions and assertions contained herein are the views of
the authors and ate not to be construed as offtcial oras
reflecting the views of the Department of the Army.
Address reprint requests to Mike McBiles, MD, Department of
Radiology, Brooke Army Medical Center, Fort Sam Houston,
TX, 78234.
Copyright 9 2000 by W.B. Saunders Company
0897-1889/00/1303-0003510.00/0
doi: 10.1053/jdim. 2000. 8055
and radiologist time and expertise, demands of the
clinician, rapid report and information dissemina-
tion, cost, medical legal issues, and adequate
patient care.
PACS downtime is an uncommon but possibly
catastrophic event that tests the resiliency of the
radiology department, the foresight of department
administrators, and preparedness of radiology de-
partment personnel. As radiology departments move
to a complete digital environment, dependence on
this technology becomes more complete. The inevi-
table system crash, of even planned outages, can
bring a smoothly running department to its knees,
create chaos and trepidation among clinicians and
referring services, and demoralize and frustrate
radiology department personnel. This report out-
lines the experiences of a large radiology depart-
ment when presented with failures of its PACS. We
present our empirical solutions to the immediate
workflow problems generated by these failures.
These solutions may benefit other institutions when
they are presented with similar problems.
BACKGROUND
Brooke Army Medical Center has completely
converted to soft copy reading in all areas except
mammography. This system uses high-resolution
monitors within the radiology department and an
extensive network of PACS and HIS terminals for
clinician use on wards, emergency room, operating
rooms, intensive care, and clinics. This configura-
tion is not limited to only filmless image generation
and interpretation but approaches the model of
"real-time radiology" as proposed by Thrall. 2 He
described an integrated computerized system of
study request, digital imaging, report generation,
and widespread and instantaneous image and report
availability.
The HIS and PACS systems are physically
independent in both their computers and their
associated networks. The architecture for the PACS
system is a spoke and hub con¡ (Fig 1)
along optical fiber connections to all viewing
stations. The core of the PACS system consists of 2
PACS servers, and 3 optical disc jukeboxes (1
terabyte each) with their respective controllers. The
PACS server performs the database functions,
short-term images image storage and retrieval, RIS
136
JournalofDigitallmaging, Vo113,
No 3 (August), 2000: pp 136-142

COPING WlTH PACS DOWNTIME 137
CR Plate CR Acquisition
Reader Gateway
Film Film Digitizer
Digitizer Workstationt
Gateway
MiniPACS ]
UItrasound Units Server/ |
Gateway
HIS Terminal
I
HIS Server
CT or MRI 1
Gateway
1
MiniPACS Gamma Cameras
Server/Gateway
Optical dise
jukebox
PACS
~~ers
HIS
Gateway
A
8
I
IL:
Diagnostic
Workstation
HIS Terminal
-!/~
Clinieal
Workstation
HIS Terminal
T~
Teleradiology
worl~tation
Film Printer Digital Digital
Radiography Radiography
Gateway Unit
Fig 1. Schematic diagram of PACS/HIS Architecture, Brooke Army Medical Center. HIS and PACS networks are independent and
connected through a gateway computer, lmportant sites of potential failure: {A) Failure of gateway or miniPACS server between
individual imaging modality and the PACS server. (B) Failure of the PACS server. (C) Failure of the HIS/PACS gateway or the HIS
server.
functions for the PACS, and interface functions to
RIS module of the HIS. It also is the central input
point for all imaging modalities and the film
digitizer and is the output source of image ¡ for
all viewing stations and laser film printers. Transfer
of all recently acquired images to long-term storage
on optical jukeboxes is made from the PACS server
after 2 to 5 days (Table
1).3,4
All plain films are acquired either through com-
puted radiography (8 computed radiograph [CR]
readers located both in the hospital and off site), or
directly through a digital radiography (DR) system.
Each of these CR or DR units requires a gateway
computer to translate vendor-speci¡ image for-
mats into DICOM3 format used by the image
server. All other modalities are interfaced using
DICOM3 compatible gateways with either indi-
vidual service miniPACS (10 cameras in nuclem-
Table 1. Current Major PACS Components
Equipment Function Model Vendor Software Installation
PACS Server Sparc 20 Sun Microsystems, Palo Alto, CA Solaris 5.4 1994
GE Advantage
v7.11.1
High speed 5.6 1999
GE Advantage
v7.11.1
Mitra Broker 1998
PACS Server UltraSparc 30
HIS to PACS Gateway Detl 2200
Sun Microsystems, Palo Alto, CA
Dell, Round Rock, TX
NOTE. Data frorn Brooke Arrny Medical Center. Although a large scale PACS has been in place at BAMC since 1993, software,
hardware, and major imaging devices have undergone extensive change and revision in almost all areas.

138 McBILES AND CHACKO
medicine, 6 units in ultrasound, and 5 fluoroscopy
units) or the individual image acquisition consoles
of each piece of equipment (3 spiral computed
tomography [CT] scanners, 2 1.5 Tesla magnetic
resonance [MR] units).
Five laser film imagers, largely a legacy from
pre-PACS installation, ate used mainly for hard
copy output when the patient is sent to another
institution and in very limited circumstances in the
operating room and for teaching uses. However,
they assume ah important role when some types of
PACS failure occur. The laser film imagers can be
reconnected directly to the CR and DR units during
some types of PACS failure.
Images are sent from the PACS server to high-
resolution radiology department viewing stations,
where formal readings are rendered, and short
preliminary reports are typed by the radiologist and
attached to the images on the PACS system. They
also ate immediately available to medium-resolu-
tion clinician terminals where both images and the
preliminary report can be viewed when either is
available. Images can be entered into the PACS
system independent of the HIS, but formal reports
can not be generated unless they are first created
and exist in the HIS database. Merging of PACS
images and HIS study request is performed by
radiology technologists through a rapid, transpar-
ent, and fault tolerant interface. The preliminary
radiologist typed report is not available on the HIS.
The HIS was developed independently of the
PACS as a Department of Defense-wide system. In
addition to a myriad of information management
functions including pathology, laboratory and clinic
administration, it contains a nongraphical RIS
module that performs the functions of scheduling,
patient demographics management, and radiology
report generation. The PACS system is independent
of the HIS system and only limited demographic,
study tracking, and image report data are allowed
through the HIS or PACS gateway. HIS terminals
are ubiquitous; they ate found in all clinician
offices, reception areas, radiology imaging areas,
and radiology reading rooms. All study requests,
except for those from outside the institution, ate
generated electronically by clinicians at their HIS
terminals. Radiology study reports are typed into
the HIS by transcriptionists and are available in
both preliminary and final veri¡ forro to the
clinicians within 4 to 48 hours. Importantly, the
final and preliminary reports are automatically
transferred from the HIS to the PACS system and
replace the brief typed radiologist report when they
are available.
HIS and PACS integration has eliminated hard
copy reading lists and readout books, because the
short impressions typed on the PACS are widely
available. Digital communication in medicine (DI-
COM) work lists generated by the PACS are
extremely efficient in managing the reading list
workload. A result of this PACS architecture is that
all online imaging studies are available instantly for
viewing at all radiology and clinical workstations,
making the image interpretation process and clini-
cian viewing of a study independent of a specific
location.
TYPES OF PACS DOWNTIME AND
TEMPORARY SOLUTIONS
HIS and HIS to PACS Gateway Failure
Unexpected failures occur approximately 5 times
a year and last 1 to 2 hours. No major workflow
modifications are necessary. If the HIS fails or is
brought down for scheduled maintenance or soft-
ware upgrade, then the traditional paper-based
system of study orde¡ is temporarily reinsti-
tuted. The process of merging PACS and HIS
studies performed du¡ the downtime must be
monitored carefully, because it easy to neglect to
enter studies into the HIS once they have already
been perforrned. A major impetus for performing
merging in our institution is the transcription
process, which is brought to a halt when the HIS
fails, because study demographics are assigned and
stored by the HIS (Fig l A).
PA CS Server Downtime
PACS server downtime precludes access to radi-
ology images on both radiology and clinician
viewing stations and requires major restructuring
of department workflow (Fig 1B). This downtime
has several causes. Within the last 3 years, the most
damaging failure was caused by an unexpected
crippling software failure, which rendered the
PACS server inoperable for 3 days. Less severe
unplanned failures lasting 15 minutes to 3 hours
have occurred approximately 2 to 3 times ayear.
There have been 3 planned major software up-
grades requiring 8 to 24 hours system downtime in
the last 3 years. Finally, the servers ate brought
down for approximately 1 to 2 hours late at night
once a week for database backup.

COPING WITH PACS DOWNTIME 139
During these failures, the HIS is still functional,
so soft copy study orders are still accomplished. As
soon as this type of failure is recognized or
anticipated, a plain film reading center is activated.
This involves the following actions.
1. Emergency services, orthopedic, intensive care
services, and clinics are notified immediately of the
nature of the failure, informed of the duration of the
expected downtime, and requested to schedule only
urgent studies. The reason for this request is that the
transition to film-based reading is very labor inten-
sive and will cause at least 1 hour to be lost before
smooth film-based reading processes are accom-
plished. Our experience has been that clinicians are
understanding and cooperative if informed immedi-
ately of changes in study availability.
2. If the PACS is brought offiine for a planned
reason, such as software upgrade, the last intensive
care unit film is printed routinely so future compari-
son can be made. Requests for similar printings
from other clinicians ate taken for patients in whom
comparison films may be needed during the ex-
pected downtime.
3. CR and DR images are stored on magneto-
optical (MO) discs for reloading on the PACS when
it becomes operational.
4. The laser film printers are disconnected from
the network and reattached to the CR or DR units in
stand-alone mode.
5. Intensive care unit, emergency room, and
orthopedic films are printed in duplicate and l copy
given to the referring clinician by placing it in a
spot easily accessible to the service. A second copy
is used by the radiologists in generating the official
report. Although this policy may seem wasteful of
film, it was instituted for several reasons. Film loss
rates in our hands and at other institutions has
histo¡ been 20% to 38%. 5 We feel it is
inappropriate to allow this to occur during ah
already trying time of system failure. Especially in
the intensive care unit setting, nonavailability of
previous studies can be disastrous. If the system is
down for only a short time, film costs ate minimal
when compared with the benefits of close film
control, rapid interpretation, and availability to
clinicians.
6. Individual services (except those in which
plain films are read) are placed in a stand-alone
mode in which image interpretation is performed
from the vendor-specific consoles or independent
mini-PACS. This increases communications bur-
den on the department. In our experience, a 50%
increase in telephonic and physical traffic to indi-
vidual services is expected, with concomitant dis-
ruption in efficiency of all radiology personnel.
7. During PACS downtime after hours, the laser
printers in MRI and CT are disconnected from the
PACS network and reconnected to the CT and MRI
consoles. Studies then are printed in single copy
and placed in the plain film reading center. Because
the PACS common interface for reading CT and
MRI are no longer available, efficient interpretation
is difficult for the lone on-call radiologist who
frequently is not proficient in use of the non-PACS
display software. This is not necessary during the
duty day when staffing is optimal and vendor
specific display software at imaging consoles can
be used for study interpretation by the individual
services. They generally have not been trained in
the use of these vendor-specific display interfaces.
In our institution, nuclear medicine and ultrasound
mini-PACS workstations continue to be used for
soft copy reading, because these systems are used
for day-to-day readout, and radiologists have expe-
rience in their use.
8. Additional radiology technologists are called
in. Conversion to a filrn-based reading system
requires a significant overhead cost in temas of film
processor maintenance, additional time to develop
films, film manipulation, and in creating a film and
interpretation center where none had existed be-
fore. Ultimately, when the PACS server becomes
functional additional personnel also will be needed
to insure studies performed during equipment fail-
ure are transferred adequately to the PACS system.
9. A plain film reading center is created. A major
shift in workflow to a traditional reading room style
is necessary and requires the following compo-
nents: (1) large-volume high-speed alternator near
a PACS viewing station so that side-by-side com-
parisons with prior studies can be made, (2) HIS
terminal, (3) hot light, (4) telephone, (5) transcrip-
tion device. (6) Also required are film storage bins
for temporary archiving films that have been read
and removed from the alternator. At our institution,
the films are stored in order of removal from the
altemator and the list of patients kept for quick
referral. Although alternate systems have been
tried, such as alphabetical filing, these have been
found to be more time consuming, and equipment
downtime has been short enough so that more
elaborate systems have not been needed. (7) Sepa-

140 McBILES AND CHACKQ
rate alternator is needed for intensive care unit
films and separate storage bins. Because compari-
son films are extremely important in this setting, all
films remain on the board until patient transfer.
This is especially important if the CR or DR
gateway is expected to be down more than 6 hours.
(8) Radiologists read the plain films, type impres-
sions into the PACS terminal, and dictate the
examination into the transcription system. Because
only the image transfer capability to PACs is
inoperative and all other HIS of PACS functions are
operational, impact on hospital func¡ is mini-
mal.
Although many of the actions and necessary
equipment outlined above and below may seem
obvious, only carefuI preptanning and positioning
of equipment will assure that appropriate actions
take place during the chaotic period after failure.
Our expe¡ has been that unused hard copy
reading equipment quickly disappears once soft
copy reading is embraced, and necessary backup
communication and viewing equipment frequenfly
is not available in a user-friendly environment once
hard copy reading is abandoned.
When the PACS system becomes operational.
studies that have been stored locally on miniPACS
or on MO discs (in the case of CR of DR units)
must be transferred to the PACS system. This
process must be monitored ctosely because 100%
transfer efficiency is not always accomplished
because of procedural errors in computer entry
during the hectic failure pe¡
Gateway Failure Between lmaging Modality or
Service and PACS or MiniPACS Failure
With the exception of CR or DR failures, these
failures usually have little impact on global system
and mainly are an inconvenience to the individual
service. Fortunately, miniPACS computers and the
gateways between major imaging devices rarely
fail, with failure rates generally less than once a
year for each gateway and less than twice ayear for
the miniPACS computer. Because more than 1
simultaneous gateway failure is highly unlikely and
has never happened in our system, a clue to PACS
server failure is the inability of multiple services to
communicate with the PACS. With the exception of
plain film radiology, all services have the cap•bility
of operating in a digital stand-alone mode either
within their own mini-PACS or individual imaging
consoles. There is a 2- to 7-day local image storage
capability, and so services (except for plain film
radiology) can function with little modification in
work¡ without contact with the PACS for ah
extended period. Clinicians usually accept tempo-
rary loss of access to these images on their own
local PACS terminals, and simply revert to pre-
PACS modes of physically traveling to the indi-
vidual service if they need to view a study. Because
the HIS and PACS ate not affected, electronic study
orde¡ transcription, and the practice of typing
preliminary reports on the PACS (albeit within a
"dummy" study without images) continue uninter-
rupted. When the mini-PACS/CR/DR gateway be-
comes functional, it is essential that a technologist
reviews the transfer of studies from the failure
pe¡ to insure all mini-PACS studies are merged
with the corresponding study on the PACS (Fig 1C).
CR reader and CR gateway failures have little
effect at our institution because of the ability of our
8 CR readers to absorb remaining workload among
them if one goes down.
DISCUSSION
Migration from a film-based to filmless depart-
ment can have enormous advantages in terms of
markedly improved study accountability, wide-
spread and rapid image and report availability,
rapid image retrieval and image comparison, and
flexibility of workspace configuration. Certain fail-
ures of the PACS may require varying degrees of
temporary reinstatement of a film environment.
The degree and exact mechanics of this conversion
are obviously dependent on the system configura-
t[on, the length of /he expected failure, legacy
equipment available, and level of training on
back-up systems. After 7 years of expe¡ with
large scale PACS, the following principles can be
offered for dealing with PACS failures:
1. Once a well-functioning PACS and HIS are in
place, there is considerable resistance to even a
temporary return to film-based reading by radiolo-
gists, technologists, and clinicians. Every effort to
preserve a digital imaging environment shou•d be
made. Although 100% reading of softcopy images
is highly desirable, achieving this goal at all times
may result in unacceptable expense of additional
backup equipment and service contracts. Espe-
cially when legacy equipment is available and
failures sufficiently infrequent, temporary return to
hardcopy interpretation may be a palatable and
cost-effective alternative.

COPING WITH PACS DOWNTIME 141
2. The longer the interval since film has been
routinely used, the less institutional memory is able
to recall the steps necessary to perform film based
reading. For technologists, detailed instructions on
film screen techniques and careful written instruc-
tions in setting upa film imaging center are nec-
essary.
3. Accurate diagnosis of the underlying problem
and rapid response by maintenance personnel is
crucial. Around the clock in-house availability of
technical personnel obviously is desirable, but may
be financially impractical. Because of the lack of
technical expertise, crashes during nonduty hours
have the greatest potential for disaster because both
misdiagnosis of the problem and inability to recog-
nize the seriousness of some types of failure are
more likely to occur. The result is sornetimes
painfully extended periods of system downtime.
This problem is best solved by education of radiolo-
gists and technologists in basic system architecture
and before-the-fact instruction on failure proce-
dures. Detailed instruction manuals with up-to-date
telephone numbers and well-defined procedures are
imperative, because even well-trained personnel
can rapidly loose sight of critical goals in the
chaotic and hectic times of system failure. Written
instructions on the types of failure warranting
emergent calls to maintenance personnel, along
with the authority to make these calls, also will
prevent sometimes expensive maintenance calls in
situations that can be dealt with less expensively
during duty hours.
4. The critical natnre of certain failures must be
recognized and maintenance contracts written to
reflect the need for rapid response in these situa-
tions. Expert consultation should be available readily
to key personnel, such as the chief technologist and
radiologist on duty during nonduty hours.
5. Planning during equipment acquisition for
the inevitable long-term (greater than 12 hours)
PACS failure should include adequate local memory
storage for at least 2 days worth of patient data, a
backup system of transfer, or both. Special atten-
tion to the transparency, speed and ease of use of
the backup system, and careful consideration of
failure situations is crucial. In many instances
additional software of equipment may need to be
purchased to assure a smooth transition in failure
situations. An example at our institution of these
situations is the need for MO disc drives and
additional printer drivers for the CR readers.
6. Uninterruptable power supplies for critical
components are mandatory. Claims from engineer-
ing personnel touting the reliability of facility
electrical power should be regarded with skepti-
cism. As a minimum, gateway servers, miniPACS
servers, and the PACS and HIS systems sbould
have uninterruptable power supplies.
7. Additional help should be brought in during
severe PACS failures. Expecting the skeleton crews
working during off duty hours to perform ad-
equately at the inevitable markedly increased work-
loads (25% to 50% above normal) 6 during these
times increases the chance for significant error and
clinician dissatisfaction with the radiology product.
8. Expertise and supplies to support temporary
film-based reading must be maintained. Because
many film-based supplies have a short shelf life,
only limited quantities can be expected to be on
hand in the event of severe failures. Logistic
support must be planned and available during these
emergencies. Dark rooms and at least some ves-
tiges of the traditional reading room must be
maintained.
Many of our poticies are institution specific and
stem from the relative independence of the H1S and
PIS module of our PACS. For example, the ability
to enter preliminary reports into our PACS system
is an important feature at our institution because of
the widespread availability of PACS terminals
within the hospital. Institutions with limited PACS
availability to clinicians (or lack of a similar
capability on their HIS) will need to modify their
system of preliminary report dissemination. Simi-
larly, institutions at which the HIS and PACS are
more closely integrated, possibly running on the
same platform with integrated software, will re-
quire even closer attention to failure algorithms,
because hardware failure is likely to affect both
systems.
Another institution-specific issue is our availabil-
ity of adequate emergency filming capabilities in
the forro of laser film imagers and adequate CR and
DR assets. Departments without adequate film
printing assets may need to adopt alternate tech-
niques. At one extreme, these techniques may use
more extensive use of film screen and wet proces-
sor capabilities. An alternative approach is to
attempt to avoid hard copy interpretation entirely
by purchase of backup gateways, workstations,
power supplies, and extensive training in backup

142 McBILES AND CHACKO
soft copy interpretation on the various vendor
specific workstations for each modality. This latter
approach was considered and rejected at our institu-
tion because of prohibitive cost, availability of
legacy laser imagers, unwieldy logistics of recon-
nection, and inadequate timely access for clinicians
to the soft copy images.
Individual service preference of soft copy inter-
pretation also dictates procedures during PACS
failure. In our institution, ultrasound interpretation
is performed on ultrasound miniPACS because
color is not available on the PACS system. Simi-
larly, nuclear medicine studies are interpreted on an
independent miniPACS for the same reason and
because of the lack of cine loop review on the
PACS system. In these services, PACS capabilities
determine the extent of PACS use during routine
image interpretation. As expected, PACS failures
affect these services to a lesser extent. Institutions
with different levels of PACS sophistication and the
resulting greater or lesser reliance on PACS may
find that they ate affected in significantly different
ways by PACS failure.
Many of the processes outlined above during
PACS failure should have applicability at other
institutions with heavy reliance on digital imaging
and PACS. Our relatively independent HIS and
PACS architecture, and the spoke and wheel con-
figuration of our PACS present problems that may
apply less with the newer distributed archiving
architecture. Nevertheless, the above outlined gen-
eral types of failure can and will occur with
disconcerting frequency regardless of architecture.
Careful planning can minimize the effects these
failures on the hospital and the radiology depart-
rnent,
REFERENCES
1. Thompson TL: PACS speed up reading, except when they
crash. Diagnostic Imaging IR 38-40, 1998 (suppl)
2. Woosley B: The death of filmless radiology. Diagnostic
Imaging 21:51, 1999
3. Smith DV, Smith S, Bender GN, et al: Evaluation of the
medical diagnostic imaging system based on 2 years of clinical
experience. J Digit lmaging 8:75-87, 1995
4. Carrino JA, Unkel PJ, Miller ID, et al: Large-scale PACS
implementation. J Digit Imaging 11:3-7, 1998
5. Leckie R, Goeringer F, Smith D: Eady evaluation of
MDIS workstations at Madigan Anny Medical Center. SPIE
1897:336-348, 1993
6. Reiner Bt, Siegel EL, Hooper FJ, et al: Effect of film-based
versus filmless operation on the productivity of CT technolo-
gists. Radiology 207:481-485, 1998