Same system, different outcomes: comparing the transitions from two paper-based systems to the same computerized physician order entry system.

Abstract

To compare how nurses in two different paper-based systems perceive the impact of a computerized physician order entry (CPOE) system on their medication-related activities.
13 non-surgical, adult inpatient wards in a Dutch academic hospital.
Questionnaire survey of 295 nurses before and 304 nurses after the implementation of a CPOE system. These nurses worked with two different paper-based medication systems before the implementation: 'Kardex-system' and 'TIMED-system'. In the Kardex-system, the structure of the nursing medication work was similar to that of after the CPOE implementation, while in the TIMED-system, it was different. 'Adaptive Structuration Theory' (AST) was used to interpret the results.
The response rates were 52.2% (154/295) before and 44.7% (136/304) after the implementation. Kardex-nurses reported more positive effects than TIMED-nurses. TIMED-nurses reported that the computerized system was more inflexible, more difficult to work with, and slower than the TIMED-system. In the TIMED group, the overall mean score of the computerized process was not significantly different from that of the paper-based process. Moreover, nurses in both groups were more satisfied with the post-implementation process than with the pre-implementation process. Nevertheless, none of groups reported a better workflow support in the computerized system when compared to that of the paper-based systems.
Our findings suggest that not only the technology but also large differences between pre- and post-implementation work structure influence the perceptions of users, and probably make the transition more difficult. This study also suggests that greater satisfaction with a system may not necessarily be a reflection of better workflow support.

Full text

Page 1

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
Same system, different outcomes: Comparing the
transitions from two paper-based systems to the same
computerized physician order entry system
Zahra Niazkhania,∗, Heleen van der Sijsb, Habibollah Pirnejada,
William K. Redekopc, Jos Aartsa
aInstitute of Health Policy and Management (iBMG), Erasmus University Medical Center, Rotterdam, The Netherlands
bDepartment of Hospital Pharmacy, Erasmus University Medical Center, Rotterdam, The Netherlands
cInstitute for Medical Technology Assessment (iMTA), Erasmus University Medical Center, Rotterdam, The Netherlands
a r t i c l ei n f o
Article history:
Received 20 November 2007
Received in revised form 5 June 2008
Accepted 28 June 2008
Keywords:
Evaluation Studies
Prescriptions, Drug
Medication systems, Hospital
Medical Order Entry Systems
Computer Communication
Networks
Questionnaires
a b s t r a c t
Objective: To compare how nurses in two different paper-based systems perceive the impact
of a computerized physician order entry (CPOE) system on their medication-related activi-
ties.
Setting: 13 non-surgical, adult inpatient wards in a Dutch academic hospital.
Methods:Questionnairesurveyof295nursesbeforeand304nursesaftertheimplementation
of a CPOE system. These nurses worked with two different paper-based medication systems
beforetheimplementation:‘Kardex-system’and‘TIMED-system’.IntheKardex-system,the
structure of the nursing medication work was similar to that of after the CPOE implemen-
tation, while in the TIMED-system, it was different. ‘Adaptive Structuration Theory’ (AST)
was used to interpret the results.
Results: The response rates were 52.2% (154/295) before and 44.7% (136/304) after the imple-
mentation.Kardex-nursesreportedmorepositiveeffectsthanTIMED-nurses.TIMED-nurses
reported that the computerized system was more inflexible, more difficult to work with, and
slower than the TIMED-system. In the TIMED group, the overall mean score of the computer-
ized process was not significantly different from that of the paper-based process. Moreover,
nurses in both groups were more satisfied with the post-implementation process than with
the pre-implementation process. Nevertheless, none of groups reported a better workflow
support in the computerized system when compared to that of the paper-based systems.
Conclusions: Our findings suggest that not only the technology but also large differences
between pre- and post-implementation work structure influence the perceptions of users,
and probably make the transition more difficult. This study also suggests that greater sat-
isfaction with a system may not necessarily be a reflection of better workflow support.
© 2008 Elsevier Ireland Ltd. All rights reserved.
∗Corresponding author at: Institute of Health Policy and Management, Erasmus University Medical Center, P.O. Box 1738, 3000 DR Rotter-
dam, The Netherlands. Tel.: +31 10 408 8811; fax: +31 10 408 9094.
E-mail address: z.niazkhani@erasmusmc.nl (Z. Niazkhani).
1386-5056/$ – see front matter © 2008 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.ijmedinf.2008.06.012

Page 2

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
171
1.Introduction
The implementation of a computerized physician order entry
(CPOE) system is considered as a pivotal transitional step
towards the more effective management of medications [1].
A CPOE system is defined as a computer application where
a physician directly enters medical orders. Because nurses
are also involved in patient care, they inevitably interact with
these systems or their outputs. Studies have shown that a
CPOEsystemcaneliminateanumberofintermediatestepsfor
nurses. For example, they no longer have to deal with illegi-
ble and incomplete hand-written orders, which are a common
source of extra workload for nurses [2–4]. The system, more-
over, facilitates order communication to other parties such
as the pharmacy, which in turn saves considerable time for
nurses [5,6].
However, something which has recently received consider-
able attention is the extent to which these systems change the
nature of workflow for health professionals, including nurses
[7–9]. In fact, in addition to the literature that reports bene-
fits of CPOE systems, there is a growing number of studies
that focus on unintended changes in many aspects of work-
flow following the implementation [8,10]. Beuscart-Zephir et
al. described the role of nurses in distributed decision mak-
ing in the medication ordering and administration process
[11]. Coleman observed that nurses normally interpret physi-
cians’ intents in their orders [12]. Therefore, if nurses were
to be bypassed after implementation of a CPOE system, the
system would not be able to handle this interpretation effec-
tively. Both studies criticized the fact that the organizational
role of nurse was ignored during the design of CPOE systems
[11,12]. Moreover, in a study of perceived impact of CPOE sys-
tems, nurses reported a sense of loss of control over their
work [13]. Goorman and Berg argued that at least some of
the problems with these systems occur because of the clash
betweenthenursingworkflowmodelembeddedinthesystem
and actual nursing practice [14]. This evidence indicates that
nurse-related medication activities, and more importantly
their organizational role in the medication process, deserve
more attention in the design, implementation and evaluation
of CPOE systems.
Depending on different work organizations, nurses may
be assigned different roles and responsibilities. As the imple-
mentation of CPOE systems brings a new work organization
along with it, this unavoidably transforms their roles and
activities. The study of how nurses perceive this transforma-
tion in the transition from a paper-based to a computerized
work structure can give insight into how this transition can
effectively be managed. In 2001, a computerized medication
order entry system was implemented in a Dutch academic
hospital. Several different paper-based medication systems
were in use before the implementation. To compare the per-
ceived impact of this CPOE system on nursing medication
practice, we conducted a before-and-after study in two differ-
ent paper-based medication systems. In particular, we com-
pared the perceived benefits and/or drawbacks of the com-
puterized system with those of the two different paper-based
systems. We also examined nurse satisfaction and perceived
workflow support before and after CPOE implementation.
2.Theoretical background
We used the ‘Adaptive Structuration Theory (AST)’ [15] as a
theoretical framework to study the changes that occurred
in two different work practices following the CPOE imple-
mentation. AST is based on Anthony Giddens’ Structuration
Theory [16]. This theory is formulated as “the production and
reproduction of the social systems through members’ use
of rules and resources in interaction”. DeSanctis and Poole
adaptedGiddens’theorytostudytheinteractionofgroupsand
organizations with information technology (IT), and called it
‘Adaptive Structuration Theory’ [15].
ASTcriticizesthetechnocentricviewoftechnologyuseand
emphasizes its social aspects. This theory focuses on “social
structures, rules and resources provided by technologies and
institutions as the basis for human activity” (page 125) [15].
The social structures in this theory include the technology
itself, the content and constraints of a given work task, the
organizational environment, corporate information, histories
of task accomplishment, cultural beliefs, modes of conduct
and so on. These structures act as templates for planning
and accomplishing tasks and may vary across groups. Design-
ers incorporate some of the structures of institutions into
the technology; the structures may be reproduced so as to
imitate their non-technology counterparts, or they may be
modified, thus creating new work structures within the tech-
nology. The AST helps to explain the different outcomes after
the implementation of one information system in different
work structures.
3.Study context
3.1.Study environment and the CPOE system
This study was conducted at Erasmus University Medical
Center (Erasmus MC) in Rotterdam, a 1237-bed academic
hospital in the Netherlands. We studied a commercially
availablecomputerizedmedication
named Medicatie/EVS®. To retrieve patient and drug data,
Medicatie/EVS communicates with the existing hospital infor-
mation system (HIS) and patient medical record (Patient 98).
This system was first piloted in six wards of two specialties
fromDecember2001toDecember2002.Itwasfollowedbysub-
sequent implementation in 39 wards from September 2003 to
March 2005.
orderentrysystem
3.1.1.
administration processes
Before the CPOE implementation, Erasmus MC had two differ-
ent paper-based systems on adult wards: Kardex and TIMED.
IntheKardex-system,toprescribemedications,physicians
wrote a drug’s name, dosage form, dosage regimen, adminis-
tration route, start date, and exact administration time on a
special tear-off order form with two additional carbon copies
(Fig. 1a). Nurses could add missing information (e.g., dosage
form or strength), but no transcription took place. The original
order was put on a Kardex-card for registration of drug admin-
istration. This was registered by signing next to the order on
Two paper-based medication ordering and

Page 3

172
international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
the Kardex-card. This registration form had room for ten days
and after that a new form for the next ten days could be added
on the card. To request non-stock items, nurses had to manu-
ally write drug requests and send them to the pharmacy. For
urgent medications unavailable in the ward stock, nurses had
to refer to the pharmacy personally with the hand-written
drug requests. These requests then were entered into the
HIS by pharmacy technicians. These processes in the Kardex-
system are shown in Fig. 2.
In the TIMED-system, physicians wrote a medication’s
name,dosageregimen,administrationroute,andstartdateon
a pre-printed slip (Fig. 1b). A nurse had to transcribe a physi-
cian’s orders, select the suitable dosage forms available in the
hospital,andchoosetheiradministrationtimes.Thenurse,for
instance, translated the dosage regimen of an order of three
timesdailyintotheexactadministrationtimesduringtheday.
The transcribed orders together with an administration regis-
tration form then were put in the patient medication chart.
Each day a new administration form was put next to the tran-
scribed order form (Fig. 2). Once a drug was administered, this
was registered by sticking the flag labels of the administered
drugs on the administration form. Whenever flag labels were
absent, the nurse had to write the drug name and dosage on
the administration form. For urgent and also non-stock med-
ications, the same procedures as in the Kardex-system were
followed.
For the sake of clarity, we will refer to ‘Kardex units’ as
wards which used the Kardex-system before the implemen-
tation. Likewise, ‘TIMED units’ are those wards that had the
TIMED-system.
3.1.2.
administration process
The CPOE system is available in all physician offices as well as
in all workstations throughout the hospital. Only physicians
The computerized medication ordering and
and midwives are authorized for electronic order entry in this
hospital. Physicians must enter their medication orders into
the system; nurses may not accept any hand-written prescrip-
tion. A physician enters a medication order by selecting a drug
and its dosage form, strength, administration route, dosage
regimen, start date and time. A detailed description of the pre-
scription process with the Medicatie/EVS has been published
elsewhere [17]. After electronic ordering, medication orders
are printed on special labels called Medication Order (MO)-
labels (Fig. 1c). Nurses were trained in groups to work with the
system.
The printed labels are affixed to a Kardex-card which is
specific for the medication administration record. Next to
the MO-labels on the Kardex-card, nurses are supposed to
provide a signature once they give medications to patients.
Transcription of a prescription by the nurse is no longer nec-
essary. Nurses are notified about the availability of drugs in
the ward stock by means of codes specified on the MO-labels
(“J”=available, “N”=unavailable). Pharmacy technicians con-
trol the supply of in-stock items by scanning them at wards
two or three times a week. Whenever an MO-label contains
a drug that is out of stock, nurses can select it in the system
and thereby sent an electronic drug request to the pharmacy.
Technicians in the pharmacy check these non-stock drug
requests twice a day at 8 o’clock and 12 o’clock and provide
the requested drugs later that day. The process after the CPOE
implementation is shown in Fig. 3.
Comparison of Figs. 2 and 3 shows that the medication
orderingandadministrationprocessaftertheimplementation
resembles that of the Kardex-system, while it is completely
different from that of the TIMED-system. In both Kardex and
TIMED units, we compared nurse attitudes towards the com-
puterized process in the post-implementation phase with
their attitudes towards the paper-based process in the pre-
implementation phase.
Fig. 1 – The medication orders in different medication systems. (a) Kardex system, (b) TIMED system, and (c) CPOE system.

Page 4

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
173
Fig. 2 – The medication ordering and administration processes in Kardex-system and TIMED-system; MO (Medication
Order); HIS (Hospital Information System); NS (Non-Stock); for requesting urgent NS drugs, nurses often directly referred to
the pharmacy with hand-written requests.
4. Methods
4.1.Study design and measurements
Our evaluation was based on questionnaire administered to
nurses before and after the CPOE implementation. Design of
the questionnaire was based in part on previously published
questionnaires for the assessment of user satisfaction with
CPOE – such as [18] – and was done in a close collaboration
with nursing staff. In addition to demographics, the original
questionnairecontained28–40questionstomeasureattitudes
regarding the paper-based systems (Kardex and TIMED) and
theCPOEsystem.Inthepresentstudy,wereportontheresults
of the questions that were similar in the questionnaires used
in the three systems. The list of these questions is available as
supplementary material. These questions asked respondents
Fig. 3 – The medication ordering and administration process after the CPOE system; this system is available in all the
computers throughout the hospital; MO-label (Medication Order-label); NS (Non-Stock).

Page 5

174
international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
about: overall reaction towards the medication process (1.1,
1.2, and 1.3), the characteristics of medication orders (2.1, 2.2,
and 2.3), registration of drug administration (3.1, 3.2, 3.3, 3.4,
and 3.5), the learning and speed of the process (4.1 and 4.2),
and managing the non-stock medication orders (5.1 and 5.2).
Thesequestionsweredesignedtoevaluatetheattitudesbased
on a 5-point Likert scale. We also included two other ques-
tions which asked the respondents to give their impression of
workflow support (6.1) and system preference (6.2).
The questionnaire was checked for the applicability and
understandability of its wording by two nurses in each sys-
tem. The questionnaire was considered ready for distribution
after modifications suggested by these nurses. In each phase,
a packet containing the questionnaire and a cover letter
explaining the aim of the study were distributed by head
nurses among all nurses in the target wards. The completed
questionnaires were collected by the head nurses or directly
sent to the researcher via the hospital’s internal mail ser-
vice.
4.2. Course of the study and participants
Nurses working in 13 non-surgical, adult inpatient wards
were chosen to participate in this study. Six wards used the
Kardex-system and consisted of Psychiatry (three wards), and
HematologyandOncology(threewards).Sevenwardsusedthe
TIMED-system and consisted of Internal Medicine (six wards)
andNeurology(oneward).TheCPOEsystemwasimplemented
in these wards one after another. The questionnaires were
sent two weeks before and approximately five months after
the introduction of the CPOE system. Since the introduction of
the system across the hospital was conducted in a step-wise
basis, the distribution of the questionnaire in both phases fol-
lowed the implementation order (September 2003 to October
2004).
All nurses who were working in the selected units dur-
ing the course of this study were invited to participate. In
the pre-implementation phase, 295 nurses received the ques-
tionnaire, of whom 154 nurses responded (52.2%). In the
Table 1 – Characteristics of survey respondents
CharacteristicsKardex units TIMED units
Pre-implementation
N (%)
Post-implementation
N (%)
Pre-implementation
N (%)
Post-implementation
N (%)
Number of questionnaires distributed
Number of respondents
144
66
144
48
151
88
160a
88
Specialty
Psychiatry
Hematology and oncology
Internal medicine
Neurology
Female
30(45.5)
36(54.5)
–
–
54(81.8)
23(47.9)
25(52.1)
–
–
35(72.9)
–
–
–
–
76(86.3)
12(13.7)
71(80.7)
73(83.0)
15(17.0)
73(83.0)
Age (years old)
≤23
24–33
34–43
44–53
≥54
Professional status
Practicing nurse
Head nurse
Others
1(1.5)
20(30.3)
18(27.3)
21(31.8)
5(7.6)
1(2.1)
14(29.2)
10(20.8)
18(37.5)
1(2.1)
13(14.8)
25(28.4)
19(21.6)
22(25.0)
5(5.7)
17(19.3)
27(30.7)
17(19.3)
22(25.0)
3(3.4)
52(78.1)
7(10.6)
6(9.1)
38(79.2)
7(14.9)
2(4.2)
65(73.9)
6(6.8)
16(18.2)
63(71.6)
8(9.1)
16(18.2)
Home use of computer
Never
Sometimesb
Regularlyc
Oftend
7(10.6)
11(16.7)
14(21.2)
33(50.0)
5(10.4)
5(10.4)
12(25.2)
25(52.1)
4(4.5)
5(5.7)
31(35.2)
46(52.3)
8(9.1)
11(12.5)
27(30.7)
41(46.6)
Use of computer at work
Never
Sometimes
Regularly
Often
Prior experience with an electronic
prescription system
0(0)
9(13.9)
20(30.3)
36(54.5)
6(9.0)
1(2.1)
0(0)
14(29.2)
32(66.7)
5(10.5)
1(1.1)
5(5.7)
17(19.3)
62(70.5)
7(7.9)
0(0)
4(4.5)
21(23.9)
60(68.2)
10(11.3)
aNine nurses who had been forgotten in the first phase received the questionnaire only in the second phase.
bOnce a month or less.
cOnce in a week to few times per month.
dDaily to few times per week.

Page 6

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
175
post-implementation phase, 304 nurses were contacted, of
whom 136 nurses responded (44.7%). Overall, 290 question-
naireswerereturned.Twohundredandelevennurses(70.56%)
participated in at least one phase of the study. In total, at least
79 nurses were identified as nurses who participated in both
phaseswhile132nursescompletedonlyonequestionnaire.As
therecordingoftheidentificationnumberwasnotmandatory,
it is possible that more nurses answered the questionnaires
in both phases. One nurse in the post-implementation phase,
who did not use computers at work, was excluded from the
analysis of one question requiring the use of computers at
work.
4.3. Data analysis
Analyses using the Kolmogorov-Smirnov (Lilliefors) test
revealed that the scores provided by the respondents were
often not normally distributed. Therefore, we tested for differ-
ence between scores for before and after the implementation
using the Mann–Whitney U test. Overall scores of the sys-
tems, which were normally distributed, were compared using
the t tests. The t tests were also performed to test for dif-
ferences between the change scores (mean differences and
standard errors in paper vs. electronic system) in Kardex units
and change scores in TIMED units. 95% confidence intervals
(CIs) for means are reported. We used the Spearman correla-
tion coefficient to measure the degree of association between
variables and overall satisfaction with the computerized pro-
cess.TheMann–WhitneyUtestswereperformedtodetermine
changes in ratings of the preference of the systems and the
perceivedsupportofworkflow(items6.1and6.2)betweenpre-
and post-implementation. We measured the internal consis-
tency of the questions (1.1–5.2) using Cronbach’s Alpha. An
alpha level of .05 was used for all statistical tests. All statisti-
cal analyses were performed using SPSS for Windows (version
14).
5.Results
Table 1 provides the demographics of the different study
groups. Most nurses were women, practicing nurses, often
used computers both at home and at work, and had no
prior experience with an electronic prescription system.
With regard to demographics, there were no important
differences neither between respondents of pre- and post-
implementation phases nor between respondents in Kardex
units and respondents in TIMED units. Cronbach’s Alpha for
questions 1.1–5.2 was 0.84 for the paper-based and 0.88 for
the computerized process, representing a high internal con-
sistency of the questionnaire.
5.1.
post-implementation, and between Kardex and TIMED
units
Comparison between pre- and
5.1.1.
An overall mean score for each nurse was calculated by
summing the scores for the 15 items of the questionnaire
(1=minimum, 5=maximum). Afterwards, the overall mean
Overall mean scores
Fig. 4 – Box plots showing median, interquartile range, and
whiskers for overall scores of the process in the
paper-based systems and the electronic system in different
units; in Kardex units the overall score improved after the
implementation, p<.001; however, in TIMED units, this
improvement was not significant, p>.05.
score was calculated for pre- and post-implementation in
Kardex and TIMED units. Kardex-nurses, whose medication
process after the implementation was similar to that of before
the implementation, gave a higher mean score for the com-
puterized process than for the paper-based process (3.6 vs.
3.2; p<.001). However, in TIMED units in which the elec-
tronic system brought a completely new work structure for
nurses, the overall mean score of the computerized process
was not significantly different from that of the paper-based
process (3.5 vs. 3.4, p>.05). Fig. 4 shows median, interquar-
tile range, and whiskers for the overall scores of pre- and
post-implementation in these two units. Considering the
similarity or differences of work structures in pre- and post-
implementation phases, there was a significantly greater
improvement in the overall score for Kardex units (i.e., from a
paper-based to a computerized process) than for TIMED units
(mean difference=−0.29; CI: −0.55, −0.02; p<.05).
Table 2 summarizes the mean scores for individual ques-
tions and the significance of difference between pre- and
post-implementation within Kardex or TIMED units. It also
provides the significance of difference between these units
with regard to the change scores.
5.1.2.
In Kardex units, there was greater satisfaction with the com-
puterized process than with the paper-based process (3.7 vs.
2.9, p<.001) (item 1.3). This was also the case for TIMED units,
where satisfaction was 0.3 points higher with the comput-
erized process than with the paper-based process (3.5 vs.
3.2, p<.05). The comparison between units showed that the
improvement in satisfaction (from paper-based to computer-
ized process) was significantly greater in Kardex units than in
TIMED units (mean difference=−.5; CI: −.93, −.01; p<.05).
In contrast with Kardex units, nurses in TIMED units
reported that their paper-based process was significantly
Overall reactions

Page 7

176
international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
Table 2 – Mean scores to the individual questions in the Kardex and TIMED units categorized based on the paper-based and computerized systems
Item#Questionnaire items Kardex unitsTIMED units
p value*
Mean
p value†
Mean
p value‡
Pre-
implementation
(No.)
Post-
implementation
(No.)
Pre-
implementation
(No.)
Post-
implementation
(No.)
1.1Overall impression
(difficult...easy)
Overall impression
(rigid...flexible)
Overall impression
(frustrating...satisfying)
The layout of the medication
orders (confusing...clear)
The legibility of the medication
orders (bad...good)
The completeness of the
medication orders (bad...good)
The arrangement of data
(unhelpful...helpful)
The layout of the drug overview
form (confusing...clear)
The drug overview form
(unreliable...reliable)
The administration record
(confusing...clear)
The administration record
(unreliable...reliable)
The learning of the system
(difficult...easy)
The speed of prescribing by
physicians (too slow...fast
enough)
Insight over available drugs in
the pharmacy
(unsatisfactory...satisfactory)
The ordering of non-stock drugs
(difficult...easy)
3.7 (64)3.8 (48) NS§
4.2 (87)3.9 (86)
†††
1.23.2 (64)3.3 (46) NS3.5 (83) 3.0 (85)
†††
1.32.9 (62) 3.7 (46)
†††
3.2 (84) 3.5 (84)
††
2.1 3.3 (64)3.4 (48)NS3.7 (88)3.6 (87)NS NS
2.22.1 (66) 3.9 (48)
†††
2.8 (88)4.0 (88)
††† ††
2.32.9 (66)3.7 (48)
†††
3.4 (88) 3.9 (88)
†††
NS
3.13.5 (66)3.7 (48)NS3.7 (88) 3.8 (88)NSNS
3.23.4 (66)3.6 (48) NS3.5 (88)3.8 (88)
††
NS
3.32.9 (66)3.6 (47)
†††
3.2 (87)3.7 (88)
†††
NS
3.42.9 (66)3.5 (48)
††
3.1 (88)3.3 (88)NS
†
3.5 3.0 (65)3.4 (46)
††
3.2 (88)3.1 (88)NS
†
4.14.0 (65)3.9 (48)NS4.2 (88) 3.8 (87)
††
NS
4.2 3.2 (66)3.5 (47) NS3.3 (88)2.8 (86)
†††††
5.1
2.9 (66)3.2 (47)NS2.0 (86) 2.8 (85)
††††
5.23.7 (64)3.7 (47)NS 3.4 (88) 3.4 (83)NSNS
∗p value by ‘t test’ for difference between the change scores in Kardex units and change scores in TIMED units.
†p value by ‘Mann–Whitney U test’ between pre- and post-implementation in Kardex units.
‡p value by ‘Mann–Whitney U test’ between pre- and post-implementation in TIMED units.
§NS: not significant; †: p<.05; † †: p<.01; † † †: p<.001.

Page 8

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
177
Table 3 – Relationship between overall score of the computerized process and ratings of its specific aspects
Survey itemsa
Item #
nb
rc
Significance
1.1
1.2
1.3
2.1
2.2
2.3
3.1
3.2
3.3
3.4
3.5
4.1
4.2
5.1
5.2
7.2
7.3
7.4
Overall impression (difficult...easy)
Overall impression (rigid...flexible)
Overall impression (frustrating...satisfying)
Layout of the medication orders (confusing...clear)
Prescription legibility (bad...good)
Completeness of medication orders (bad....good)
Arrangement of data (unhelpful...helpful)
Layout of the drug overview form (confusing...clear)
Drug overview form (unreliable...reliable)
Administration record (confusing...clear)
Administration record (unreliable...reliable)
Learning of the system (difficult...easy)
Speed of prescribing by physicians (slow...fast enough)
Insight over available drugs in the pharmacy (unsatisfactory...satisfactory)
Ordering of non-stock drugs (difficult...easy)
Specialty
Gender
Age categories
134
131
130
135
136
136
136
136
135
136
134
135
133
132
131
136
134
130
.628
.571
.748
.609
.493
.531
.608
.627
.568
.657
.585
.550
.467
.486
.456
.204
.171
.182
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.001
<.05
<.05
<.05
aItems 1.1 to 5.2 are based on a 5-point likert scale.
bNot all of the 136 respondents after the implementation answered each question.
cSpearman correlation coefficient (rho) for the relationship between each item and overall score.
easier to use than the computerized process that was com-
pletely new for them (4.2 vs. 3.9, p<.01) (item 1.1). They also
rated their paper-based process significantly more flexible
than the computerized process (3.5 vs. 3.0, p<.01) (item 1.2).
Moreover, learning of the computerized process in TIMED
units was more difficult than learning of the paper-based pro-
cess (3.8 vs. 4.2, p<.01). Kardex-nurses however reported it
to be as easy as their paper-based process (3.9 vs. 4, p>.05)
(item 4.1). TIMED nurses believed that the ordering process by
physicians was slower in the computerized system compared
with the paper-based system (2.8 vs. 3.3, p<.01) (item 4.2). The
change in attitudes towards speed in TIMED units was signif-
icantly greater than the change seen in Kardex units (mean
difference=−0.8; CI: −1.3, −0.4; p<.001). This difference prob-
ably arose because the electronic system completely changed
the structure of medication ordering and administration pro-
cess not only for TIMED-nurses but also for TIMED-physicians.
Specifically, it required these physicians to enter detailed
orders into the system instead of concise, paper-based medi-
cation orders (Fig. 1b).
5.1.3.
Nurses in both groups rated the legibility and completeness
of medication orders (items 2.2 and 2.3) higher in the com-
puterized system than in the paper-based system (p<.001)
(Table 2). Interestingly, the degree of this improvement related
to legibility was perceived to be higher in Kardex units than in
TIMED units (mean difference=−0.64; CI: −1.1, −0.2; p<.01).
The computerized system did not change the clarity of the
layout in medication orders (item 2.1) compared with that of
the paper-based orders, neither in Kardex nor in TIMED units
(p>.05).
Characteristics of medication orders
5.1.4.
In both units, the computerized system was associated with
a greater reliability of the drug overview form (p<.001) (item
Registration of administration
3.3). In Kardex units, the clarity of administration record in the
computerizedsystemwassignificantlygreaterthanthatinthe
paper-based system (3.5 vs. 2.9, p<.01) (item 3.4). Likewise, the
administration record in the computerized system was per-
ceived to be more reliable than that in the paper-based system
(3.4 vs. 3.0, p<.01) (item 3.5). However, after the implementa-
tion, TIMED-nurses, who no longer used the flag labels of the
administered drugs for registration purposes, reported no any
significant difference between pre- and post-implementation
phases with regard to the clarity and reliability of the admin-
istration record (p>.05) (items 3.4 and 3.5).
5.1.5.
TIMED nurses reported a better insight regarding the available
drugs in the pharmacy after the implementation compared
with the situation before (2.8 vs. 2.0, p<.001) (item 5.1). How-
ever, in Kardex units, this difference between two phases was
not significant (p>.05). Moreover, the electronic system did
not facilitate the ordering of non-stock medications (item 5.2),
neither in Kardex nor in TIMED units (p>.05) (Table 2).
Drug delivery request from the pharmacy
5.1.6.
When nurses were asked whether they wanted to change the
current process (item 6.2), the majority of the respondents
in both Kardex (97.91%) and TIMED units (79.31%) preferred
to continue using the computerized system. However, as the
Mann–Whitney U tests showed, nurses believed that the com-
puterized system did not support their work processes more
than the paper-based systems did (item 6.1); this was the case
for both Kardex (p=.07) and TIMED units (p=.38).
System preference and workflow support
5.2.
process
Correlation of overall score of the computerized
Overall score of the computerized process was strongly cor-
related with the user satisfaction (r=0.75, p<.001), clarity

Page 9

178
international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
of administration record (r=0.66, p<.001, ease of the pro-
cess (r=0.63, p<.001), and clarity of layout of drug overview
form (r=0.63, p<.001). Overall score was not correlated with
professional status, experience with computers at home or
work, and the wards where nurses work. Table 3 summa-
rizes the correlates of overall score in the computerized
process.
6.Discussion
Our study showed that although the system eliminated
the workload of transcription and translation tasks for
the TIMED-nurses, they showed a less positive attitude
compared with the Kardex-nurses. Nurses in both groups
benefited from improved legibility and completeness of
medication orders and greater reliability of drug overview
after the implementation. TIMED-nurses were sometimes
less positive about the new medication process than the
pre-implementation process, with the new process being
associated with increased rigidity and difficulty, and a
slower physician prescription speed. These increased prob-
lems were not seen by the Kardex-nurses. Furthermore,
in TIMED units, the computerized process did not result
in a significantly higher overall score compared with the
paper-based process. The overall score of the computerized
process was most strongly correlated with user satisfaction,
clarity of the administration record, and easiness of the
process.
Moreover, in our study, nurses in both groups found that
the computerized system did not support workflow better
thanthepaper-basedsystems.However,nursesinbothgroups
favored the medication process after the implementation over
the situation before the implementation. It is often expected
thattheprocessfollowingcomputerizedphysicianorderentry
becomes more efficient for nurses [19]. However, our cases
showed that in spite of a higher satisfaction and perceived
advantages of the system, none of groups reported a greater
workflow support after the implementation compared to that
of the paper-based systems. Similar to this finding, in one
survey of ICU nurses, 56.7% of respondents commented that
the post-CPOE workflow had become less efficient, although
they were also generally satisfied with the system [20]. These
findings therefore imply that workflow support with a CPOE
system is a multi-layered issue and suggest that greater sat-
isfaction with a system is not necessarily a reflection of
bettersupportforworkflow.Becausemedicalworkflowiscom-
plex and multidimensional, efficiency of the process in fact
depends on the interplay between multiple factors in work-
flow [21]. A health professional user may be satisfied with
some aspects of the workflow involving the system while dis-
satisfied with other aspects. In a mixed method study, we
further investigated this issue in TIMED units [22]. We found
that while the CPOE system improved certain non-supportive
features of the TIMED-system, it lacked its main supportive
features.
Sociologists who study medical practice often assert that
the construction of medical practice is intertwined with the
capabilities of tools used in daily practice [23,24]. Two dif-
ferent paper-based orders in our case therefore were the
representatives of two different nursing organizational pro-
cesses. In Kardex units, some of the social structures of the
medication process incorporated into the system were sim-
ilar in the paper-based and computerized processes. Based
on the principles of ‘Adaptive Structuration Theory’ described
in our theoretical background, we suggest that the familiar-
ity of the Kardex-nurses with these structures led to more
positive perceived effects. In contrast, the implementation of
the system imposed new social structures for the workflow
of TIMED nurses resulting in less positive perceived effects.
It therefore could be the case that the TIMED-nurses, as a
consequence of a different work organization, experienced
more difficulty in appropriation of the system in the prac-
tice.
The concise and semi-structured physician orders in the
TIMED system were highly abstract representations of orders.
Our results can be understood if we consider that the transla-
tion and transcription tasks to process these orders compelled
nurses to take a more active role in the medication process.
Nurses would here assume a more authoritative position in
the medication process than their counterparts in Kardex
units. While the roles and responsibilities of nurses in Kardex
units remained intact, the implementation of the CPOE sys-
tem may have changed and challenged the position of nurses
in TIMED units. This is in accordance with previous studies
that emphasize the organizational context of clinical prac-
tice in which the implementation and application of a CPOE
system take place [25,26]. Aarts and Berg in a qualitative
study of two hospitals described how changes in the exist-
ing organizational contexts resulted in different outcomes of
the implementation of the same CPOE system [27]. In our
case, TIMED nurses judged the effectiveness of the new pro-
cess in light of the effectiveness of the paper-based process
(Fig. 4). This suggests that user perceptions can be influenced
by the extent to which a technology changes established work
patterns.
6.1.Unexpected results and unanswered questions
Unexpected results were also seen in our study. In both our
study and the study by Lee et al. [4], nurses were pleased
with improved legibility and completeness of medication
orders.However,thecomputerizedsysteminourstudydidnot
improve the clarity of the layout in medication orders, even
though a relationship between the two had been expected.
Clarity of the layout refers in general to the ability of nurses
to clearly visualize a medication order or its concepts. Some
studies have suggested that the negotiation between nurses
and physicians during medical rounds helps a nurse to under-
stand an order better [28]. Although the limited negotiation
between physicians and nurses with the implemented CPOE
system may explain our finding, another reason might be the
fact that the highly detailed medication orders are all printed
withblackink,makingallorderslookalikeandprohibitingthe
use of any of the visual cues that are available in paper-based
orders. These factors can reduce the clarity of an order and
thereby make it difficult for busy nurses to correctly under-
stand the orders. Hence, the clarity of an order entails more
than just legibility and completeness.

Page 10

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
179
Furthermore,intheory,acomputerizedorderentryprocess
should improve communication of nurses with the pharmacy
department, since many elements in the order entry process
are automated. The CPOE system examined in our study did
not achieve this: it did not appear to facilitate ordering of
non-stock items from the pharmacy. This quantitative study
provides no answers about how and why this was the case. In
a qualitative study, we explored how the organizational and
social context in which the system was implemented affected
its intended behavior [29].
Moreover, to determine a possible carry over effect, we did
a sub-analysis on the wards that implemented the system
early vs. those that implemented it late. No differences in
change in satisfaction and overall score were seen between
the wards in Kardex units that implemented the system
early vs. those that implemented it later. While differences in
change in satisfaction and overall score were seen between
wards in TIMED units that implemented the system early
vs. those that implemented it later, collinearity made it
difficult to determine whether this was due to early vs.
late implementation or due to other reasons (e.g., type of
ward).
6.2. Strengths and weaknesses of the study
User satisfaction studies often tend to focus on an IT sys-
tem’s technical characteristics [18,30]. In our study, we were
interested in evaluating the perceived effects of CPOE on dif-
ferent activities in the nursing medication process. To our
knowledge no other study has performed a before-after CPOE
implementation study of two different paper-based systems.
Whileotherstudieshaveextendedourunderstandingofnurs-
ing practice in paper-based and CPOE-based environments,
they have mostly described the CPOE environment and com-
pared it with a paper-based environment (see, for example
[11,28]). These studies did not use the same pool of personnel
in the two environments. We found that a study of simi-
lar groups of respondents before and after implementation
of a CPOE system can yield valuable insight into the true
impact of the system on workflow. Two other strengths of
our study are the similarity of respondent groups and the
absence of any relevant differences between the two phases
(besides the implementation of a CPOE system). These factors
reduce the chance that there are other explanations for our
findings.
However,ourstudyhaslimitationsthatdeservediscussion.
First, we focused on the perceived impact of the system on
nursing practice and did not measure the actual impact of
the system on the quality of care (e.g., the quality of medi-
cation orders). This was beyond the scope of this study and
should be examined more carefully in the future. Another
possible limitation is non-response bias which may have
arisen during our study. Our results therefore have to be inter-
preted cautiously. Since the participation rate in this study
was fairly good (70.6% of nurses), this would suggest that the
impact of non-response bias was limited, though a higher
rate would be more convincing for validity purposes. Third,
our findings mainly relate to the transitional phase of a CPOE
implementation project. The long term impact of a CPOE
system would require taking measurements after a longer
period of time. By that time users would have become more
acquainted with the strengths and weaknesses of the sys-
tem and thereby more capable of identifying and reporting
them.
7. Conclusion
Implementing CPOE systems reorganizes the medication
work. These systems are referred to as transformational tech-
nology and nursing practice is not an exception. The pattern
of nursing work in general and their roles and responsibilities
in particular are all affected. Yet, the perceived impact of the
system on nursing practice depends in part on the extent to
which the system changes existing work patterns. Use of AST
allowed us to explain why the outcome of transition from a
paper-based to computerized process can vary widely across
groups that are using the same information technology. How-
ever, our approach does not explain how these two groups
appropriated this technology in their practice. Observational
studies are needed to describe and explain the appropriation
process.
Our study suggests that not merely the CPOE system, the
technology itself, influences the perceptions of its users. The
size of the differences between pre- and post-implementation
work processes can also play an important role. Large dif-
ferences can in fact make it much more difficult for health
professionals to switch from a paper to a computerized sys-
tem. The strengths and limitations of a new system for
existing work organizations should therefore be carefully
considered beforehand. If a difficult transition is expected,
activities such as extra support or training sessions can be
planned to alleviate that. This insight should inform strate-
gies for change management when these systems are being
designed, implemented, and maintained.
Contributions of the authors
HS designed the questionnaire and collected data. ZN and
HP designed the study and analyzed data. ZN wrote the
early draft of the manuscript. HS was also involved in the
comments on the manuscript. WR was involved in analysis
of data and critical review and editing of the manuscript.
ZN, HP, and JA participated in the development of the con-
cept and critical review of the content. ZN revised different
versions of the manuscript. JA scientifically supervised the
paper.
Acknowledgements
The authors gratefully acknowledge the time and effort
of all head nurses and all of those who completed the
questionnaires. We also thank Enrico Coiera, Antoinette
de Bont, and Marc Berg for their insightful comments on
an earlier version of this paper. The first and the third
authors were supported by a grant from the Iranian Min-
istry of Health and Medical Education while conducting this
research.

Page 11

180
international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
Summary points
What was known before the study
• The implementation of CPOE systems changes work
practice and the role and responsibilities of care
providers.
• Nurses’ roles, responsibilities, and activities in the
medication process deserve more attention in the
design, implementation and evaluation of CPOE sys-
tems.
What the study has added to the body of knowledge
• One CPOE system may have different impact on differ-
ent work structures.
• Depending on the previous work structure, the degree
of change in work structure and in roles and respon-
sibilities of care providers after the implementation of
a CPOE system can greatly influence the system’s per-
ceived effects. The strengths and limitations of a new
system for existing work structure should therefore be
carefully considered beforehand.
• Workflow support with a CPOE system is a multi-
layered issue; hence, mere greater satisfaction with a
system may not necessarily be a reflection of greater
support for workflow.
Appendix A. Supplementary data
Supplementary data associated with this article can be found,
in the online version, at doi:10.1016/j.ijmedinf.2008.06.012.
references
[1] R. Briere (Ed.), Crossing the Quality Chasm, a New Health
System for the 21st Century, National Academy Press,
Washington, DC, 2001.
[2] P. Francois, E. Chirpaz, H. Bontemps, J. Labarere, J.L. Bosson,
J. Calop, Evaluation of prescription-writing quality in a
French university hospital, Clin. Perform. Qual. Health Care
5 (July–September (3)) (1997) 111–115.
[3] E.H. Winslow, V.A. Nestor, S.K. Davidoff, P.G. Thompson, J.C.
Borum, Legibility and completeness of physicians’
handwritten medication orders, Heart Lung 26 (March–April
(2)) (1997) 158–164.
[4] F. Lee, J.M. Teich, C.D. Spurr, D.W. Bates, Implementation of
physician order entry: user satisfaction and self-reported
usage patterns, J. Am. Med. Inform. Assoc. 3 (1) (1996)
42–55.
[5] L. Cordero, L. Kuehn, R.R. Kumar, H.S. Mekhjian, Impact of
computerized physician order entry on clinical practice in a
newborn intensive care unit, J. Perinatol. 24 (February (2))
(2004) 88–93.
[6] H.S. Mekhjian, R.R. Kumar, L. Kuehn, T.D. Bentley, P. Teater,
A. Thomas, et al., Immediate benefits realized following
implementation of physician order entry at an academic
medical center, J. Am. Med. Inform. Assoc. 9
(September–October (5)) (2002) 529–539.
[7] P.N. Gorman, M.B. Lavelle, J.S. Ash, Order creation and
communication in healthcare, Methods Inf. Med. 42 (4)
(2003) 376–384.
[8] R. Dykstra, Computerized physician order entry and
communication: reciprocal impacts, in: Proc. AMIA Symp.,
2002, pp. 230–234.
[9] J.D. Carpenter, P.N. Gorman, What’s so special about
medications: a pharmacist’s observations from the POE
study, in: Proc. AMIA Symp., 2001, pp. 95–99.
[10] J. Richards, Pandora’s box: physician order entry and nursing
work redesign, Can. J. Nurs. Leadersh. 13 (May–June (2))
(2000) 15–19.
[11] M.C. Beuscart-Zephir, S. Pelayo, F. Anceaux, J.J. Meaux, M.
Degroisse, P. Degoulet, Impact of CPOE on doctor-nurse
cooperation for the medication ordering and administration
process, Int. J. Med. Inform. 74 (August (7–8)) (2005)
629–641.
[12] R.W. Coleman, Translation and interpretation: the hidden
processes and problems revealed by computerized physician
order entry systems, J. Crit. Care 19 (December (4)) (2004)
279–282.
[13] C. Weir, V. Johnsen, D. Roscoe, A. Cribbs, The impact of
physician order entry on nursing roles, in: Proc. AMIA Annu.
Fall Symp., 1996, pp. 714–717.
[14] E. Goorman, M. Berg, Modelling nursing activities: electronic
patient records and their discontents, Nurs. Inq. 7 (March
(1)) (2000) 3–9.
[15] G. DeSanctis, M. Poole, Capturing the complexity in
advanced technology use: adaptive structuration theory,
Organ. Sci. 5 (2) (1994) 121–147.
[16] A. Giddens, The Constitution of Society: Outline of the
Theory of Structuration, University of California Press, 1986.
[17] M.D. Kalmeijer, W. Holtzer, R. van Dongen, H.J. Guchelaar,
Implementation of a computerized physician medication
order entry system at the Academic Medical Centre in
Amsterdam, Pharm. World Sci. 25 (June (3)) (2003) 88–93.
[18] H.J. Murff, J. Kannry, Physician satisfaction with two order
entry systems, J. Am. Med. Inform. Assoc. 8
(September–October (5)) (2001) 499–509.
[19] M.J. Ball, C. Weaver, P.A. Abbott, Enabling technologies
promise to revitalize the role of nursing in an era of patient
safety, Int. J. Med. Inform. 69 (January (1)) (2003) 29–38.
[20] M.L. Popernack, A critical change in a day in the life of
intensive care nurses: rising to the e-challenge of an
integrated clinical information system, Crit. Care Nurs. Q. 29
(October–December (4)) (2006) 362–375.
[21] M. Berg, The search for synergy: interrelating medical work
and patient care information systems, Methods Inf. Med. 42
(4) (2003) 337–344.
[22] H. Pirnejad, et al., Impact of a computerized Physician order
entry system on nurse-physician collaboration in the
medication process, Int. J. Med. Inform. (2008),
doi:10.1016/j.ijmedinf.2008.04.001.
[23] M. Berg, G. Bowker, The multiple bodies of the medical
record: towards a sociology of an artifact, Sociol. Q. 38 (1997)
511–535.
[24] M. Berg, Practices of reading and writing: the constitutive
role of the patient record in medical work, Sociol. Health
Illness 18 (4) (1996) 499–524.
[25] T.A. Massaro, Introducing physician order entry at a major
academic medical center: I. Impact on organizational culture
and behavior, Acad. Med. 68 (January (1)) (1993) 20–25.
[26] B. Kaplan, Addressing organizational issues into the
evaluation of medical systems, J. Am. Med. Inform. Assoc. 4
(2) (1997) 94–101.
[27] J. Aarts, M. Berg, Same systems, different
outcomes—comparing the implementation of computerized
physician order entry in two Dutch hospitals, Methods Inf.
Med. 45 (1) (2006) 53–61.

Page 12

international journal of medical informatics 7 8 ( 2 0 0 9 ) 170–181
181
[28] M.C. Beuscart-Zephir, S. Pelayo, F. Anceaux, D. Maxwell, S.
Guerlinger, Cognitive analysis of physicians and nurses
cooperation in the medication ordering and administration
process, Int. J. Med. Inform. 76S (June) (2007) S65–S77.
[29] Z. Niazkhani, H. Pirnejad, A. de Bont, J. Aarts, Evaluating
inter-professional work support by a computerized
physician order entry system, Stud. Health Technol. Inform.
136 (2008) 321–326.
[30] M. Weiner, T. Gress, D.R. Thiemann, M. Jenckes, S.L. Reel, S.F.
Mandell, et al., Contrasting views of physicians and nurses
about an inpatient computer-based provider order-entry
system, J. Am. Med. Inform. Assoc. 6 (3) (1999) 234–244.