A Cognitive Task Analysis of Information Management Strategies
in a Computerized Provider Order Entry Environment
CHARLENE R. WEIR, PHD, RN, JONATHAN J. R. NEBEKER, MD, BRET L. HICKEN, PHD, MSPH,
REBECCA CAMPO, MS, FRANK DREWS, PHD, BETH LEBAR, RN, MSN
A b s t r a c t
computerized decision support dramatically changes the information environment of the practicing clinician. Prior
work patterns based on paper, verbal exchange, and manual methods are replaced with automated, computerized,
and potentially less flexible systems. The objective of this study is to explore the information management
strategies that clinicians use in the process of adapting to a CPOE system using cognitive task analysis techniques.
Design: Observation and semi-structured interviews were conducted with 88 primary-care clinicians at 10
Veterans Administration Medical Centers.
Measurements: Interviews were taped, transcribed, and extensively analyzed to identify key information
management goals, strategies, and tasks. Tasks were aggregated into groups, common components across tasks
were clarified, and underlying goals and strategies identified.
Results: Nearly half of the identified tasks were not fully supported by the available technology. Six core
components of tasks were identified. Four meta-cognitive information management goals emerged: 1) Relevance
Screening; 2) Ensuring Accuracy; 3) Minimizing memory load; and 4) Negotiating Responsibility. Strategies used
to support these goals are presented.
Conclusion: Users develop a wide array of information management strategies that allow them to successfully
adapt to new technology. Supporting the ability of users to develop adaptive strategies to support meta-cognitive
goals is a key component of a successful system.
? J Am Med Inform Assoc. 2007;14:65–75. DOI 10.1197/jamia.M2231.
Objective: Computerized Provider Order Entry (CPOE) with electronic documentation, and
The electronic medical record with Computerized Provider
Order Entry (CPOE) has instituted a new era of electronic
information processing in clinical care.1,2–4Electronic docu-
mentation, electronic order entry, and decision support
dramatically change the information environment of the
practicing clinician. Prior work patterns based on paper,
verbal exchange, and manual methods are replaced with
computerized systems that are potentially less flexible, be-
cause they require prescribed methods of data entry and
Successful implementation of CPOE requires close attention
to the local details of information exchange and workflow
processes. Understanding workflow is difficult as many
processes are often “hidden” because clinicians (and even
the information technology administration) are unaware of
subtle behavioral changes that occur as clinicians adapt to
the environmental constraints. These adaptations consist of
both the change in behavior of clinicians and the ongoing
changes in the technology as the system of care evolves
together. As Rasmussen notes (1994) “. . . an effective orga-
nization of work depends on self-organizing and adaptive
‘mechanisms’ which enable the system to change its prop-
erties in order to maintain a match with current needs . . . .”
In the process of adapting to the system, users develop
multiple strategies to seek, sort, categorize, communicate,
and document information.5–8These strategies reflect the
resolution between basic information processing needs of
the user and the functional characteristics of the technolog-
ical system. Often, new strategies eventually become seam-
lessly embedded into work processes as implementation
The purpose of this study is to clarify how users have
adapted to a well-established CPOE system. We conducted
Affiliations of the authors: Geriatric Research, Education, and
Clinical Center (GRECC) and Salt Lake Informatics, Decision En-
hancement and Surveillance (IDEAS) Center, George E. Wahlen
Department of Veterans Affairs Medical Center (CRW, BLH, JN),
Salt Lake City, UT; Departments of Bio-Medical Informatics (CRW),
Internal Medicine (JN), Psychology (RC, FD), College of Nursing
(BL), University of Utah, Salt Lake City, UT.
This study was supported by VA HSR&D # MRC 03-237, TRP
02-147-2 and RCD 02-176-2.
The authors acknowledge the help and support of Hank Rappaport,
MD at the Office of Information, Veterans Health Administration,
Department of Veterans Affairs. Dr. Rappaport provided important
insight into the VA system development.
Correspondence and reprints: Charlene Weir, PhD, Geriatric Re-
search, Education, and Clinical Center (182), George E. Wahlen
Department of Veterans Affairs Medical Center, 500 Foothill Dr.,
Salt Lake City, UT 84148; Tel: (801) 582-1565; Fax: (801) 584-5640;
Received for review: 07/31/06; accepted for publication: 10/13/06.
Journal of the American Medical Informatics AssociationVolume 14Number 1Jan / Feb 2007
a Cognitive Task and Activity Analysis of the information
management behavior of users across several primary care
settings.10–13Cognitive Task Analysis is a qualitative
method of analysis derived from human factors and cogni-
tive psychology. The purpose is to understand individuals’
mental representations of activities and work processes they
are engaged in and how those perceptions relate to behavior.
VHA’s Computerized Patient Record System
The Veterans Administration (VA) has achieved widespread
adoption of their Computerized Patient Record System
(CPRS) since its dissemination in 1999. CPRS contains func-
tionality for CPOE, electronic note entry, clinical decision
support, electronic consult requests, results reporting, and
bar code medication administration. The VA system is a full
Patient Information System. Both CPOE and electronic note
entry are mandated and almost all of the medical record is
now electronic across the 128 medical centers in the VA. The
VA is also an excellent laboratory to study the impact of
CPOE. Because CPRS is adopted system-wide, the organi-
zational and geographical factors that contribute to success-
ful adoption of a CPOE system can be examined. In addi-
tion, because CPRS is leveraged extensively to improve the
quality of care, the usefulness of technology in enhancing
adoption of guidelines, clinical reminders, and quality im-
provement activities can be evaluated.
In early 2003 the VHA released the first new extension to
CPRS called Care Management (CM). CM extended the
functionality of CPRS by allowing a multiple-patient view,
the ability to create generic “tasks,” and a population-based
query tool. Although this functionality was much desired by
providers, adoption has been very low. To clarify the issues
surrounding the low adoption rate of CM, we conducted a
Cognitive Task Analysis of the information management
activities across multiple primary care settings. The goal was
to explore current patterns of computer use as well as to
identify some general information management principles
that might guide future development. The results of the full
evaluation are reported elsewhere (Nebeker, Weir, Hicken,
and Rappaport, under review).14
CPOE and Errors
Although information systems can significantly improve
care, they may also have unintended negative results.15–18
When the amount of information increases and the display
format changes, individuals must develop new and different
strategies to manage the basic cognitive functions of mem-
ory, attention, planning, and learning. These strategies may
or may not improve practice and potentially threaten the
success of the implementation. Although there is no direct
evidence with CPOE systems, research in cognitive psychol-
ogy has demonstrated that information overload causes
significant effects on individual information processing. For
example, excessive information may cause individuals to
underestimate the rate of events19and to be overconfident.20
Restrictive and limited displays of different kinds of data
may result in co-variation errors (e.g., items on a list might
be unconsciously perceived as equally probable), inappro-
priate risk assessments, or priming of inappropriate behav-
ior scripts (e.g., focus on curative perspective when pallia-
tive care is needed).21–23The impact on decision making
may be subtle and may have profound consequences for the
quality of care.
Other studies have found significant impact of CPOE sys-
tems on collaborative and communication processes. Many
CPOE systems are designed assuming a linear process for
writing orders (e.g., physician writes orders and allied staff
implement them). However, often the process is not linear
but rather circular, recursive, redundant, and involves ex-
tensive synchronous communication processes. For exam-
ple, a nurse may write an order for a fluid bolus for a
post-operative patient with low urine output in the middle
of the night because she knows that the physician will cover
her the next day. The pharmacist may discontinue an order
and rewrite the new one if the order is incorrect or not in
stock in the form ordered. The physician will be notified
later. A nurse may refuse to administer what the physician
ordered until it is clarified. Eliminating these processes of
communication or making them more difficult may result in
Cognitive Task Analysis
Cognitive task analysis (CTA) is defined as “the extension of
traditional (behavioral) task analysis techniques to yield
information about the knowledge, thought processes, and
goal structures that underlie observable task performance”
(p. 3).27The aims of a cognitive task analysis are to identify
the concepts, contextual cues, goals, and strategies that
constitute the mental model of individuals within a domain
of action.5,28–30The methods used span a variety of tech-
niques including observation of physical actions, semi-
structured interviews to determine the corresponding cog-
nitive activity, formal cognitive mapping, and observation
using a “talk-aloud” technique (where individuals voice out
loud what they are thinking).31Most authors promote using
multiple techniques and adapting the techniques to the
context. All promote some form of goal analysis.28In this
study, we used a hierarchical goal analysis approach, adapt-
ing procedures from two theoretical perspectives, Action
Identification Theory32and general CTA models.27Similar
analyses have been conducted in relation to clinical practice
guidelines33and the ordering process.34
Cognitive Task Representations
Cognitive theories stipulate that cognition (thinking, de-
cision making and subsequent behavior) arise out of
activation of associative schemas of neural connections in
memory that consist of behavioral sequences, semantic
content, values, and emotional content.35,36The more
well-learned the behavior, the stronger the associative
links. One particular type of representation is action
representations, which have several unique characteristics.
First, they are primarily goal-based and consequently,
tend to stay “activated” until the resolution criteria have
been reached. In contrast, “activated” memory or knowl-
edge representations decay linearly.37,38Action represen-
tations are also thought to be organized hierarchically,
ranging from basic concrete procedural or mechanical rep-
resentations of bodily movements to values and goals at the
more abstract level in a many-to-many relationship.32,36,39
In this study, we adapted techniques developed by Valla-
cher and Wegner32to characterize the hierarchical structure
of action representations. In this method, individuals are
WEIR et al., Information Management Strategies
asked to respond to the following open-ended questions,
“What are you doing?” “You are doing XX in order to do
what?” “You are accomplishing XX, by doing what?” Our
focus was on activities relating to information management
in the primary care setting, in contrast to clinical medical
tasks, although the two are highly inter-related. Information
management tasks focus on the searching, handling, and
manipulating of information, including diverse areas such
as scheduling, ordering, or communicating with patients
and include all mediums, e.g., verbal, electronic notes, and
Site recruitment consisted of a two-stage process. VA insti-
tutions were selected based on institutional readiness, fol-
lowed by geographic and size criterion in order to enhance
representativeness. To assess readiness, Care Management
implementation plans were solicited from all 135 VHA sites.
The authors (CW, BH, and JN) rated each site’s readiness
using a structured coding scheme that included points to be
given for plan specificity, the presence of clinical champions,
early marketing strategies and a training protocol. Scores
could range from 1–10. Three authors (CW, BH, and JN)
conducted the ratings independently and discussed differ-
ences until a consensus was reached. Sites with average
scores of 6 or above were then approached to participate in
the study and the final choice of enrollment balanced
geographical and size representativeness. A total of 13 sites
were included in the overall study and 10 sites were
involved in the interview process. Two sites had been used
for early piloting and another site was unable to get timely
institutional review board (IRB) approval. All participating
sites had IRB approval for the study.
Because CTA requires significant domain-specific knowl-
edge and familiarity with the motivations and the normative
situational awareness of participants, the degree of content
expertise held by the interviewers can be essential to pro-
viding an accurate portrayal of tasks. The three interviewers
involved in this study are very familiar with the VA system.
JN is a physician in active practice at the VA. BH is a clinical
psychologist who at the time of the study carried a small
clinical load at the VA. CW is a nurse and social psychologist
who served as Project Manager for implementing CPRS at
the Salt Lake City VA.
At each site, a primary care clinic was randomly selected.
Primary care teams consisted of providers, Nurse Practition-
ers, Physician Assistants, and/or staff physicians (we did
not include residents). Also included were RNs, LPNs, and
Nursing Assistants as well as social workers, dieticians,
pharmacists, and clerks. A total of 88 interviews were
conducted. Overall, 14 nurses, 53 ordering providers, 8
pharmacists, 2 dieticians, 3 clerks, and 8 social workers were
A semi-structured interview was used that focused on how,
when, what, and where information was collected in pri-
mary care, with a focus on goals and control strategies
people used to manage their environment. Three investiga-
tors (CW, BH, and JN) extensively piloted the survey
together in order to standardize the process. The interview
included general queries regarding information manage-
ment activities and had an overall chronological structure
starting with questions regarding how they would begin the
day, prepare for a visit, follow-up activities that were
required, and clinic-related activities. Appendix A contains
the interview instrument.
The interviews were conducted at the clinic during normal
clinic hours. The interviews followed the general format, but
the order of topics varied to allow for respondents’ idiosyn-
cratic responses. To facilitate the goal-based approach, in-
terviewees were asked probe questions, specifically “you
did XX action in order to accomplish what purpose?” or
“you accomplished XX goal by doing what action?” For
example, interviewees would be asked, “When you come to
work in the morning, what are the actions you do at that
time to prepare for your day?” When a respondent replied
with “I print a list of my appointments from VistA,” the
interviewer would respond with “and you printed that list
in order to . . .”. Similarly, if it were not clear, interviewer
would ask “you printed that list by doing . . . .?” This
methodology is used by Vallacher and Wegner to map
cognitive action structures32and is consistent with the
general principles of Cognitive Task Analysis.27The inter-
view would proceed in that manner covering the day
chronologically, including actions associated with getting
ready to see a patient, communicating with the staff, lunch
breaks, etc. Interviewees were allowed to talk at length in an
open-ended manner, with some prompting when needed.
Observations were embedded into the interview process
and were quite easy to do because the researchers were
visiting the clinic during patient care hours. In addition to
the verbal content collected during the interview, clinicians
were asked at times to demonstrate how they actually did
many of the activities they described. For example, if they
said “I processed my alerts,” they were asked to actually
process their alerts while we observed. About a third of the
interviewees allowed the researcher to sit in during a patient
visit. This aspect of the interview was essential, because
some clinicians had trouble reporting accurately their be-
havior in enough detail. Clinicians were asked to “think
aloud” when doing these actions whenever possible.31,40
Finally, every time a clinician reported using some kind of
artifact (e.g., paper, tools, or forms) they were asked to show
it to the interviewer. For example, if they printed the daily
schedule that lists patient’s appointments for that day and
took ongoing notes on that list, we examined the list. Often,
this would provide an opportunity for providers to describe
in detail their strategies for organizing their own work (e.g.,
unique symbols for tracking progress, important content, or
relevant “stories” about an event).
In summary, the combination of structured interviews,
observations, and examination of artifacts provided a robust
view of how CPOE was being used. The goal of this form of
data collection was to maximize the emergence of goals,
perceptions, beliefs, strategies, and concerns. All interviews
Journal of the American Medical Informatics Association Volume 14Number 1Jan / Feb 2007
were taped in order to facilitate data collection and tran-
scribed later for analysis. In addition, field notes were typed,
collated, and included in the analysis. Observation and
interview time totaled approximately 110 hours.
Qualitative data analysis involves using inductive processes
to bring order, structure, and meaning to a large amount of
data. Analytic procedures began with multiple readings of
the interviews, development of coding schemes to facilitate
abstraction, and an iterative process of validating the codes
by comparing independent reviewers, discussion, and revi-
sion. Coding was expanded through processes of aggrega-
tion across levels of abstraction and by analysis across
persons and events.41
For this study, the analysis process was conducted with two
of the investigators (CW and BH) plus two graduate stu-
dents (Nursing Informatics and Psychology PhD candidates)
meeting weekly for a total of 60 hours over 4 months of time.
Initially the meetings focused on identifying themes, issues,
and patterns. Later, the team worked on establishing and
revising coding schemes in order to further develop the
categories. To accomplish that task, each member of the
team independently highlighted the transcribed text and
then discussed differences and then revised the codes using
NVivo, version 2.0. The process was iterative with descrip-
tions of boundary conditions and prototypes continually
being revised until agreement levels reached at least a
moderate level (kappas ranged from of 0.48 to 0.64).42This
level of agreement provided some assurance that the cate-
gories could be reliably described. The remaining categori-
zation and coding of text was done by consensus. The group
met for over 60 hours and the subsequent coding of all of the
text took another 120 hours across investigators.
The results will be reported at three levels of analysis. The
first level presents the basic tasks that clinicians reported
actually doing. These include the exact stated items, such as
“Identify when the patient was last here” or “Determine the
reason for the visit.” The second level is a common compo-
nents analysis where we extracted common structures
across tasks. The third analysis is at the goal level where
generalized information management strategies and over-
arching information management goals are identified.
Although we grouped similar tasks together to form cate-
gories, the basic wording was kept as close as possible to the
original clinician’s reports as possible. A list of representa-
tive reported tasks (not the complete list) is displayed in
Table 1. The list is organized chronologically starting with
activities that occur at the beginning of the day, followed by
tasks associated with seeing the patient, those involving
follow-up or tracking over time as well as general ongoing
time management actions. As a result, some tasks appear
twice, such as “determine what medications the patient is
on” and “recent medical activity.” This action occurs both
prior to a visit and during a visit.
In order to identify the relationship between tasks and
available computerized functionality, we coded each task
along six available mediums. CPRS refers to the patient
information system display normally used by providers. CM
refers to Care Management, a new multi-patient view. VistA
refers to the screen that allows entrance to all of the VA
computerized systems, and includes not only clinical pro-
grams, such as lab, radiology, and others, but also employee
time tracking, provider education, and patient scheduling.
Paper referred to either lists manually written, spreadsheets
created by providers and printed, or lists printed out from
VistA. The Phone column is self-explanatory. The Person
category not only includes in-person conversations, but also
e-mails. The cells were color coded to reflect coverage of
tasks. Grey in the cell indicates the task is partially covered
by the medium. Black indicates full coverage and white
indicates little or no coverage.
When scanning across the table, it appears that most tasks
have some technical support, either from CPRS, VistA, CM,
or all three. However, very few were completely supported.
Even something as straightforward as determining the pa-
tient’s actual medication profile was frustrating for clini-
cians because not all drugs are listed in the VA profile
(especially non-VA drugs) and the actual dosage may have
been changed, but a new order was not really entered. And,
some tasks required manual extraction and storage on
paper, such as tracking a patient’s preparation for outpatient
surgery. The data could be extracted from CPRS or VistA,
but the overview display of the information required that it
be taken down on paper. Having to move from medium to
medium was one of the most salient complaints regarding
the information system heard during the interviews. Many
of the tasks were done differently from site to site and those
were usually the rows with all gray cells (indicating some
support with no single medium providing full support).
Common Components Analysis
Common components across tasks were abstracted from the
list of tasks in Table 1 using an iterative process of abstrac-
tion. Table 2 presents the common abstracted structure.
First, all tasks require some cue to begin. Most of the time,
the cue is the well-learned ordering sequence of procedures.
For example, the cue to schedule an appointment often
arises from the normal routine care of the patient. The
patient finishes their appointment and knows to go to the
check out desk and the appearance of the patient in front of
the clerk asking for a next appointment cues the clerk.
However, the cue to schedule an appointment could also
come from reading an electronic note from another provider
suggesting the need for a consult or the nurse’s putting a
note on the provider’s door reporting on a phone call. In any
scenario, the simple task of scheduling an appointment
becomes a collaborative effort; the physician has to indicate
that one is needed, the nurse may have to identify for the
clerk the appropriate time, the patient has to know about the
need for an appointment and the clerk has to actually enter
the time into the computer. At each step, the previous action
provides the cue for the next step and often, different
individuals will do each component. Interestingly, the spe-
cific form that a sequence of cues would take was unique
(but consistent) for each site and embedded into local
workflow. However, the exact work processes used varied
The second common component is status. Tasks can be
pending, in progress, ready to be communicated, or resolved.
WEIR et al., Information Management Strategies
Status is an aspect of tasks that needs to be tracked contin-
uously. Everyone sharing in a task requires this information.
Tracking the status of tasks is a regular and repetitive
component of workflow. The lack of a simple way to track
the status of ongoing tasks was a common complaint and
often would require substantial searching on the part of the
provider. Resolved is a status that a task can take and because
every task has a unique and complex manner of completion,
the resolved status could be multi-faceted. Examples of
types of resolutions include writing in the chart, communi-
cating to another provider, communicating to the patient,
and finishing the task itself. Commonly, the task has to be
“crossed-off” from personal overview planning lists (which
nearly every provider has) in addition to the other resolution
Third, all tasks have a time component, including when
they are due, length of time to complete the task (a test
can be ordered, but scheduling requires a 2-week wait) or
in the case of repetitive tasks (e.g., ordering monthly
narcotics), the time interval. Often everyone in the clinic
knows the time component implicitly because it is com-
monly held clinical knowledge (e.g., the time from a
Coumadin change to the next INR test). Time information
is an integral part of interpreting information and is
always sought after if it is not available. When the
information is not so clearly known, such as how long it
has been since a patient has been called, or how many
times a test has been repeated, then providers often need
to resort to verbal communication.
Fourth, every task has an owner or owners. Identifying task
ownership was a common information need. Unfortunately,
in the current information system, identifying who owns the
Table 1 y Task Coverage by Medium
TaskDescriptions CPRSCM VISTAPAPER PHONEPERSON
After or between visits
Determine who’s working today
Identify patients who have appts. today
follow-up appointments to be made
why patient is here
patient’s actual current medication
patient’s current problems
patient’s primary care provider
all other providers
patient’s % service connected status
results from recent procedures
who has seen patient today
patient compliance with meds
Clinical Reminders specific to patient
adverse drug events
availability of community resources
why patient is on drug
patient about illness, results
procedure, x-ray or consult
patient’s response to treatment
where patient is in protocol or guideline
of patient’s illness trajectory
patient’s lab results over time with meds
patient’s care plan
to follow-up on procedure or lab results
phone and e-mail messages from patient
messages and notifications
plan to another provider
that other provider received information
that patient completed lab or procedure
Inform patient of lab or procedure results
regarding who is supervising students
Black ? full coverage; Grey ? partial coverage; White ? little or no coverage
Table 2 y Common Task Components
CueingA stimulus or embedded procedure that
initiates the beginning of a task, including
time passed, lab result, or another provider.
The current characteristic of the task, whether
it is pending, resolved, etc.
Date due, latency until it begins,
contingencies (e.g., on lab), or recurrent.
Results, responsibility, or status
Person responsible for ensuring completion.
The relationship of task to other tasks, e.g.,
Journal of the American Medical Informatics Association Volume 14Number 1 Jan / Feb 2007
task is difficult, except when it is a formal order (which is
only a small part of the communicated information). Not
only does every task have an owner, but also as tasks evolve,
sometimes the owner would change. Often, tasks involve
significant coordination of activities between individuals.
For example, the different tasks created as part of the
process of getting an outpatient ready for surgery requires
the primary care provider to provide a history and physical,
nurses must schedule the surgery based on patient’s needs,
clerks must determine the schedule and order protocol labs
and other nurses must ensure that education is done and
consents signed. Often, the person with whom the respon-
sibility is shared is known by name, other times only by role.
Determining the name of the person in a designated role is
information that is currently not available on the computer
and often requires substantial time to determine.
Most importantly, nearly all tasks require some form of
communication to another person in the workplace. Usually
this communication goes beyond simply documenting it in
the notes. The material to be communicated varies, but may
include one or more of the following: content, state, the
responsible individual, and time. Tasks vary in the degree to
which communication is an inherent component of the
process, but often in order to get something done, another
person has to be contacted. For many tasks, communication
is central, getting an appointment means telling the patient,
the clerk, and maybe the nurse when, why, and maybe even
how to make the appointment. The triage nurse takes every
patient interaction and often must contact 3–4 individuals
before she can go on to the next patient’s call.
Some forms of communication are standardized in medical
practice, such as orders and consults. Many others are less
structured but are so embedded into workflow that clini-
cians are usually not aware of them. The variety, complexity,
and idiosyncratic nature of local procedures for sharing
information among providers were quite surprising. For
example, the physician may not formally request that the
intake nurse ask questions about medications or symptoms,
but expects that those questions are asked and infers that if
he or she does not get notified, there were no problems.
When a patient calls in, the nurse answering the call makes
a decision regarding the severity of the problem, which
provider or providers should be told and when. Local
procedures determine whether a decision is made by proto-
col, whether the physician is phoned, or whether an e-mail
is sent. In some clinics, the nurse conducts an intake inter-
view, in others a clerk asks the patients an initial set of
questions and in still others, no initial interview takes place.
Results from the interview may be written in a note,
handwritten and taped to the door, written on paper and
given to the patient, or simply communicated verbally.
Sometimes several of these forms would be taking place at
In addition, every communication requires feedback and
confirmation. Staff would go to great lengths with compli-
cated work-arounds to ensure that their communications
were received. These work-arounds ranged from requiring
initials on a paper being physically passed from provider to
provider to assigning a specific provider as a co-signer on an
electronic note (meaning that an alert would be sent to the
receiver notifying him or her of an incoming communication
Finally, most tasks are linked to other tasks. Specifying the
links between tasks as well as role and time dependencies
prior to the initiation of the task is part of planning care and
an intricate component of the implementation of sequences
of care, such as guidelines. Because the links must be
explicit, observable, and traceable, they are often recorded
on paper, such as personal lists of patients requiring follow-
up, formal tracking sheets for Coumadin patients that are
posted, or lists of patients who need narcotic renewals.
Progress notes are rarely used because they cannot be
updated and cannot be linked. Neither CPRS nor CM
support detailed specification of task linkage, nor is there a
common “work board” to which all relevant providers
Information Management Goals
Four general categories (Table 3) of information manage-
ment goals emerged from the analysis. The higher-level
goals are generalized patterns of strategies that resulted
from extensive analysis of the transcripts. They are complex
behavioral patterns that bridge the human, work processes,
and the computer. Although similar goals are present across
individuals, the exact strategy(ies) may differ significantly.
At times these strategies are episodic and observable, but
more often they are implicit, that is they are completely
embedded into the work processes. Often, providers would
not be aware of them and only extensive questioning would
reveal their presence. In most cases, these high-level infor-
mation-processing goals were only apparent to the investi-
gators after intensive reviews of the transcripts.
The first goal, Relevance Screening included strategies to
narrow the field of attention by setting up mechanisms to
screen, sort, and prioritize information. These kinds of
strategies were the most numerous. For example, providers
would report having the clerks add the reason for the visit
into the scheduling package so that they would not have to
search the notes looking for a likely reason for the upcoming
visit during preparation time. Every day a clinic list would
be printed for all staff and the list contained the reason. Or,
Table 3 y Information Management Goals
Relevance screening Reducing the amount of information
to sort through, highlighting
relevant data, making relevant
Increasing the size and scope of
presented information in order to
not miss anything, setting up
redundant processes, correlating
items, or checking multiple
Tracking the status of tasks, setting
up automatic procedures that
serve as reminders, setting up
structured division of labor or
setting up lists of tasks.
Giving and receiving orders,
consults, sharing information, and
WEIR et al., Information Management Strategies
providers might customize their notification screen so that
only the highest priority alerts would be visible on the
notification window, thereby removing minimizing sorting
effort. Selectively screening notes based on knowing the
provider (Dr. Smith is known to be accurate and thorough)
or using only one’s own notes was very common. With over
100 notes displayed in chronological order, being able to
read only a few notes at a time is an important time saver.
Since electronic notes can be considerably longer than writ-
ten notes, taking the time to scan through all of the notes can
become almost impossible. Of course, recent ER notes would
be read as well as recent results from selected consults.
Because of the amount and density of the information, most
providers would have to resort to personally writing on
paper important highlights from the patient history. They
would avoid large categories of notes if they believed that
the amount of material presented would not contain suffi-
cient relevant information given the effort. For example,
most providers reported avoiding notes that used templates
altogether (which included many nursing notes). Other
times, the strategy would exist at the whole clinic level, such
as having all alerts sent to a covering or triage nurse. The
nurse assigned to the task would then sift through the alerts
and communicate only the most acute or important items.
The overarching goal with this strategy is to minimize the
information overload that accompanies the use of an elec-
The next category, called Ensuring Accuracy seemed almost
the opposite strategy and included actions to increase the
available information. The underlying goal of this group of
strategies appeared to be to ensure accuracy and complete-
ness. Many providers expressed significant distrust in the
accuracy of the system and would try to build in redun-
dancy. Some providers would have the results of all of their
orders alerted back to them to avoid missing important
information or have clinic nurses review daily labs and alert
them of patient problems. Or, they might copy the entire
results of lab panels, procedures, or x-rays into open
progress notes as a reference when they are talking to the
patient and then delete the ones not used for the final
completion of their notes in order to ensure that they did not
miss anything. Some providers would give out their per-
sonal number to make sure that the patient could get in
contact with them directly or would set up clinic systems to
call every patient following a visit. Calling the specialist
after making a referral was often done to make sure that
they understood the reason for the consult. The simplest
form of this group of strategies is the ubiquitous scanning of
all sources of information, ranging from the numerous
electronic notes, VISTA, e-mail, and paper notes. Of course,
there were significant individual differences between pro-
viders. The overarching goal associated with these sets of
strategies is to maximize certainty and confidence in their
retrieval of information.
The third category, Minimizing Memory Load, included
strategies to set up personal reminder systems or to keep
track of required tasks over time. A large variety of strate-
gies were used for this purpose, such as a provider’s
creating an unsigned addendum or progress note as a
reminder to complete a task (providers receive alerts about
unsigned notes). Another strategy in this category was to
use the VistA e-mail to send oneself a message on a specific
date. Or, a fake clinic might be created filled with appoint-
ments of patients requiring follow-up (a clinic list can then
be scanned and reviewed by everyone and comes up on a
pre-determined day). The patients listed in one fake clinic
were those needing monthly refill of narcotics for a single
provider, thereby allowing the names to come up on his
appointment list for that clinic on that day. Almost all
providers kept a paper “to do” list for the day, marking off
tasks as they were finished and adding others as they came
up. Paper calendars would be used because they could be
posted and serve as highly salient visual cues. Finally, every
clinic had numerous visual aids that would cue staff as to
whom the patient had seen, who would be seeing the patient
next, and what was left to do for that patient’s appointment.
These reminders were largely in the form of check-off lists,
patient folders located in different places, and names written
and erased on boards visible to all.
The fourth category, Negotiating Responsibility, included a
variety of processes to assign tasks or negotiate hand-offs.
There are basically two kinds of hand-offs, within roles and
between roles. Nurses and physicians generally divide up
the workload according to standard roles, although there is
a great deal of gray area between the two. Nurses would
write orders and conduct procedures without orders if that
was standard clinic practice. No clinic was alike in terms of
the extent to which protocols were used. Other providers
(such as physical therapy, social work, or dieticians) would
be assigned the patient using a variety of mechanisms.
Sometimes, ordering providers would simply write that
they wanted the social worker to see the patient in a
progress note, expecting that the social worker would read it
(as a member of the team). Other times, the assignment
would be done through e-mails, creation of a virtual clinic
that was attached to that provider, or more formally through
the CPRS consult package. The diversity in procedures
between clinics was notable. The need for administration to
track workload was an important consideration in making
Hand-offs within roles occurred either at change of shift
(usually nurses) or between providers when one went on
vacation or residents rotated off the clinic. These issues were
dealt with differently at every site, but often required
significant redirecting of alerts, re-organizing the team struc-
ture in the computer, and canceling or moving appoint-
ments. This process is quite complicated because most
providers work in teams and so sometimes the whole team
needs to be alerted. The management of teams is an older
functionality of VISTA and coordinates the alerting and
notification structure of CPRS. Interestingly, it became clear
in our interviews that the mechanisms of team management
(and how things were set up locally) were often only known
by a very small number of individuals at each site, if at all.
In addition, these procedures differed substantially across
sites making it quite difficult for developers of new pro-
grams (e.g., Care Management) to understand standard
practice in the field. Some sites had an individual in the
clinic assigned to “manage” team lists. This person would
ensure that every provider was on the correct list and when
clinicians rotated in and out, the lists were updated. At other
sites, the clinicians were taught how to put themselves on
Journal of the American Medical Informatics AssociationVolume 14Number 1 Jan / Feb 2007
and off lists and were expected to manage it themselves.
And, in other sites, the computer office alone provided list
In summary, providers were observed to be engaging in a
wide variety of strategies that allowed them to effectively
adapt to the computerized information environment. The
overall goals of minimizing cognitive load, enhancing accu-
racy, remembering important information, and negotiating
responsibility were noted. The specific strategies used to
meet these goals varied significantly between individuals
within a clinic and also substantially across clinics.
Findings from this CTA analysis extend our understanding
of user’s information management strategies in a mature
CPOE primary care setting. All ten sites had been fully
committed to CPOE for over four years and practice patterns
had evolved and developed around the system. As a result,
both individuals and the system underwent a mutual trans-
formation. This evolutionary aspect of technical systems, an
issue that has been discussed by several authors associated
with the sociocultural approach was clearly in evidence in
One of the most important findings from this work is the
analysis of information management goals, which deepens our
general understanding of user’s interactions with informa-
tion systems. The strategies identified through this analysis
highlight fundamental cognitive activities relating to the
management of information in a clinical system. These
strategies are congruent with the concepts of “ecological
rationality” espoused by Simon, Gigerenzer, and others.7,43
As Girgerenzer (2000) notes, “Ecological rationality refers to
the study of how cognitive strategies exploit the represen-
tation and structure of information in the environment to
make reasonable judgments and decisions” (p. 57).7Any
computerized information system needs to be designed to
support strategies that allow for “fast and frugal” decision-
making.23,44These strategies are often used by decision-
makers in the real world. They include multiple methods for
rapidly identifying the minimally sufficient data to support
a decision,23attempts to efficiently balance accuracy versus
speed,45and creating personal information displays that
support recognition-primed actions.8,46,47Even sophisti-
cated CPOE systems may not support these goals as dis-
cussed by Horsky et al. (2003).34Designing systems to
support these activities are essential, as no amount of
perfection in design would eliminate the need for flexibility
in a complex medical environment. The specific “epistemo-
logical goals” at play at any one time vary across individuals
and even minute by minute within the same individual,
suggesting that efforts to “standardize” processes might be
In addition to the goal analysis a few points are especially
important to note. First, an examination of the list of
common tasks identified in these primary care clinics re-
vealed that most were not fully supported by any single
available technology, although most were covered by at
least one. Several tasks, however, were not covered at all.
Although it was possible to use a variety of communication
tools, none were integrated into CPRS. The tracking, corre-
lating, and overview tasks commonly associated with plan-
ning, forecasting, and complex decision-making were not
Another important finding is that for nearly all tasks, some
form of communication with other providers was necessary.
And, although documenting in the electronic progress notes
was important, other actions were usually required to draw
the attention of relevant other staff, to negotiate mutual
responsibility, and to confirm information exchange. These
“attention-getting” activities were either embedded into the
work processes (social workers knew to read every progress
note by the physician) or involved shared check-off lists and
even verbal interactions. The complexity and pervasiveness
of communication actions become visible in a computerized
environment.2,51–53And when they are not well addressed,
they are often a source of error.15,54It was beyond the scope
of this study to analyze communication patterns, but this
would be an important area for future investigations.
The common components analysis revealed that it is possible to
discern a common structure across tasks. The core compo-
nents of each task were found to include cueing, communi-
cation, time, state, ownership, and linkage. Cueing and some
aspects of time were well supported in the information
system, but communication, state, and linkage were partic-
ularly not supported. The result was that users had to
interact with multiple sources of data, sign-ons, and formats
in order to get things done. The result was the evolution of
significant work-arounds embedded into the work pro-
cesses. In future redesigns, these generic components can be
systematically incorporated into available functionality, de-
pending on the specific task. Ensuring that tasks can be
initiated, conducted, and finished with minimal navigation
between screens, software, and tools is essential.
Another important and interesting finding was that al-
though the tasks and goals were common, each site was
unique in the specific processes used. Several factors may
provide explanations. One possible factor was the variability
in team coherence. Although we did not measure this factor,
it was clear that those clinics where the nurses, physicians,
clerks, and social workers were tightly organized had very
different strategies than those where the different roles
functioned independently and in parallel. Another factor
creating differences between sites was the type and intensity
of support from the IT office. Perhaps, the work-arounds
visible in many of our interviews arose because of a failure
of the technical office to adequately support the socio-
clinical system.55Another possible source of the variation is
the degree of user expertise, both clinical and technical. Sites
where there were substantial numbers of trainees, part-time
workers, and frequent turn-arounds of trainees might re-
quire different strategies (more structured and automatic)
than those where individuals could simply learn procedures
and then would be able to stay with those procedures for
long periods of time. These variables and the relationship to
specific strategies used are important areas for future re-
In a study using this approach, the goal is not to compare
sites, or to make rate-based conclusions regarding the inci-
dence of specific tasks. Nor can a study with this design
provide any strong inferences about the causal relationship
WEIR et al., Information Management Strategies
between the information system, task structures, strategies,
and outcomes. The results from this study are valuable for
providing an in-depth and structured analysis of how clini-
cians are handling information in a computerized environ-
ment and to generate hypotheses for future investigations.
Conclusions and Recommendations
The findings from this study support some important rec-
ommendations. First, implementation of every information
system will result in users creating unique strategies that
either expand functionality or work around limitations.
Understanding those strategies should be a component of all
design life cycles. Secondly, a CPOE system can produce
significant information overload problems for users. Users
can be helped to manage the problem by minimizing the
different technologies, supporting all of the task core com-
ponents, and by providing overview functionality. Third,
communication between providers is a ubiquitous aspect of
health care delivery. Information systems may not be able to
replace communication systems but should not interfere
with them. Communication is often embedded into artifacts
and work processes that may not be clearly visible. Finally,
EHR system designers should sample sites and users widely
to ensure representatiaveness, because adaptation is creative
1. Kushniruk A, Patel V. Cognitive evaluation of decision-making
processes and assessment of information technology in medi-
cine. Int J Med Inform. 1998;51:83–90.
2. Aarts J, Ash JS, Berg M. Extending the understanding of
computerized physician order entry: implications for profes-
sional collaboration, workflow and quality of care. Int J Med
Inform. 2006; in press.
3. Goorman E, Berg M. Modeling nursing activities: electronic
patient records and their discontents. Nurs Inq. Mar 2000;7:3–9.
4. Gorman PN, Lavelle JS, Ash JS. Order creation and communi-
cation in healthcare. Methods Inf Med. 2003;42:376–84.
5. Rasmussen J, Pejterson A, Goodstein L. Cognitive Systems
Engineering. New York: John Wiley & Sons; 1994.
6. Fielder K, Juslin P. Taking the interface between mind and
environment seriously. In: Fielder K, Juslin P, eds. Information
sampling and adaptive cognition. Cambridge: Cambridge Uni-
versity Press; 2006:3–32.
7. Gigerenzer G. Adaptive thinking: rationality in the real world.
New York: Oxford University Press; 2000.
8. Woods D, Hollnagel E. Joint cognitive systems. London: Taylor
& Francis; 2006.
9. Berg M. The search for synergy: interrelating medical work and
patient care information systems. Methods Inf Med. 2003;42:
10. Berg M. Implementing information systems in health care
organizations: myths and challenges. Int J Med Inform. Dec
11. Berg M. Rationalizing medical work. Decision support tech-
niques and medical practices. Cambridge: MIT Press; 1997.
12. Bijker W, Law JE. Shaping technology-building society. Studies
in socio-technical change. Cambridge: MIT Press; 1992.
13. Kaplan B. Addressing organizational issues into the evaluation
of medical systems. J Am Med Inform Assoc. 1997;4:94–101.
14. Nebeker J, Weir C, Hicken B, Rappaport S. Expected and
unexpected barriers to the adoption of alert-processing and
task-management supplements to an established electronic
medical record. Manuscript under review.
15. Ash JS, Berg M, Coiera E. Some unintended consequences of
information technology in health care: the nature of patient care
information system-related errors. J Am Med Inform Assoc.
16. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected
increased mortality after implementation of a commercially sold
computerized physician order entry system. Pediatrics. 2005;
17. King WJ, Paice N, Rangrej J, Forestell GJ, Swartz R. The effect of
computerized physician order entry on medication errors and
adverse drug events in pediatric inpatients. Pediatrics. 2003;112:
18. Koppel R, Metlay JP, Cohen A, et al. Role of computerized
physician order entry systems in facilitating medication errors.
19. Hertwig R, Barron G, Weber E, Erev I. The role of information
sampling in risky choice. In: Fiedler K, Juslin P, eds. Information
sampling and adaptive cognition. New York, NY: Cambridge
University Press; 2006:72–91.
20. Koehler D. Explanation, imagination, and confidence in judg-
ment. Psychological Review. 1991;110:499–510.
21. Betsch T, Haberstroh S, Blockner A, Haar T, Fiedler K. The
effects of routine strength on adaptation and information search
in recurrent decision-making. Organizational Behavior and Hu-
man Decision Processes. 2001;84:23–53.
22. Fiedler K, Juslin P. Information sampling and adaptive cogni-
tion. New York, NY: Cambridge University Press; 2006.
23. Gigerenzer G, Todd P, Group A. Simple heuristics that make us
smart. Oxford: Oxford University Press; 1999.
24. Berger R, Kichak B. Computerized physician order entry: help-
ful or harmful. J Am Med Inform Assoc. 2004;11:100–3.
25. Beuscart-Zephir M, Pelayo S, Anceaux F, Meaux J, Degroisse M,
Degoulet P. Impact of CPOE on doctor-nurse cooperation for
the medication ordering and administering process. Int. J. of
Medical Informatics. 2005;74:629–41.
26. Gorman P, Lavelle M, Ash J. Order creation and communication
in healthcare. Meth. Inf. Med. 2003;42:376–84.
27. Schraagen J, Chipman S, Shalin V. Cognitive task analysis.
Mahway, NJ: Lawrence Erlbaum Associates; 2000.
28. Gordon S, Gill R. Cognitive task analysis. In: Zsambok C, Klein
G, eds. Naturalistic decision making. Mahway, NJ: Lawrence
Erlbaum Associates; 1997:139–40.
29. Klein G. Cognitive task analysis of teams. In: Schraagen J,
Chipman S, Shute V, eds. Cognitive task analysis. Mahwah, NJ;
Lawrence Erlbaum Associates, Inc., 2000.
30. Vicente K. Work domain analysis and task analysis: A differ-
ence that matters. In: Schraagen J, Shalin V, eds. Cognitive task
analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2000.
31. Ericsson K, Simon H. Protocol analysis: verbal reports as data.
Cambridge, MA: MIT Press; 1984.
32. Vallacher R, Wegner D. What do people think they are doing?
Action identification and human behavior. Psych Rev. 1987;94:
33. Patel V, Arocha J, Diermeier M, How J, Mottur-Pilson C.
Cognitive psychological studies of representation and use of
clinical practice guidelines. Int. J. Med Inform. 2001;63:147–67.
34. Horsky J, Kaufman D, Oppenheim M, Patel V. A framework for
analyzing the cognitive complexity of computer-assisted clinical
ordering. J BioMed Info. 2003;36:4–22.
35. Anderson J. Rules of the mind. Hillsdale, NJ: Lawrence Erlbaum
36. Smith E. Mental representations and memory. In: Gilbert D,
Fiske S, Linszey G, eds. That handbook of social psychology.
Vol 1. Boston: McGraw-Hill; 1998:391–445.
37. Bargh J, et al. The automated will: non-conscious activation and
pursuit of behavioral goals. J Pers Soc Psychol. 2001;81:1014–27.
Journal of the American Medical Informatics AssociationVolume 14Number 1 Jan / Feb 2007
38. Bargh J, Gollwitzer P. Environmental control of goal-directed
action: automatic and strategic contingencies between situations
and behavior. Nebr Symp Motiv. 1994;41:71–124.
39. Johnson P, Johnson H, Hamilton F. Getting the knowledge into
HCI: theoretical and practical aspects of task knowledge struc-
tures. In: Schraagen J, Chipman S, Shute V, eds. Cognitive task
analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:201–
40. Jaspers M, Steen T, Bos C. The think aloud method: a guide to
user interface design. Int J Med Inform. 2004;73:781–95.
41. Roseman CB, Patrick WK, Tucker RV. Health technology assess-
ment: a policy-relevant technology diffusion model for emerg-
ing economies of the Asia and Pacific region. Asia Pac J Public
42. Fleiss J. Statistical methods for rates and proportions. 2nd ed.
New York: John Wiley & Sons; 1981.
43. Simon H. Models of bounded rationality. Cambridge MA: MIT
44. Shepard R. On subjectively optimum selections among multi-
attribute alternatives. In: Edwards W, Tversky A, eds. Decision
making. Baltimore: Penquin Books; 1967:257–83.
45. Payne J, Bettman J, Johnson E. The Adaptive Decision-Maker.
New York: Cambridge University Press; 1993.
46. Klein G, Orasanu J, Calderwood R. Decision making in action:
models and methods. Norwood, NJ: Ablex Publishing Corpo-
47. Bhavnani S, John B. The strategic use of complex computer
systems. Human-computer interaction. 2000;15:107–37.
48. Endsley M. Theoretical underpinnings of situation awareness: A
critical review. In: Endsley M, Garland D, eds. Situation aware-
ness analysis and measurement. Mahway, New Jersey: Law-
rence Erlbaum Associates; 2000:3–32.
49. Miller G, Galanter E, Pribram K. Plans and the structure of
behavior. New York: Holt Publishers; 1960.
50. Kushniruk AW. Analysis of complex decision-making processes
in health care: cognitive approaches to health informatics.
J Biomed Inform. 2001;34:365–76.
51. Coiera E. When conversation is better than computation. J Am
Med Inform Assoc. 2000;7:277–86.
52. Suchman L. Constituting shared workspaces. In: Engestrom Y,
Middleton D, eds. Cognition and communication at work.
Cambridge, UK: Cambridge University press; 1998:35–60.
53. Xiao Y. Artifacts and collaborative work in healthcare: method-
ological, theoretical and technological implications of the tangi-
ble. J Biomed Inform. 2005;38:26–33.
54. Dykstra R. Computerized physician order entry and communi-
cation: reciprocal impacts. Proc AMIA Symp. 2002:230–4.
55. Weir C, Hicken B, Nebeker J. Crossing the quality chasm: The
role of information technology departments. Am J Med Qual (in
Appendix A: Interview Questions
Introduction: We are conducting a research study on how the
computer interfaces with clinical workflow in primary care in
order to make the computer more useful. This study is funded
by the computer development office through HSR&D. We
would like to ask you a few questions about how you organize
your workflow in a typical day. We are tape recording your
answers so that we can more accurately capture what people
are saying. Your name will not be on the tape recording nor
will it be associated in any way with your responses. Because
consent. ?INFORMED CONSENT?
We are organizing the interview chronologically, starting
with what you do to prepare for your day, going through
your visit with the patient and finishing with what you do to
follow-up. At each stage, I am going to focus on what you
are thinking, what goals you are considering, what activities
you do to meet these goals, and your concerns. Remember,
there are no right answers. ?The items in italics are only asked
if they do not come up in conversation?
Preparing to See a Patient
1. Please recall a typical day when you are working in the
clinic. What do you usually do to get ready, to prepare
yourself? ?remember to probe with you do XX in order
to and you accomplish XX, by doing ??
• When do you start getting ready, usually? How do you
arrange the information you need?
• When do you review your notifications in CPRS?
When do you answer your phone messages? (How
many?) When do you look at your e-mail? (How
many?) Do you use e-mail?
• How do you keep track of things that you need to do
as they arise from these sources?
• Do you keep a list of any kind? How do you know you
have accomplished the tasks on the list?
• When you are reviewing notes, what note do you check
• Are there any kinds of notes that you do not review?
• What are you trying to do at this point? What are your
most pressing concerns?
• How do you keep track of items that you find that you
need to focus on for each patient as you review the
2. Do you find the documentation on the patient compre-
hensive, accurate, and available?
• What do you think of templates? What do you think of cut
and paste methods?
3. How would you describe your team?
4. Do you communicate with your team on a regular basis?
5. Do you start a note? How do you usually set it up? What
are you trying to accomplish with the set-up?
6. When there is something you don’t know (a disease, a
drug), what do you usually do?
During the Patient Visit
1. As you visit with the patient, do you interact with the
computer as well?
2. Does the physical set up of the room, the patient, and the
computer work well?
3. What are the central areas of focus during the patient’s
4. What are your main goals in documenting the patient’s
Follow-up After the Visit
1. After the patient leaves, what do you usually do? Do you
finish the documentation then or later?
• When you document, do you ever cut and paste? Do you use
2. What kind of methods do you use to communicate with
WEIR et al., Information Management Strategies
• Do you mail them lab results? Download full-text
• Do you call abnormals? Do others call for you? Do you
give out your phone number?
3. How do you follow-up on their labs, procedures, results, etc?
• Do you ever know if a patient has missed a consult or
procedure before the patient comes in next?
• Do you try to find out what happened to them?
4. When you are gone, who covers for you? How is that set
5. How often do you communicate with members of your
team about a patient?
6. What are the methods you use for communicating and why?
• How often do you use e-mail; VISTA mail? Do you create a
note with a co-signer?
Journal of the American Medical Informatics AssociationVolume 14Number 1 Jan / Feb 2007