Towards a Science of Checklists

Full text

Towards a Science of Checklists
Hajo A. Reijers
Vrije Universiteit Amsterdam
h.a.reijers@vu.nl
Henrik Leopold
Vrije Universiteit Amsterdam
h.leopold@vu.nl
Jan Recker
Queensland University of
Technology
j.recker@qut.edu.au
Abstract
Checklists are in use in many work domains,
including aviation, manufacturing, quality control, and
healthcare. Despite their adoption, the literature shows
both breadth and persistence of problems with the
organizational usage of checklists. In this paper, we
conduct a structured literature survey to analyze
checklists from the perspective of informational
artifacts. Our contribution is a respective
conceptualization of checklists and a rigorous analysis
of their problems. As we will argue, these insights help
to consider how the capabilities of IT systems can be
leveraged to improve checklists and address their
problematic aspects. We present our work as a basis
for IT-oriented research into a relevant yet under-
examined information practice in organizational work
routines.
1. Introduction
Checklists are pervasive. From the simple shopping
lists in daily life to comprehensive pre-flight checklists
in aviation, people use these devices to guide and
verify their actions. The use of checklists has also
shown to be highly beneficial. Probably the most well-
known example is the surgical checklist that was
developed on basis of guidelines by the World Health
Organization, which led to halving the mortality rates
in patient populations of the hospitals who started
applying it [1]. Because of their success to reduce the
likelihood of human error to occur under daily work
conditions, checklists have also become highly
pervasive in aviation, where they are a mandatory part
of practice [2].
Despite this positive connection between the use of
checklists and organizational performance, they do not
come without problems. For example, “checklist
fatigue” has become a well-observed phenomenon,
whereby the overwhelming number of available or
required checklists becomes a hindrance rather than an
aid [3]. Also, checklist adoption varies drastically in
spite of their demonstrated benefits [4].
While researchers and practitioners have been
working on addressing these and other checklist issues,
two characteristics of the proposed solutions are worth
pointing out. First of all, the design and
implementation of checklists has mostly been a
domain-centered approach: aviation engineers work on
aviation checklists, medical professionals on medical
checklists, and so forth. Secondly, none of the
solutions we have seen are coupled to something that
could be construed fundamental properties of
checklists. Instead, they focus on organizational
measures to improve the acceptance and use of
checklists or deal with their surface aspects only. These
limitations have led to calls for a “science of
checklists” [5] – yet, with little follow-up to date.
In this paper, we wish to shed a new, information
systems-oriented perspective on checklists.
Specifically, we propose to view and analyze
checklists as principally informational artifacts that
guide work. This allows for a reflection on IT system
attributes that can be exploited to reshape checklists
and, by doing so, overcome pertinent issues with their
usage. The upshot is that checklists may become
applicable in even wider societal settings than is
currently the case, with all their associated benefits.
Following a structured literature review, the main
contributions of this work are (1) a conceptualization
of checklists as informational artifacts and (2) a
rigorous analysis of existing checklist problems. We
will build on these to argue that an IT-driven overhaul
of checklists may both be feasible and attractive.
We proceed as follows. First, we introduce the
notion of a checklist in Section 2. In Section 3, we
explain the methodological details of our structured
literature survey. Section 4 then presents our
conceptualization of checklists as informational
artifacts. In Section 5, we provide a comprehensive
overview of the problems with checklist usage, as well
as the known strategies to deal with these. Our
proposals to address pertinent issues are presented in
Section 6. Section 7 concludes this paper.

2. What is a Checklist?
A lemma in the Merriam-Webster dictionary
describes a checklist as a “list of things to be checked
or done”. This definition captures the supportive nature
of a checklist to “tick off” work that has been carried
out and to serve as a reminder of what is still left to do.
Note how the concept of a “to-do list” specifically taps
into the latter aspect of a checklist. This is why we will
consider it as being subsumed by the wider checklist
notion.
Another notion worth mentioning is that of the
“timeout” aspect of checklists, which is popular in the
healthcare domain. It refers to the particular pause that
allows a team to go through a relevant checklist just
before an important procedure. Although it is
colloquially used as a synonym for a checklist, we
think it is preferable to distinguish the list from the
interruption of work.
For the sake of clarity, we follow [2] for our
definition of a checklist: “A checklist is typically a list
of action items or criteria arranged in a systematic
manner, allowing the user to record the
presence/absence of the individual items listed to
ensure that all are considered or completed”.
As an example of a checklist, consider the Surgical
Safety Checklist from the World Health Organization
(see Figure 1). This checklist is intended to support
surgical personnel to guarantee the safety of patients
before, during, and after a surgery. Already a first
glance reveals that this checklist is more than a simple
collection of check boxes. We will be using this
example to explain our conceptualization of checklists
below.
3. Methodology
To be able to conceptualize checklists as
informational artifacts and to develop a comprehensive
overview of the issues that are associated with them,
we carried out a structured literature survey. Our goal
was to aggregate data on a narrow scope, viz., concepts
and issues with checklists reported in academic
literature. Therefore, our literature review is a type of
qualitative systematic review [26]. We carried out the
review as follows.
Since checklists are used in a wide variety of
domains, we did not opt for any domain-specific
libraries or databases. Instead, we decided to use
Google Scholar, which can be considered as the most
comprehensive search engine for the academic
literature at this point, with coverage of all scientific
domains. As search terms we used the exact phrases
“checklist characteristics”, “characteristics of
checklists”, “problem with a checklist”, “problems
with a checklist”, “problem with checklists”, and
“problems with checklists”. In this way, we covered
both literature discussing the concepts of checklists as
well as the problems that occurred with their usage.
Figure 1: Example of a checklist [6]

Note that more generic search terms such as
“checklist” or “checklist problem” turned out to be not
selective enough: They often led to papers that
reported on the actual use of checklists to diagnose and
solve a range of problems, i.e. they captured an
operational instead of a reflective perspective on
checklists and their management.
Our search led to the identification of 101 academic
papers of which 56 papers could be retrieved. All of
these papers were read in full to identify concepts and
issues that seemed potentially applicable to more than
a single checklist. To complement this broad-brush
procedure, we conducted a qualitative search, which
we initiated on the basis of a recent publication in
Nature: “The Trouble With Checklists” [27]. We
analyzed this paper for additional concepts and
problem types and traced all the cited references. We
screened these papers for relevance and added them to
our pool in case we would expect the identification of
additional recurring checklist concepts and issues from
them. We repeated this procedure for all references of
newly found papers until the search was exhausted. For
all the papers that were added in this way, we also
screened the papers citing these works themselves, in
this way implementing a “snowballing” procedure.
This additional search procedure led to the
identification of 31 further papers.
In a final step, we iteratively applied open coding to
identify and group concepts and issues. The result was
a checklist conceptualization based on their scope and
properties as well as a number of reoccurring issue
types. In what follows, we will first introduce the
conceptualization of checklists.
4. Conceptualization of Checklists as
Informational Artifacts
The key postulation behind our work is that
checklists are a type of informational artifact that
conceptualize activities and decisions in work routines.
Informational artifacts encapsulate, abstract, and
represent all relevant information about some real-
world phenomena in a single abstraction. This
assumption suggests that the essential purpose of
checklists is twofold: (1) describe work routines
(which we call the aspect of representation) and (2)
guide decisions and tasks within such routines (the
aspect of prescription). The nature of checklists is then
that they are a type of conceptual model [7] that
provide a purposeful and relevant representation of a
particular real-world domain (the aspect of
abstraction).
This assumption is useful for delineating properties
of checklists as information artifacts in two ways. First,
because it implies that the nature and purpose of
checklists is similar to other approaches used for
conceptualizing activities and decisions carried out in
some real-world domain such as process models [8][9],
routine networks [10], or state machines [11]. This will
be fruitful for identifying IT strategies available for
this artifacts and applying them to checklists, as we
will illustrate below. Second, because it allows
developing an understanding of relevant properties in
general [12] that describe types of checklists, which is
useful to discriminate specific kinds of checklists that
are or may be in use. We return to this issue as well.
To offer a first conceptualization of the spectrum of
kinds of checklists, we identify seven properties (see
Table 1). Five of them relate to the entire checklist,
while two relate to its constituting elements in
particular, i.e. the checklist items.
Table 1: Properties of checklists
Scope
Property
Selected values
Entire
Checklist
Representation
Paper, poster,
mechanical, electronic,
vocal
Prescriptiveness
Do-list, call-do
response
Scope
System engineering,
human performance
Abstraction
Normal, abnormal,
emergency
Audience
Individual, group
Checklist
items
Type
Check, score, multiple
choice, branched,
interrogative
Behavioral
relation
Arbitrary, strongly
sequential, weakly
sequential, parallel
4.1. Properties of the Entire Checklist
The first checklist property concerns the aspect of
representation of checklists, that is, the question of
how a work routine, its activities and decisions, are
conceptualized in a checklist. The most common and
also simplest representation of a checklist is a paper-
based checklist. The advantage of a paper-based
checklist is its low technical complexity and high
reliability [13]. Furthermore, it is portable and easily
reproducible. An alternative paper-based type of
checklist is a poster checklist [14]. In comparison to a
regular paper-based checklist, it has the advantage that
it is visible to a larger group of people. In particular, in
a surgical setting this may represent an important
characteristic of the checklist. Non-paper-based

representations of checklists include mechanical,
electronic, and vocal checklists. Mechanical checklists
consist of a panel composed of several plastic slides
moving over a list of checklist items [15]. One
important feature of mechanical checklists is that they
only show the non-accomplished items. They are, for
instance, used in aviation in the context of takeoff and
landing. The term electronic checklist or automated
checklist applies to any checklist that is shown on a
digital display, which assumes that it is digitally
represented in some way. Electronic checklists are
marked using a cursor and have the advantage of
dynamic updates and a wide range of visualization
opportunities. The biggest advantage of electronic
checklists, however, is the opportunity of an associated
system to technically check whether a task was
actually performed [13]. Especially in critical domains
such as aviation, this can be an important feature.
Vocal checklists are typically considered as a special
type of electronic checklist. In essence, a vocal
checklist is technical unit that “reads” the checklist
items to the user [16]. The user confirms the
accomplishment of a task by pushing a button. Recent
research indicated that such audible checklists can lead
to a decrease of in-flight errors by pilots [17]. Relating
the aspect of representation to the checklist from
Figure 1, we observe a paper-based checklist.
However, it is also worth noticing that this checklist
instructs to read certain aspects out loud (“read
specimen labels aloud”).
The second checklist property concerns the aspect
of prescription of checklists, that is, the question of
which type of guidance is provided through the
checklist to a user. We found two dominant checklist
prescriptions in use: call-do response and do-verify
[13] The call-do-response method, often also called do-
list or challenge-do-response checklist, follows a step-
by-step “cookbook” approach [16]. When using the
list, a user first calls an item from the list, then
performs the action, and finally verifies the successful
accomplishment of that action. This method is
particularly effective when different persons are
involved. One person could then perform the action,
while the other takes care of the verification [13]. A
typical application scenario for a call-do-response
checklist is a surgical setting where safety-relevant
aspects can be cross-checked [18]. A checklist that
follows the do-verify method has the character of a
backup. Users first rely on their memory to perform a
number of required tasks. Then, they use the checklist
to verify that each task has been accomplished
successfully. As an example, consider a pilot, who first
configures the aircraft according to memory and then
uses the checklist to verify that all configurations were
correctly set [16]. Note that the checklist from Figure 1
is a typical call-do-response checklist. The users of this
checklist are supposed to go through the list step by
step and verify for each item that has been completed
successfully.
The third checklist property, scope, concerns the
coverage aspect, that is, to what extent the checklist
covers the entire list of required tasks. We distinguish
two approaches to coverage: the system engineering
approach and the human performance approach. The
main rationale behind the systems engineering
approach is that all items related to a task should be
checked. As a result, the user potentially faces a long
checklist with an extensive number of items. The
human performance approach, by contrast, emphasizes
that a detailed checklist is no guarantee for preventing
human failure. According to the human performance
approach, an extensive checklist rather carries the risk
that users will fail to use it correctly or deny to use it
altogether [13]. It can be seen that the checklist from
Figure 1 represents an example of the human
performance approach. It actually explicitly
emphasizes this approach below the checklist: “This
checklist is not intended to be comprehensive”.
The forth checklist property concerns the aspect of
abstraction, that is, the question which properties of
the phenomena in the real-world domain are explicated
through the checklist. Here we distinguish normal
operations (checklists that capture routines how they
should work in most cases), non-normal (checklists
that capture escalations such as workarounds [19]), and
emergency checklists (checklists that only capture
singular routines such as disaster management or
others). The checklist from Figure 1 illustrates that this
distinction is not always and necessarily sharp. While it
is intended for normal operations, it also encourages
anticipating critical events (see the middle column).
The fifth checklist property concerns the aspect of
audience, that is, whether the checklist is meant to be
used by an individual or by a group of people. An
example for a checklist that typically is meant to be
used by two people is the previously introduced
challenge-do-response checklist [5]. The checklist
from Figure 1 illustrates this nicely. The target group
of this checklist includes several roles including a
nurse, an anesthesiologist, and a surgeon. For each of
the three columns, the addressed roles are explicitly
mentioned. In the right-hand column, we can even find
specific instructions how their joint use of the checklist
is meant to take place: “Nurse verbally confirms”.
4.2 Properties of the Checklist Items
While the previously discussed properties relate to
the entire checklist, there also two key characteristics

of a checklist that particularly relate to the items that
compose the checklist.
The first property concerns the type of a checklist
item. While the term “checklist” suggests that items
simply need to be ticked off, checklist items are, in
fact, much more multi-faceted. We distinguish between
five item types: check, score, multiple-choice,
interrogative, and branched. The check type represents
the simplest form of a checklist item: It consists of a
task or a goal that has to be accomplished. Typically,
the item is ticked off once the associated task has been
performed. However, especially in aviation, check
items are not marked [13]. Checklist items of the score
type provide the user with the possibility to assign a
score to an item (e.g., from 1 to 10). These items are
often used in checklists for evaluation purposes. An
example is the trauma checklist from [20], which
consists of a number of symptoms that are scored from
0 to 3. Items of the multiple-choice type offer the user
with several response possibilities [21]. Hence, they
are used in situations where the responses or outcomes
are already known upfront. The simplest form of a
multiple choice item is the yes/no item. For instance, a
checklist that consists of 23 yes/no items to diagnose
autism among toddlers is described in [22]. However,
multiple-choice items may also relate to more specific
outcomes. When several multiple-choice items are
combined in such a way that the outcome of one choice
leads to another, we refer to this as a branched item
[21]. A branched item explicitly shows the
dependencies between items and, therefore, can be
beneficial for representing nested choices. Checklist
items that are interrogative require the user to provide
feedback. Thus, they typically consist of a question and
an empty field [21]. Such items are particularly useful
when the set of answers cannot be anticipated. The
checklist from Figure 1 illustrates that a single
checklist can combine several item types. We observe
a mix of check items (“Has the patient confirmed
his/her identity, site, procedure, and consent?”),
multiple choice items (“Is the site marked?”), and
interrogative items (“What are the concerns for
recovery and management of this patient?”).
The second property concerns the behavioral
relation between checklist items, that is, in which order
the tasks related to two items have to be carried out.
We distinguish between four types of behavioral
relations: arbitrary, strongly sequential, weakly
sequential, and parallel. An arbitrary relation between
two items defines that the completion order of two
items is of no importance [23]. The main purpose of
checklists with arbitrary items is therefore to serve as a
mnemonic device. When two checklist items are in a
strongly sequential relationship, the order between
these items has to be preserved to obtain valid
outcomes. As an example, consider the case of a pre-
flight checklist, where the accuracy of certain
instruments depends on zeroing their settings in the
first place [23]. It is important to note that a sequential
relationship can also lead to the repetition of an item.
Such iterations are necessary when problems or
discoveries from later checklist items require the
reconsideration of earlier ones [24]. A weakly
sequential relationship between two items specifies
that the completion order is only important from a
psychological or efficiency point of view. It may, for
instance, be beneficial to ask the user to carry out
cognitively demanding tasks first, such that they are
accomplished at the required level of quality [23].
While it is not as common as the previously described
relations, two checklist items can also be in a parallel
relationship. Such a situation occurs when a second
item has to be completed while the realization of the
first one is still in progress. An example of a parallel
relationship can be found in the approach and landing
checklist for aircrafts [25]. Among others, it requires
the pilot to gear down while also taking care of the
landing flaps. Since it might be not sufficient to wait
for the first item to be completed, both tasks are
associated with items that are in a parallel relationship.
In practice, the combination of the above described
relationships takes place in different ways. For
instance, the checklist from Figure 1 combines strongly
sequential order with weakly sequential order items.
The three columns clearly need to be executed from
left to right. The checklist even further specifies the
point of time by stating “before induction of
anesthesia”, “before skin incision”, and “before patient
leaves operating room”. Within each column, the order
is partially strongly and partially weakly sequential.
For instance, the confirmation of the patient’s identity
is clearly supposed to take place before the site is
marked. The order in which allergies of aspiration risk
are determined is certainly of less importance.
However, going from top to bottom is the intended
mode of use.
The above discussion illustrates the power but also
the complexity of the checklist as an informational
artifact. It is thus not surprising that practitioners face
different types of problems with checklists. We take a
detailed look at these in the next section.
5. Checklists Problems and Solutions
As a result of our literature survey, we derived 71
checklist issues, which we grouped into 21 recurring
issue types. In this section, we introduce these
recurring problems with checklist use, as well as the
state-of-the-art strategies advocated to address these.

Table 2: Issues with checklists
Category
Problem
Citations
Related Checklist Property
Checklist
Design
Measurement problems with items
14
Checklist item: Type
Difficulty to come up with a standardized
version
2
Checklist: Representation, Scope, and Abstraction
Operational Use
Checklist not sensitive to context or case
19
Checklist: Abstraction
Non-compliance
14
Checklist: Prescriptiveness
“Checklist fatigue”
12
Checklist: Prescriptiveness, Scope
Reliant on human judgment
10
Checklist: Representation¸
Prescriptiveness
Poor integration with existing process
5
Checklist: Representation, Scope
Cognitive issues
4
Checklist: Representation, Scope, and Abstraction
Difficult to read status/ receive feedback
from checklist
4
Checklist: Representation,
Abstraction, and Prescriptiveness
No predictive/prescriptive power
3
Checklist: Prescriptiveness
Duplication of tasks in existing process
3
Checklist item: Behavioral relation
Difficulty to deal with exceptions
3
Checklist: Abstraction
Use highly susceptible to production pressure
1
Checklist: Prescriptiveness
Selection of wrong parts/ paths
1
Checklist item: Behavioral relation
Checklist
Management
Difficulty to update
5
Checklist: Representation
Selection of wrong checklist
2
Checklist: Abstraction, Audience
Difficulty to manage variety of checklists
2
Checklist: Scope, Abstraction, and Audience
Organizational
Context
False impression work is well done/ tasks are
well understood
9
Checklist: Prescriptiveness
Senior staff loathes use/ fears loss of
autonomy
6
Checklist: Prescriptiveness, Audience
Success depends on implementation
4
Checklist: Representation
Does not foster communication/teamwork
2
Checklist: Scope
Creates anxiety with subjects (patients)
2
Checklist: Abstraction
5.1. Reported Problems with Checklists
The 21 issue types we identified are summarized
in Table 2, along with their citations as a proxy for
perceived importance, and their association to the
properties of checklists as information artifacts
(Table 1). The identified issues are also categorized
into four classes:
 Checklist Design: Issues that pertain to the
initial development of a checklist for guidance in
a specific context;
 Operational Use: Issues that relate to the
operational use of checklist that has been
designed for a problem domain;
 Checklist Management: Issues that deal with
maintaining a checklist or a group of checklists
beyond their initial design and initial use;
 Organizational Context: Problems that involve
the organizational setting in which a checklist is
used.
In Table 2 we further linked the issue types to the
checklist properties that we discussed in Section 4.
Note that the properties that are most often involved

in the issues we identified are Representation,
Prescriptiveness, and Abstraction.
5.2. Reported Solutions for Checklist
Problems
We reasoned that papers that would devote
considerable attention to describe and characterize
structural problems with checklists would also be the
most likely sources to discuss solutions proposed or
even already applied to deal with these. Using the
same pool of papers and once again relying on an
iterative style of open coding, we re-analyzed the
literature to identify strategies that diminish or
eliminate recurring problems associated to the use of
checklists as captured in Table 2. We found 14 such
strategies in the pool of papers. These strategies can
be divided into two categories.
The first category, organizational strategies,
covers those strategies that focus on the introduction
and uptake of a checklist within its organizational
setting:
1. Include the use of a checklist within an
organizational improvement cycle, including the
pro-vision of feedback as to its actual usage and
its effectiveness [1][27][28][29][30][31];
2. Extensively train the staff that is expected to use
a checklist prior to its first use [13][27][28]
[30][31][32][33][34];
3. Provide insight into the evidence behind the
checklist items and clarify the method that has
been used to develop the checklist [1][31];
4. Instill accountability for the actual (non)use of a
checklist or even enforce its usage [29][31]
[33][34][35];
5. Select champions for the use of a checklist and
insist on an organization’s leadership to pro-
mote and support the checklist program
[28][31][32][33];
6. Clearly define the different roles and
responsibilities that people have in the use of a
checklist [13][33];
7. Closely integrate a checklist with the existing
systems and operational processes
[29][31][32][35][37];
8. Limit the organizational use of checklists to
where their application is appropriate [38][39].
The second category of strategies, adaptation
strategies, deal with solutions that involve changing
the checklist itself:
9. Properly design checklists, in particular to the
extent that they are clear and simple
[13][29][35][40][41][42];
10. Adapt centrally designed checklists to fit with
the local circumstances in which it they are to be
applied [27][29][31][33][36];
11. Establish an end-to-end coverage of the
checklist, for example: from pre-operative
procedures to the actual discharge of a patient
[35][30];
12. Use the advantages of electronic checklists
[13][35][40][43][44];
13. Similarly design different checklists when they
are used by the same staff (to make infrequent
use easier) [43];
14. Allow that a checklist can be combined with
other modes of process guidance [43].
In the following section, we will discuss the insights
we obtained from studying these strategies.
6. Discussion
We evaluated the suggested strategies versus the
identified problems and arrived at three observations.
Observation 1: State-of-the-art ignores the
informational nature of checklists. We found the
identified organizational strategies not to be much
different from those strategies that are proposed when
introducing new concepts or IT systems in
organizations [45] or on governance issues with other
representation methods for work procedures, say,
process models [46]. Similar to these, the checklist
strategies focus on sponsorship, training, and
adoption, while also calling for a proper reflection on
appropriate use. In this sense, we contend that these
strategies are not at all checklist-specific. It is
striking, though, that the strategies in this category
are the ones that receive most attention in the recent
literature. As a case in point, all identified
publications that advocate organizational strategies
are published in the year 2009 or later. Also, the most
recent papers that we found emphasize this type of
strategy, see [27][30][31][39]. This suggests that the
state-of-the-art on overcoming checklist issues does
not relate to the nature of the checklist as an
informational artifact.
Observation 2: Many important problems are
not attended to. We found that the various strategies
suggested in the literature indicate an imbalance in
how they address the problems that we identified
earlier. While some categories of problems are
thoroughly covered, the problems in the other
categories receive far less attention or none at all.
Specifically:

 All problems in the ‘Organizational Context’
category are explicitly addressed. Indeed, issues
that deal with the lack of adoption of the
guidelines by the clinical staff or their
inappropriate use of checklists seem to be at the
core of many of the organizational strategies that
we found, see [28][31].
 The problems in the ‘Checklist Design’ category
are generally considered as solved or avoidable.
For example, in publications from the previous
century, the development of items to measure
progress in an operational process or to guide a
user were seen as considerable, see e.g. [47]. In
more recent literature, the view is that “figuring
out what should form the content of a checklist
for a […] problem is a nonetheless achievable
ambition” [32]. Also, there appears to be
consensus that checklists should not be applied
in all situations, for example when unexpected
events are frequent [39]. By contrast, our
analysis of the issues in Table 2 suggests, that
several design properties of checklists appear to
be recurring issues, in particular those in relation
to Representation, Prescriptiveness, and
Abstraction.
 As to the problems in the ‘Operational Use’
category, many of the pertinent issues do receive
attention – in particular the issues of “checklist
fatigue” and non-compliance – yet three notable
problems lack any substantial reflection on to
how they are to be overcome:
o checklists are not sensitive to context or
case [31];
o checklists have no predictive/
prescriptive power [48]; and
o checklists have difficulties in dealing
with exceptions [35].
Notably, these three problems all relate to
properties of the checklist of an artifact, in terms
of its representational, prescriptive and
abstraction aspect. The problem of context/case
sensitivity is the most cited problem within the
‘Operational Use’ category. While some
solutions specifically aim at adapting a checklist
to fit with the local circumstances [29][31], this
always relates to the a-priori design of the
checklist – not its run-time adaptation. The other
two problems have not been addressed at all.
 None of problems in the ‘Checklist
Management’ category have been addressed by
any of the solutions. This indicates that a
checklist is not recognized as an artifact that has
a life cycle of its own and, as such, could be
supported by instruments and techniques to
manage this life cycle.
From this discussion, we conclude that important
problems are left unattended to.
Observation 3: IT solutions have potential.
Solution strategy #12, to use the advantages of an
electronic checklist, is, from all the ones identified,
the only strategy that picks up on the informational
nature of a checklist (i.e. at a deep level structure
instead of its physical surface structure [49]). But for
some reason this strategy has only been pursued
cursorily. Yet, precisely an electronic format for a
checklist, in theory, can simultaneously address
issues related to aspects of Representation, Scope,
Audience and Prescriptiveness. More important,
through its digitization, a checklist has the potential
to evolve into a “smart machine” cf. [50], in the sense
that it can provide abilities not only to inform but also
automate work, much like contemporary enterprise
IT systems.
To determine the potential of this view, let us re-
consider the open problems we detected in the
previous subsection (highlighted grey in Table 2). As
to the ‘Operational Use’ category, an electronic
checklist, as part of an IT system, could:
 adjust itself depending on the context it is used in
or the case it is used for; for example, the logic
for behavioral relations between checklist items
may only hold under certain conditions, which
can be automatically checked; the behavior of
the checklist is then adapted without the user
having to take any action;
 predict which items may become relevant or
critical; this can be done, for example, by
automatically collecting previous data on the
usage of a checklist and monitoring new
instantiations for commonalities with these
historic cases;
 adapt the level of leniency on accepting
deviations from its idealized execution or change
its actual content at run-time if the user signals
an exceptional situation.
As to the problems in the ‘Checklist Management’
category, a digitized checklist would be easier:
 to be remotely updated, similar to how new
versions of software versions are electronically
distributed;
 to be incorporated in a decision procedure –
perhaps implemented as a “master checklist” – to
determine what the most appropriate checklist is
to be used for a specific case;
 to be incorporated as member of a product
family of checklists, with facilities for version
management, re-use of functionality,
configuration, etc.

All of these strategies admittedly lack much detail at
this point, but we hope that the reader can envisage
these and agree with us that they lie within the
capabilities of modern IT systems. Therefore,
strategies and technologies that build on the
informational nature of a checklist seem attractive to
exploit.
7. Conclusion
In this paper, we adopted an informational view
on checklists to offer a new conceptualization of
these artifacts. In addition, we carried out a thorough
analysis of the persistent problems that are associated
with the organizational use of checklists. When we
analyzed these problems with this informational view
in mind, we drew the conclusion that the capabilities
of IT systems offer a rich potential to better manage
(digital) checklists and put them to operational use.
Our paper can be seen as a call to action for the IT
systems community to embrace checklists as
appropriate and worthy artifacts of study. Until this
point, some may see checklists as trivial artifacts that
can be left to be designed by professionals from their
various application domains. Given the huge impact
that the proper usage of checklists can have on the
quality of procedures in settings such as healthcare,
manufacturing, aviation, and many more, we feel that
IT researchers may be compelled by the impact they
can make with joining this research line.
8. References
[1] Haynes, A. B. et al. (2009). “A surgical safety
checklist to reduce morbidity and mortality in a global
population.” The New England Journal of Medicine,
360, 491-499.
[2] Hales, B. M. and Pronovost, P. J. (2006). “The
checklist: A tool for error management and
performance improvement.” Journal of Critical Care,
21 (3), 231-235.
[3] Thomassen, Ø. et al. (2010). “The effect of a simple
checklist on frequent pre-induction deficiencies.” Acta
Anaesthesiologica Scandinavica, 54 (10), 1179-1184.
[4] Rosen, D. (2010). “The checklist manifesto: How to
get things right.” JAMA 303 (7), 670-673.
[5] Winters, B. D., Gurses, A. P., Lehmann, H., Sexton, J.
B., Rampersad, C. J. and Pronovost, P. J. (2009).
“Clinical review: checklists-translating evidence into
practice.” Critical Care, 13(6), 210.
[6] World Health Organization (2009). Surgical Safety
Checklist.
http://www.who.int/patientsafety/safesurgery/en/
[7] Burton-Jones, A. and Weber, R. (2014). “Building
Conceptual Modeling on the Foundation of
Ontology.” In Computing Handbook, Third Edition:
Information Systems and Information Technology
(Topi, H. and Tucker, A., Eds), pp 15-1-15-24, CRC
Press, Boca Raton, Florida.
[8] Curtis, B., Kellner, M. I. and Over, J. (1992). “Process
Modeling.” Communications of the ACM, 35 (9), 75-
90.
[9] Recker, J., Rosemann, M., Indulska, M. and Green, P.
(2009). “Business Process Modeling: A Comparative
Analysis.” Journal of the Association for Information
Systems, 10 (4), 333-363.
[10] Pentland, B. T. and Rueter, H. H. (1994).
“Organizational Routines as Grammars of Action.”
Administrative Science Quarterly, 39 (3), 484-510.
[11] Schneider, F. B. (1990). “Implementing Fault-Tolerant
Services Using the State Machine Approach: a
Tutorial.” ACM Computing Surveys, 22 (4), 299-319.
[12] Shanks, G., Moody, D. L., Nuredini, J., Tobin, D. and
Weber, R. (2010). “Representing Classes of Things
and Properties in General in Conceptual Modelling:
An Empirical Evaluation.” Journal of Database
Management, 21 (2), 1-25.
[13] Verdaasdonk, E. G. G., Stassen, L. P. S.,
Widhiasmara, P. P. and Dankelman, J. (2009).
“Requirements for the design and implementation of
checklists for surgical processes.” Surgical endoscopy,
23 (4), 715-726.
[14] Paull, D. E., Mazzia, L. M., Wood, S. D., Theis, M. S.,
Robinson, L. D., Carney, B., Neily, J., Mills, P.D. &
Bagian, J. P. (2010). “Briefing guide study:
preoperative briefing and postoperative debriefing
checklists in the Veterans Health Administration
medical team training program.” The American
Journal of Surgery, 200 (5), 620-623.
[15] Degani, A. and Wiener, E. L. (1990). “Human factors
of flight-deck checklists: the normal checklist.” Ames
Research Center.
[16] Degani, A. and Wiener, E. L. (1993). “Cockpit
checklists: Concepts, design, and use.” Human
Factors: The Journal of the Human Factors and
Ergonomics Society, 35 (2), 345-359.
[17] Hilton, B. (2012). “Comparing the Effects of
Simulated, Intelligent Audible, Checklists and Analog
Checklists in Simulated Flight.”
[18] Diamond, T. and Mole, D. J. (2005). “Anatomical
orientation and cross‐checking—the key to safer
laparoscopic cholecystectomy.” British journal of
surgery, 92 (6), 663-664.
[19] Alter, S. (2014). “Theory of Workarounds.”
Communications of the Association for Information
Systems 34 (55), 1041-1066.
[20] Briere, J. and Runtz, M. (1989). “The Trauma
Symptom Checklist (TSC-33). Early data on a new
scale.” Journal of Interpersonal Violence, 4 (2), 151-
163.
[21] Seoane, P. J. G. (2001). “Use and limitations of
checklists. Other strategies for audits and inspections.”
The Quality Assurance Journal, 5 (3), 133-136.
[22] Robins, D. L., Fein, D., Barton, M. L. and Green, J. A.
(2001). “The modified checklist for autism in toddlers:
An initial study investigating the early detection of
autism and pervasive developmental disorders.”

Journal of autism and developmental disorders, 31 (2),
131-144.
[23] Scriven, M. (2005). “The logic and methodology of
checklists.” http://preval.org/documentos/2075.pdf
[24] Scriven, M. (2007). “Key evaluation checklist
(KEC).”
[25] Mauro, R., Degani, A., Loukopoulos, L. and Barshi, I.
(2012). “The Operational Context of Procedures and
Checklists in Commercial Aviation.” In Proceedings
of the Human Factors and Ergonomics Society Annual
Meeting, 56 (1), 758-762. SAGE Publications.
[26] Paré, G., Trudel, M.-C., Jaana, M. and Kitsiou, S.
(2015). “Synthesizing Information Systems
Knowledge: A Typology of Literature Reviews.”
Information & Management, 52 (2), 183-199.
[27] Anthes, E. (2015). “The Trouble with Checklists.”
Nature, 523, 516-518.
[28] Conley, D.M. et al. (2011). “Effective surgical safety
checklist implementation.” Journal of the American
College of Surgeons, 212 (5), 873-879.
[29] Fourcade, A. et al. (2011). “Barriers to staff adoption
of a surgical safety checklist.” BMJ quality & safety,
21 (3), 191-197.
[30] Urbach, D. R. et al. (2014). “Introduction of surgical
safety checklists in Ontario, Canada.” The New
England Journal of Medicine, 370 (11), 1029-1038.
[31] Russ, S. J. et al. (2015). “A qualitative evaluation of
the barriers and facilitators toward implementation of
the WHO surgical safety checklist across hospitals in
England: Lessons from the surgical checklist
implementation project.” Annals of Surgery, 261 (1),
81-91.
[32] Bosk, C. L. et al. (2009). “Reality check for
checklists.” The Lancet, 374 (9688), 444-445
[33] Vats, A., Vincent, C. A., Nagpal, K., Davies, R. W.,
Darzi, A. and Moorthy, K. (2010). “Practical
challenges of introducing WHO surgical checklist: UK
pilot experience.” BMJ, 340.
[34] Bliss, L. A. et al. (2012). “Thirty-day outcomes
support implementation of a surgical safety checklist.”
Journal of the American College of Surgeons, 215 (6),
766-776.
[35] de Vries, E.N. et al. (2009). “Development and
validation of the SURgical PAtient Safety System
(SURPASS) checklist.” Quality and Safety in Health
Care, 18 (2), 121-126.
[36] Lingard, L. et al. (2011). “Evaluation of a preoperative
checklist and team briefing among surgeons, nurses,
and anesthesiologists to reduce failures in
communication.” Archives of Surgery, 143 (1), 12-17.
[37] de Vries, E.N. et al. (2010). “Effect of a
comprehensive surgical safety system on patient
outcomes.” The New England Journal of Medicine,
363 (20), 1928-1937.
[38] Davidoff, F. (2010). “Checklists and guidelines:
Imaging techniques for visualizing what to do.”
JAMA, 304 (2), 206-207.
[39] Stock, C. T. and Sundt, T. (2015). “Timeout for
checklists?” Annals of Surgery, 261 (5), 841-842.
[40] Blike, G. and Biddle, C. (2000). “Preanesthesia
detection of equipment faults by anesthesia providers
at an academic hospital: Comparison of standard
practice and a new electronic checklist.” AANA
Journal, 68 (6), 497-505.
[41] Helmreich, R. L. (2000). “On error management:
Lessons from aviation.” British Medical Journal, 320
(7237), 781-785.
[42] van Klei, W. A. et al. (2012). “Effects of the
introduction of the WHO Surgical Safety Checklist on
in-hospital mortality: A cohort study.” Annals of
Surgery 255 (1), 44-49.
[43] Boorman, D. (2001). “Today's electronic checklists
reduce likelihood of crew errors and help prevent
mishaps.” ICAO Journal, 1, 1-20.
[44] Hart, E. M. and Owen, H. (2005). “Errors and
omissions in anesthesia: A pilot study using a pilot's
checklist.” Anesthesia and Analgesia, 101, 246-250.
[45] Walsham, G. (1993). “IS Strategy and
Implementation: A Case Study of a Building Society.”
SIGOIS Bulletin, 14 (2), 13-16.
[46] Indulska, M., Recker, J., Rosemann, M. and Green, P.
(2009). “Process Modeling: Current Issues and Future
Challenges.” In Advanced Information Systems
Engineering - CAiSE 2009 (van Eck, P. and Gordijn,
J. and Wieringa, R., Eds), pp 501-514, Springer,
Amsterdam, The Netherlands.
[47] Wells, G.L., Leippe, M.R. and Ostrom, T. M. (1979).
“Guidelines for empirically assessing the fairness of a
lineup.” Law and Human Behavior 3 (4), 285-293.
[48] Phelps, D. C. (2004). “Information System Security:
Self-Efficacy and Security Effectiveness in Florida
Libraries”. Electronic Theses, Treatises and
Dissertations. Paper 291.
[49] Wand, Y. and Weber, R. (1990). “An Ontological
Model of an Information System.” IEEE Transactions
on Software Engineering, 16 (11), 1282-1292.
[50] Zuboff, S. (1988). In the age of the Smart Machine:
The future of work and power. USA: Basic Book.