USING OBSERVATIONAL STUDY AS A TOOL FOR DISCOVERY: UNCOVERING COGNITIVE AND COLLABORATIVE DEMANDS AND ADAPTIVE STRATEGIES

Full text

1
USING OBSERVATIONAL STUDY AS A TOOL FOR DISCOVERY:
UNCOVERING COGNITIVE AND COLLABORATIVE DEMANDS AND
ADAPTIVE STRATEGIES*
Emilie M. Roth
Roth Cognitive Engineering
Brookline, MA
Emily S. Patterson
Institute for Ergonomics, Ohio State University
Midwest Patient Safety Center of Inquiry, Veteran’s Administration
Columbus, OH
INTRODUCTION
One of the primary strengths of naturalistic observations is that they support a discovery
process (Woods, 1995; Mumaw, Roth, Vicente and Burns, 2000). They serve to draw attention
to significant phenomena and suggest new ideas whose validity and generality can then be
evaluated through additional studies.
Field observations afford the opportunity to gain a realistic view of the full complexity of the
work environment and empirically grounded hypotheses for how interventions could impact the
nature of work in that setting. They enable researchers to uncover and document cognitive and
collaborative demands imposed by a domain, the strategies that practitioners have developed in
response to those demands, and the role that existing artifacts play in meeting domain demands.
The results can be used to point to and guide the development of new types of support systems.
In this paper, two studies are used to illustrate this approach. In the first case (Roth, Malsch
& Multer, 2001), a series of field observations and structured interviews were conducted at train
dispatching centers to inform the design of a “data link” technology intended to improve
performance by reducing communications on an overloaded audio channel. In the second case
(Roth & O’Hara, 1999), observations of the use of advanced human-system interfaces in a
nuclear power plant simulator were conducted prior to implementation in the plant in order to
uncover and document unanticipitated changes in cognitive and collaborative demands as a result
of the introduction of the new technology.
The studies illustrate the methods used in conducting and analyzing the results of
observational studies, as well as the kinds of insights that can be gained from observational
studies.
OVERVIEW OF METHODOLOGY
Naturalistic observation studies employ a methodology similar in approach to other
ethnographically derived methods (e.g., Jordan & Henderson, 1995; Nardi, 1997) and the
European field study tradition (De Keyser, 1990; Heath and Luff, 2000). Observers are placed
in the actual work setting to observe and interview domain practitioners as opportunities arise.
Particular attention is placed on detailed capture of illustrative incidents that provide concrete
* Presented at the 5th Naturalistic Decision Making Conference held in Sweden, May, 2000, in press.

2
examples of the kinds of complexities that can arise in the environment, the kinds of cognitive
and collaborative strategies and facilitating activities that domain practitioners use to handle
these situations, and how existing artifacts are tailored in order to meet situation demands. These
illustrative incidents may be examples of practitioner performance in routine situations that arise
often, or they may represent a response to a relatively rare occurrence (e.g., equipment
malfunction, accident) that arises during the observational study.
Exploratory observational studies contrast to other scientific methods in that the focus of the
observations and analysis is on discovery rather than hypothesis testing. Different analysts
looking at the same domain might very well focus on different aspects and uncover different
insights if they draw on very different conceptual frameworks in selecting what is ‘interesting’ to
capture. In this type of research what matters is not 'reliability' -- would different analysts
working independently have focused on the same observations? But rather how 'generative' the
work is -- are the results insightful and productive with respect to pointing to sources of
performance problems and opportunities for improvement?
Figure 1 provides a graphic representation of the data analysis and abstraction process
used to derive generalizations from the specific observations (cf., Hollnagel et al., 1981,
Patterson and Woods, 2001). Observations and analyses are guided by (1) the questions that the
study is intended to address, (2) the sample of practitioners and activities observed and (3) the
conceptual frameworks that the observers bring to bear.
Figure 1. Conceptual frameworks and study questions that define what is ‘interesting’ and thus
guide observations and the data analysis and abstraction process.
Conceptual Frameworks
StudyQuestions
•
Distributed cognition
•
Common ground in communication
•
Supervisory control of automation
...
Data Analysis and Abstraction
R
a
w
Data:
notes, audio/video tapes:
Corpus of Illustrative Cases
Patterns Across Cases
Generalized Despriptions,
Principles, Conceptual Frameworks
•
What are cognitive and collaborative demands?
•
What strategies have developed to cope with demands?
... •
multiple domain practioners
•
multiple techniques (e.g., observation, interviews)
•
mult iple ses sions
•
mult iple ob servers

3
Conceptual frameworks play an important role in guiding observations and analyses (cf.
Lipshitz, this volume). In the studies described below observations and analyses are informed
by bodies of knowledge about the interaction of humans and supporting artifacts in complex,
socio-technical systems from the field of cognitive engineering and related behavioral and social
sciences (e.g., theory of distributed cognition, role of common ground in multi-agent
communication, principles of human automation interaction and the consequences of clumsy
automation).
These conceptual frameworks guide the identification of ‘interesting’ cases to capture. They
enable tractability in data collection, both in terms of the amount of time spent observing (on the
order of days instead months) and the level of detail of what is recorded and analyzed (sequences
of events rather than second by second behavioral and verbal interactions).
Conceptual frameworks also support the analysis process used to identify common patterns
across a corpus of illustrative cases, and to draw generalizations that have applicability beyond
the particular cases examined. These analysis stages correspond to the levels of analysis that
Lipshitz (this volume) refers to as ‘paraphrasing’ and ‘theorizing’.
It should be pointed out that conceptual frameworks not only serve as a starting point for
observation and analysis – a framework by which to interpret and aggregate findings, they are
also an output of the analysis process. The general descriptions, principles and conceptual
frameworks that emerge from the analysis of an observational study are expressed at a level of
abstraction that allow across domain comparison and application. The results of observational
studies can be used to support, expand, refine, or refute existing conceptual frameworks. They
can also be used to generate new conceptual frameworks and theories.
The differences in goals of exploratory observational studies, as contrasted with studies
designed to test a specific hypothesis, leads to different study design considerations. For
example, an important consideration in a study designed for hypothesis testing is to control the
conditions of observation and minimize variability (both with respect to the range of situations
observed and with respect to what observers record). In contrast, because the focus of
exploratory observational studies is on discovery, the objective is to broaden the set of
observations and conceptual frameworks brought to bear by observers in order to maximize the
opportunity for uncovering interesting findings and drawing productive insights.
Several techniques are used to broaden the set of observations and conceptual frameworks
brought to bear. These include:
• broadly sampling domain practice (e.g., multiple shifts, multiple practitioners at multiple
levels of experience; multiple sites)
• use of multiple converging techniques (e.g., field observations, structured interviews,
questionnaires)
• use of multiple observers who are likely to bring different conceptual frameworks
Another important concern in performing observational studies is the accuracy of the
observations and their interpretation. Several strategies are used to guard against errors and
biases. First, observers take advantage of opportunities to continuously ‘bootstrap’ their
understanding of the domain complexities, iteratively refining their notion of what is ‘interesting’
to collect and analyze (Lipshitz, this volume). Each observation provides a potential opportunity
to generate new conjectures as well as test conjectures generated in prior observations. This

4
approach allows the discovery of critical factors not predicted in advance as well as the
opportunity to discard early conjectures that are not supported by later observations. The ability
to sample the domain of practice broadly (multiple practitioners, multiple levels of domain
expertise, multiple session, multiple sites) provides an opportunity to look for commonalities
across cases as well as divergences (contrasting cases) that reveal interesting insights.
A second technique to promote accurate interpretation is to compare the insights and
perspective of multiple observers. It is common to hold meetings immediately following an
observation period where the multiple observers share their observations and interpretations.
This reduces the risk of forgetting critical details that are ‘in the head’ of the observer but not
recorded in the ‘raw data’, and improves data reliability by allowing multiple observers to
contrast their interpretations of events while their memories are still fresh and it is still possible
to pursue additional data to resolve ambiguities or differences in interpretation (e.g., by
conducting follow-up interviews with the domain practitioners that were the subject of the
observations).
The ultimate criterion in evaluating the results of an observational study is that once the
insights are made and pointed out, that other analysts (or more relevantly the domain
practitioners themselves) would agree with the findings and interpretation. A common practice is
to present the results of the study to domain practitioners (either in the form of a report or a
presentation) and solicit feedback on the accuracy of the base observations and their
interpretations.
The two studies summarized below provide concrete illustrations of this methodological
approach.
STUDY 1: INFORMING THE DESIGN OF NEW TECHNOLOGY
New technologies often fail to have the desired effects on human performance when
introduced into actual complex, socio-technical settings. This first study illustrates how a series
of observations and interviews uncovered cognitive and collaborative demands and adaptive
strategies with current communication technology that had implications for the design of a new
“data link” communication technology for train dispatching (Roth, Malsch, Multer & Coplen,
1999; Roth, Malsch & Multer, 2001).
Currently, voice radio is the primary means of communication between Railroad Dispatchers
and the railway workers they interact with (e.g., Locomotive Engineers; Maintenance of Way
Workers). The radio channels are overloaded however, creating a data overload situation for
Train Dispatchers. The railroad industry has been examining the use of data link technology to
communicate in place of or in addition to voice radio communication. The guiding questions for
this study were:
1. What activities could be supported more effectively with data link digital
communication systems?
2. What features of the existing technology are important to effective dispatcher
performance and therefore need to be considered when deploying new technology?
The study combined field observations at dispatch centers with structured interviews with
experienced dispatchers. In the first phase, Railroad Dispatchers were observed as they went
about their job in a railroad dispatch center that primarily handled passenger trains. Two
observers participated. Each observer sat next to a different railroad dispatcher and observed the
communications he or she engaged in, and the train routing and track management decisions that

5
were made. The observer asked the dispatcher questions during low workload periods.
Questions were guided by a checklist of pre-defined topics and by the observed behavior.
A total of 8 dispatchers were observed across two shifts. Observations included high
workload early morning rush-hour periods, lower workload mid-day periods, and shift turnovers.
Phase 2 consisted of structured interviews with experienced railroad dispatchers and related
personnel from the same railroad dispatch. Phase 3 involved field observations at a second
dispatch center that primarily handled freight trains. This was to assess the generality of the
results obtained at the first dispatch center. The fourth phase involved a second set of field
observations at the same dispatch center observed during Phase 1. This was to verify and expand
on the results obtained in the previous three phases. In general, the results from each phase
confirmed and extended the results from the previous phase.
Uncovering the Role of Radio ‘Party-Line’ in Facilitating Railroad Dispatching
Railroad dispatching involves extensive communication and coordination among
individuals distributed in time and space. In a typical railroad dispatch center, there are multiple
dispatchers working in parallel, each responsible for different territories, who must coordinate
with others in order to manage track usage efficiently and minimize train delays. Observations
were therefore guided by concepts from the distributed cognition, common ground, and
distributed planning literatures. The observations revealed the cognitive and collaborative
demands and the cooperative planning and error detection strategies that dispatchers have
developed with the current technology.
What makes railroad dispatching cognitively difficult is the need to deal with unplanned
demands on track usage (e.g., the need to accommodate unscheduled trains and requests for time
on the track for maintenance work), and the need for dynamic re-planning in response to
unanticipated events (e.g., train delays, track outages). The observational study revealed that
successful performance depends on the ability of dispatchers to monitor train movement beyond
their territory, anticipate delays, balance multiple demands placed on track usage, and make rapid
decisions. This requires keeping track of where trains are, whether they will reach destination
points (meets, stations) on time or will be delayed, and how long the delays will be.
To meet these demands, dispatchers have developed information-gathering strategies that
allow them to anticipate requirements for changes to schedules and planned meets early so as to
have time to take compensatory action. Many of these strategies depend on communication and
coordination among individuals distributed across time and space. This includes coordination
among dispatchers managing abutting territories within a dispatch center as well as coordination
among the various crafts within a railroad (e.g., locomotive engineers, train masters, dispatchers,
and roadway personnel).
One of the most salient findings was that railroad dispatchers took advantage of the
broadcast/’party-line’ feature of radio to anticipate and plan ahead. The ability to “listen in” on
communications directed at others that have a bearing on achievement of your own goals and to
recognize when information in your possession is of relevance to others and broadcast it, were
found to be important contributors to efficient management of track use (cf., the use of voice
loops technology in space shuttle mission control, Patterson et al., 2001).
Dispatchers routinely “listen for” information on the radio channel that is not directly
addressed to them but provides important clues to potential delays, problems or need for
assistance. As one dispatcher put it, “after a while you kind of fine tune your ear to pick up

6
certain key things.” Examples include:
• Identifying when a train has left a station: A train conductor will generally tell the
locomotive engineer “OK out of New London.” By comparing the actual departure time to
the scheduled departure, a dispatcher can calculate train delays.
• Identifying equipment problems: By overhearing conversation between a locomotive engineer
and the mechanical department, the dispatcher gets early notice of malfunctioning train
engines that will need to be replaced.
• Listening for/heading off potential interactions and conflicts: Dispatchers listen for
commitments made by others that may impact activity in their territory. The ability to listen
ahead allows dispatchers to nip potential conflicts before they arise.
• Listening for mistakes. An experienced train dispatcher will pick up key information that may
signal a misunderstanding, confusion, or error.
Implications for the Design of Data Link Technology
There are several implications for the design of data link technology from this study. First,
there was clear evidence from several observed incidents that the radio channel is now
overloaded and that there is a need to off-load some of the communication onto other media.
Data link technology provides a vehicle for taking information that is now communicated orally
and instead presenting it visually on a computer display. This has clear benefits for certain types
of information. For example having dispatchers read aloud and train crews repeat back
complicated movement authorization forms is time consuming and error prone. Transmitting the
information as a visual text or graphical display should reduce radio congestion and may reduce
the number of ‘read back’ errors and other errors of confusion and misunderstanding that
sometimes occur during verbal radio transmissions.
At the same time, the results of the observational study revealed the importance of the
“broadcast/party line” aspect of radio communication that provides a shared frame of reference
and allows dispatchers and others working on the railroad to anticipate situations and act
proactively. The study identified the need to preserve the ‘broadcast/party-line’ aspect of radio
communication when shifting to data link technology.
While data link technology is often implemented as a private communication channel where
only the specified receiver has access to the information transmitted, this is not an inherent
characteristic of the technology. It is possible to envision ‘broadcast’ versions of data link
technology where multiple individuals can access a transmitted message or view common
graphical displays regarding real time status of track and train information.
In order to explore this hypothesis under more controlled conditions, a follow-on laboratory
study was conducted. Malsch (1999) implemented two data link systems: a directed system with
no broadcasting capacity and a broadcast system. The systems were compared for their
effectiveness in a simulated railroad dispatching task with scenario elements abstracted from the
observed incidents. While both versions of data link resulted in more efficient communication as
compared to radio transmission, the broadcast version of data link produced better dispatcher
performance than the directed data link system on several measures such as train safety.
In summary, one of the most significant contributions of the study was that it revealed the
important role that the broadcast/’party-line’ feature of the radio communication media played in
facilitating safe and efficient dispatch operation in the current environment. Observed
illustrative incidents suggested that changing the design of the new data link technology in order

7
to preserve this ‘broadcast’ aspect of dispatcher communication would improve performance,
which was then confirmed in a follow-on controlled laboratory study.
STUDY 2: MAKING THE INTRODUCTION OF NEW TECHNOLOGY SAFER
The first study illustrated the use of an observational study to improve the design of a new
technology prior to implementation. Observational studies can also be used to reduce unintended
effects on performance from the introduction of new technology into a field of practice by
identifying training and operational changes that should accompany the implementation. The
next study illustrates how observation of operators in a high-fidelity simulator was used to
identify new training and operational needs prior to the implementation of advanced human-
system interfaces (HSI) in a nuclear power plant (Roth & O’Hara, 1999).
Introduction of new technology inevitably changes the nature of cognitive and collaborative
work. Some of these changes are explicitly engineered with the goal of improving performance.
However, there can also be unanticipated effects. It is easy to find examples where the
introduction of new systems have had unanticipated negative effects, creating new burdens for
practitioners, often at the busiest or most critical times (Roth, Malin, and Schreckenghost, 1997;
Woods, Johannesen, Cook and Sarter, 1994).
Woods and his colleagues (1998; Potter, Roth, Woods and Elm, 2000) have argued that new
support technologies should be regarded as hypotheses about what constitutes effective support,
and how technological change is expected to shape cognition and collaboration in the
“envisioned world” that contains the new technology (Dekker and Woods, 1999). Observational
studies provide a powerful tool for exploring the envisioned world both to evaluate the validity
of designer assumptions, and to drive further discovery and innovation. In this study, advanced
human-system interfaces (HSIs), including a computer-based procedure system, an advanced
alarm system, and a graphic-based plant information display system, were in the final phases
prior to implementation in a conventional nuclear power plant control room. Operators were
undergoing training on the use of the new interfaces on a high-fidelity full-scope simulator,
which provided an opportunity to observe the use of the technology by experienced operators
while handling plant disturbances and interview the operators immediately following the
simulation.
The guiding questions for this study were:
1) What aspects of the new HSIs were clear improvements over traditional control boards?
2) Were there any new unanticipated challenges or issues that emerged with the introduction
of the new HSIs?
The cognitive engineering literature on teamwork, the importance of shared representations
for supporting communication and coordination among team members, and the potential for new
technologies to create private ‘keyholes’ that can disrupt individual and team situation awareness
are examples of conceptual frameworks that were relevant and served to guide observations and
inquiries.
Approach
Five professional operating crews were observed and interviewed during a week of training
in a full scope, dynamic plant simulator. Each crew was unobtrusively observed during four
simulated emergency scenarios by two observers placed in an observation deck (instructor's

8
area.)
At the end of the two days of observation, the operators were interviewed in crews. The
primary purpose of the interviews was to obtain the operators’ perspective on how they used the
new HSI systems and how the new systems affected their performance as individuals and as a
team. Questions probed the perceived impact of the new systems on operator workload, situation
awareness, distribution of tasks and responsibilities among team members, and communication
and coordination among the team members.
Controlling a nuclear power plant involves dynamic, real-time communication and
coordination among individuals with dedicated roles and responsibilities. A control room crew
is typically made up of 3 individuals: a shift supervisor and two board operators, although others
augment the crew during emergencies. When there is an emergency that causes the plant to shut
down (i.e., a plant trip) in the current environment, the shift supervisor reads aloud paper-based
procedures, called Emergency Operating Procedures (EOP), that guide the crew step-by-step
through the emergency response. The board operators’ job is to read plant parameter values from
the board for the shift supervisor and take control actions as directed by the procedures that the
shift supervisor reads aloud. With the new HSI design, the parameters are automatically
provided to the shift supervisor as part of the computer-based procedure system.
Findings about Individual and Team Situation Awareness
We identified several aspects of the new HSIs that gave clear improvements, as well as had
unanticipated impacts on individual and team situation awareness. One of the most interesting
findings of the study was the impact of the HSI systems on the structure and dynamics of the
crew. The introduction of the new HSI affected the scope of responsibility of the different crew
members, the communication pattern among crew members, and the situation awareness of the
different crew members.
The new HSIs removed the need for detailed communication between the shift supervisor
and the board operator because the computer based procedure automatically provided the shift
supervisor with the plant parameter data required for him or her to work through the procedures.
The shift supervisor and board operators were able to work more in parallel. The shift supervisor
concentrated on working through the procedures and the board operators concentrated on
monitoring the advanced alarm system, graphics display, and control board HSIs. As a result, the
shift supervisor and the board operators individually reported improved situation awareness and
greater confidence in the accuracy and speed of their performance within their own locus of
responsibility.
There was an unanticipated effect, however. Operators reported that more conscious effort
was required to maintain awareness of each other's situation assessment and activities than with
the older hard-wired control board technology. While the computerized procedure reduced the
shift supervisor's overall workload, it also introduced a new demand -- the need to keep the crew
informed of his or her assessment of the situation and the status and direction of the procedural
path as he worked through the procedure. Shift supervisors reported a need to consciously
remember to inform the crew of their status through the procedure and to consciously formulate
what to communicate. The new communication requirement is a substantial cognitive task that
appeared to improve with training and experience.
Findings on the Ability To Monitor Effectiveness Of Procedures

9
Another question of interest was the impact of the new HSIs on the ability of crews to
monitor the effectiveness of the procedures in handling emergency scenarios. This included the
ability of the crews to detect and respond to cases where the actions specified in the procedures
were not fully appropriate to the specific situation. Several studies examining both actual and
simulated incidents have shown that conditions sometimes arise where response guidance in the
procedures are not fully appropriate to the situation (Kauffman, Lanik, Trager & Spence, 1992;
Roth, Mumaw & Lewis, 1994). In those cases, the ability of the crews to recognize that the
actions specified in the procedures are not fully appropriate to the specific plant conditions and to
take corrective action are important cognitive activities. As a consequence, one of the points of
focus in the present study was on how the computer-based procedures affected the operators’
ability to monitor the effectiveness of the procedures, and detect and respond to situations where
the actions specified by a procedural step were not fully appropriate to the situation.
In the study, three instances arose where the computer-based procedure provided misleading
information or directed the operators down the wrong procedural path. These instances
constitute an ‘existence proof’ of the fact that situations can arise where the procedural path taken
is not appropriate to the situation.
Given that situations can arise where the decision aid is off-track, important questions are:
(1) Can operators detect when the decision aid is off track? (2) Are they able to redirect the
decision aid and get back on track? In all three cases observed in the study, the operators were
able to correctly detect that the computer-based procedure direction was inappropriate to the
situation and overrode it. The examples illustrated important positive features of the computer-
based procedure, and raised questions about the conditions that are necessary to foster the ability
of crews to detect that a computer-based procedure is off-track and redirect it.
Implications for Training with the Introduction of the HSIs
The study suggested ways to make the introduction of the new HSIs safer through training
and operational changes. First, the new demand of supporting team situation awareness given
the elimination of “low level” communication about parameter values between the board
operator and shift supervisor can be addressed by explicit training and changes to communication
protocols to include periodic updates from the shift supervisor to the team about his or her
assessment of the situation and the location in a procedure.
Second, the three observed instances where the computer-based procedure was not
appropriate constituted an existence proof that instances where the computer-based procedure is
off-track can occur, and consequently that the task of detecting and redirecting the computer-
based procedure needs to be supported. The findings suggest the importance of having (1)
multiple diverse sources of information available to operators in the control room, and (2)
effective communication among the operators in order to detect and correct cases where the
computer-based procedure is off-track.
The ability of the operators to recognize that the actions specified were inappropriate seemed
to depend on three factors that have implications for training and operational changes:
1. accurate understanding of current plant state,
2. solid knowledge of the goals and assumptions of the procedures and the consequences
of the actions indicated by the procedure, and
3. strong communication between the shift supervisor and the board operators that
allowed the board operators to keep track of the procedural path that the shift

10
supervisor was following.
While the study provided some suggestive evidence of the kinds of factors that contribute to
the ability of crews to detect if a computer-based procedure was off-track, clearly more research
is required to fully address this issue. First, only three instances were observed and analyzed, and
these three instances might not be a representative sampling of the ways in which the computer-
based procedure could be misleading. Second, the observation that the control room crew easily
detected that the computer-based procedure was inappropriate for the situation may not
generalize to other individuals, teams, or situations, particularly since only one instance of each
situation was observed so there was no way to measure response variability. Further research is
needed to generate detailed recommendations for change and to verify that the recommendations
would have the desired effects on performance.
In summary, this study illustrates three important roles of observational studies
1. uncovering new cognitive and collaborative demands that were previously unanticipated
and could be addressed with training before the implementation of a system in an actual,
high-consequence work setting;
2. documenting illustrative cases that provide an ‘existence proof’ that certain situations can
arise that need to be explicitly considered by system designers, trainers, evaluators, and
managers; and
3. providing suggestive evidence that inform hypotheses for improving performance by
changes to training and operational procedures that can then be explored under more
controlled conditions.
DISCUSSION
In this chapter, two recent studies were used to illustrate the ways in which observational
studies can contribute to the growth of knowledge on human decision-making in complex
domains. In the first study, an important function of the current communication technology in
railroad dispatching was uncovered that had significant implications for the design of a new data
link digital communication technology. This function was “hidden” in the sense that it was an
adaptation that was not officially supported by the current technology and unlikely to have been
reported by the operators to be important. In the second study, a new demand for shift
supervisors to explicitly communicate situation assessments to a team using new advanced
displays in a nuclear power plant control room was uncovered. Because the observations were
conducted in a high-fidelity simulator prior to implementation, this new demand could be
included in training and operational changes that could be implemented at the same time as the
new system, therefore making the transition period safer. In addition, three instances where the
computer-based procedure was inappropriate to the situation were uncovered and documented.
They provided an existence proof that the situations could arise, and therefore that provisions to
support these situations and other similar situations needed to be made.
There are two phases that are important to the advancement of science. One is the
controlled experiment phase that is used to confirm a hypothesis by controlling for, and thus
eliminating, all other possible explanations for a given phenomenon. This controlled experiment
phase is generally associated with the concept of “Science”. But there is also another element of
the scientific process that is less widely discussed and that is the discovery phase. This is the
phase during which fruitful conjectures are generated that can then be tested under more

11
controlled conditions
Naturalistic observation studies are one of the tools that support this discovery phase of the
scientific process by increasing the empirical grounding of hypotheses about how tools will affect
work in complex settings. They serve to draw attention to significant phenomena and
relationships that might otherwise have been missed, and which can then be further explored in
more controlled investigations.
REFERENCES
Dekker S and Woods DD (1999). Extracting Data from the Future: Assessment and Certification
of Envisioned Systems. In S. Dekker and E. Hollnagel (Eds.), Coping with Computers in the
Cockpit. Ashgate, pp. 7-27.
De Keyser, V. (1990). Why field studies? In M. Helander, (Eds). Human Factors in Design for
Manufacturability and Process Planning, pp. 305-316. Geneva: International Ergonomics
Association.
Heath, C. and Luff, P. (2000). Technology in Action. Cambrdige, UK: Cambridge University
Press.
Hollnagel, E., Pederson, O., & Rasmussen, J. (1981). Notes on human performance analysis.
(Technical Report Riso-M-2285). Riso National Laboratory.
Kauffman, J.V., Lanik, G.F., Trager, A. and Spence, R.A. (1992). Operating Experience
Feedback Report

Human Performance in Operating Events. (NUREG-1275),
Washington, D.C.: Office for Analysis and Evaluation of Operational Data, U.S. Nuclear
Regulatory Commission.
Lipshitz, R. (2002). There is more to seeing than meets the eyeball: The art and science of
observation.
Mackay, W. (1999). Is Paper Safer? The Role of Paper Flight Strips in Air Traffic Control.
ACM Transactions on Computer-Human Interaction, Vol 6, No. 4, pp 311-340.
Malsch, N. F. (1999). Design and Testing of a Railroad Dispatching Simulator using Data-Link
Technology. Masters Thesis, Department of Aeronautics and Astronautics, Massachusetts
Institute of Technology.
Mumaw, R. J., Roth, E. M., Vicente, K. J. & Burns, C. M. (2000). There is more to monitoring a
nuclear power plant than meets the eye. Human Factors, vol 42, # 1, 36-55.
Nardi, B. A. (1997). The use of ethnographic methods in design and evaluation. In M. Helander,
T. K. Landauer & P. Prabhu, (Eds.) Handbook of Human-Computer Interaction (2nd edn.)
Amsterdam: North-Holland.
Patterson, E.S., Watts-Perotti, J., and Woods, D.D. (1999). Voice loops as coordination aids in
space shuttle mission control. Computer Supported Cooperative Work: The Journal of
Collaborative Computing, 8(4), 353-371.
Patterson, E.S., and Woods, D.D. (2001). Shift changes, updates, and the on-call model in space
shuttle mission control. Computer Supported Cooperative Work: The Journal of
Collaborative Computing, 10(3-4), 317-346.

12
Potter, S. S., Roth, E. M., Woods, D. D. & Elm, W. (2000). Bootstrapping multiple converging
cognitive task analysis techniques for system design. In J. M. Schraagen, S. F. Chipman &
V. L. Shalin (Eds.) Cognitive Task Analysis (pp. 317-340). Mahwah, NJ: Lawrence Erlbaum
Associates, Inc.
Roth, E. M., Malin, J. T. , & Schreckenghost, D. L. (1997). Paradigms for Intelligent Interface
Design. In M. Helander, T. Landauer & P. Prabhu (Eds.) Handbook of Human-Computer
Interaction (2nd edition), Amsterdam: North-Holland. (pp. 1177-1201).
Roth, E. M., Malsch, N. and Multer, J. (2001). Understanding how train dispatchers manage and
control trains: Results of a cognitive task analysis. Washington, C. C.: U. S. Department of
Transportation/Federal Railroad Administration. (DOT/FRA/ORD-01/02) Available online
at: http://www.fra.dot.gov/pdf/cta.pdf
Roth, E. M., Malsch, N., Multer, J., & Coplen, M. (1999). Understanding how train dispatchers
manage and control trains: A cognitive task analysis of a distributed planning task. . In
Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 218-
222). Santa Monica, CA: Human Factors and Ergonomics Society.
Roth, E.M., Mumaw, R.J., and Lewis, P.M., (1994). An Empirical Investigation of Operator
Performance in Cognitively Demanding Simulated Emergencies. (NUREG/CR-6208),
Washington, D.C.: U.S. Nuclear Regulatory Commission.
Roth, E. M. & O’Hara, J. (1999). Exploring the impact of advanced alarms, displays, and
computerized procedures on teams. In Proceedings of the Human Factors and Ergonomics
Society 43rd Annual Meeting (pp.158-162). Santa Monica, CA: Human Factors and
Ergonomics Society.
Vicente, K. J., Roth, E. M., Mumaw, R. J. (2001). How do operators monitor a complex,
dynamic work domain? The impact of control room technology. International Journal of
Human Computer Studies, 54, 831-856. Available online at: http://www.idealibrary.com
Woods, D. D. (1995). Process-tracing methods for the study of cognition outside the
experimental psychology laboratory. In G. A. Klein, J. Orasanu, R. Calderwood & C. E.
Zsambok (Eds.) Decision Making in Action: Models and Methods. Norwood, NJ: Ablex
Publishing Corporation.
Woods, D. D. (1998). Designs are hypotheses about how artifacts shape cognition and
collaboration. Ergonomics, 41, 168—173.
Woods, D. D., Johannesen, L. J., Cook, R. I., and Sarter, N. B. (1994). Behind Human Error:
Cognitive Systems, Computers, and Hindsight. Crew Systems Ergonomic Information and
Analysis Center (CSERIAC), Dayton, OH (State of the Art Report).