Content uploaded by Gary Klein
Author content
All content in this area was uploaded by Gary Klein on Jan 03, 2018
Content may be subject to copyright.
ORIGINS OF NDM
Amajor contribution of the naturalistic decision
making (NDM) community has been to describe
how people actually make decisions in real-world
settings. This statement might seem odd because
decision researchers had conducted experiments
and developed models for decades prior to the
emergence of NDM in 1989. However, that re-
search primarily identified optimal ways of making
decisions (defined as choices among alternatives)
in well-structured settings that could be carefully
controlled.
The heuristics and biases paradigm (e.g., Kahne-
man, Slovic, & Tversky, 1982) demonstrated that
people did not adhere to the principles of optimal
performance; respondents relied on heuristic as op-
posed to algorithmic strategies even when these
strategies generated systematic deviations from
optimal judgments as defined by the laws of prob-
ability, the axioms of expected utility theory, and
Bayesian statistics.
So by1989, it was fairly clear how people didn’t
make decisions. They didn’t generate alternative
options and compare them on the same set of eval-
uation dimensions. They did not generate proba-
bility and utility estimates for different courses of
action and elaborate these into decision trees. Even
when they did compare options, they rarely em-
ployed systematic evaluation techniques.
But what did they do instead? Researchers were
not likely to find out how people actually made de-
cisions by conducting experiments to test hypothe-
ses derived from statistical and mathematical
models of ideal choice strategies. Even the deci-
sion researchers who performed studies in field
settings, using experienced participants, primarily
assessed performance according to formal stan-
dards. (For a fuller discussion of this history, see
Lipshitz, Klein, Orasanu, & Salas, 2001.)
Unfortunately, the training methods and deci-
sion support systems developed in accord with the
formal standards did not improve decision qual-
ity and did not get adopted in field settings. People
found these tools and methods cumbersome and
irrelevant to the work they needed to do (Yates,
Veinott, & Patalano, 2003).
The initial NDM researchers tried a different
approach. Instead of beginning with formal mod-
els of decision making, we began by conducting
field research to try to discover the strategies peo-
ple used. Instead of looking for ways that people
were suboptimal, we wanted to find out how peo-
ple were able to make tough decisions under diffi-
cult conditions such as limited time, uncertainty,
high stakes, vague goals, and unstable conditions
Naturalistic Decision Making
Gary Klein, Klein Associates, Division of ARA, Fairborn, Ohio
Objective:This article describes the origins and contributions of the naturalistic decision
making (NDM) research approach. Background: NDM research emerged in the 1980s
to study how people make decisions in real-world settings. Method: The findings and
methods used by NDM researchers are presented along with their implications. Results:
The NDM framework emphasizes the role of experience in enabling people to rapidly
categorize situations to make effective decisions. Conclusion: The NDM focus on field
settings and its interest in complex conditions provide insights for human factors prac-
titioners about ways to improve performance. Application: The NDM approach has
been used to improve performance through revisions of military doctrine, training that
is focused on decision requirements, and the development of information technologies
to support decision making and related cognitive functions.
Address correspondence to Gary Klein, Klein Associates Division, Applied Research Associates, Inc., 1750 Commerce Center
Blvd. N., Fairborn, OH 45324; gary@decisionmaking.com. HUMAN FACTORS,Vol. 50, No. 3, June 2008, pp. 456–460. DOI
10.1518/001872008X288385. Copyright © 2008, Human Factors and Ergonomics Society.
GOLDEN ANNIVERSARY SPECIAL ISSUE
NATURALISTIC DECISION MAKING 457
(see Orasanu & Connolly, 1993). Researchers in
fields such as medicine (Elstein, Shulman, &
Sprafka, 1978) and business (Isenberg, 1984) had
already been studying these kinds of issues.
The basic research program at the Army
Research Institute for the Behavioral and Social
Sciences began funding several of the NDM re-
searchers during the mid-1980s. The U.S. Navy
became interested in naturalistic decisions follow-
ing the 1988 USS Vincennes shoot-down incident,
in which a U.S. Navy Aegis cruiser destroyed an
Iranian commercial airliner, mistaking it for a hos-
tile attacker. Both the Army and the Navy wanted
to help people make high-stakes decisions under
extreme time pressure and under dynamic and un-
certain conditions.
The first NDM conference, in 1989, assembled
researchers studying decision making in field set-
tings. In a chapter that emerged from that meeting,
Raanan Lipshitz (1993) identified no less than nine
NDM models that had been developed in parallel.
One of these was Hammond’s cognitive con-
tinuum theory (Hammond, Hamm, Grassia, &
Pearson, 1987), which asserts that decisions vary
in the degree to which they rely on intuitive and
analytical processes. Conditions such as amount
of information and time available determine where
decisions fall on this continuum and whether peo-
ple rely more on patterns or on functional rela-
tionships. Asecond account of decision making
was Rasmussen’s (1983) model of cognitive con-
trol, which distinguished skill-based, rule-based,
and knowledge-based behavior operating within
the context of a decision ladder that permitted
heuristic cutoff paths. A third, the recognition-
primed decision model (Klein, 1989), is discussed
in more detail later.
Working separately, we all reached similar con-
clusions. People were not generating and compar-
ing option sets. People were using prior experience
to rapidly categorize situations. People were rely-
ing on some kind of synthesis of their experience–
call it a schema or a prototype or a category – to
make these judgments. The categories suggested
appropriate courses of action. The static notion of
decisions as gambles, which portrays people as
passively awaiting the outcomes of their bets, did
not fit leaders who were actively trying to shape
events.
The NDM researchers studied people in field
settings, such as Navy commanders, jurors, nuclear
power plant operators, Army small unit leaders,
anesthesiologists, airline pilots, nurses, and high-
way engineers. From this perspective, making a
decision means committing oneself to a course of
action where plausible alternatives exist, even if the
person does not identify or compare these alter-
natives.
The NDM movement shifted our conception
of human decision making from a domain-
independent general approach to a knowledge-
based approach exemplified by decision makers
who had substantial experience. The decision-
making process was expanded to include a prior
stage of perception and recognition of situations,
as well as generation of appropriate responses, not
just choice from among given options. This per-
spective took advantage of advances in cognitive
psychology such as knowledge representation
concepts of scripts, schemas, and mental models,
to contrast expert versus novice behavior.
To provide a fuller account of the NDM view of
decision making, I will describe the recognition-
primed decision model; I am more familiar with
it than with the others, and it has received a fair
amount of attention. However, all of the nine NDM
models Lipshitz (1993) listed show a strong fam-
ily resemblance.
Recognition-Primed Decision Model
The recognition-primed decision (RPD) model
describes how people use their experience in the
form of a repertoire of patterns (Klein, Calder-
wood, & Clinton-Cirocco, 1986). These patterns
describe the primary causal factors operating in the
situation. The patterns highlight the most relevant
cues, provide expectancies, identify plausible
goals, and suggest typical types of reactions in that
type of situation. When people need to make a
decision they can quickly match the situation to
the patterns they have learned. If they find a clear
match, they can carry out the most typical course
of action. In that way, people can successfully
make extremely rapid decisions. The RPD model
explains how people can make good decisions
without comparing options.
However, there is more to the RPD model than
pattern matching. How can a person evaluate an
option without comparing it with others? We found
that the fireground commanders we studied eval-
uated a course of action by using mental simula-
tion to imagine how it would play out within the
context of the current situation. If it would work,
then the commanders could initiate the action. If
458 June 2008 –
Human Factors
it almost worked, they could try to adapt it or else
consider other actions that were somewhat less
typical, continuing until they found an option that
felt comfortable. This process exemplifies Herbert
Simon’s (1957) notion of satisficing – looking for
the first workable option rather than trying to find
the best possible option. Because fires grow expo-
nentially, the faster the commanders could react,
the easier their job.
Therefore, the RPD model is a blend of intuition
and analysis. The pattern matching is the intuitive
part, and the mental simulation is the conscious,
deliberate, and analytical part. This blend corre-
sponds to the System 1(fast and unconscious)/Sys-
tem 2 (slow and deliberate) account of cognition put
forward by Kahneman (2003), Epstein (1994), and
others (for an overview, see Evans, 2008). A purely
intuitive strategy relying only on pattern matching
would be too risky because sometimes the pattern
matching generates flawed options. Acompletely
deliberative and analytical strategy would be too
slow; the fires would be out of control by the time
the commanders finished deliberating.
We formulated the RPD model based on in-
depth interviews with fireground commanders
about recent and challenging incidents and found
that the percentage of RPD strategies generally
ranged from 80% to 90% (Klein, 1989) (see Fig-
ure 1). Other researchers have replicated these
findings (see Klein, 1998).
Most critically, we tested the prediction from
the RPD model that for experienced decision mak-
ers, the first option they consider is usually sat-
isfactory. Klein, Wolf, Militello, and Zsambok
(1995) found that chess players were not randomly
generating moves that they would then evaluate.
Rather, the first moves that occurred to them were
much better than would be expected by chance.
These findings support the RPD hypothesis that
the first option considered is usually satisfactory.
These results were later replicated by Johnson and
Raab (2003).
Contributions of NDM
The demands of NDM research have spurred
the development of cognitive field research and
cognitive task analysis methods, as described by
Crandall, Klein, and Hoffman (2006). These meth-
ods have contributed to the field of human fac-
tors and ergonomics by enabling practitioners to
explore the cognitive underpinnings of different
types of work.
NDM has affected Army doctrine. The current
edition of the Army Field Manual on Command
and Control (FM 101-5) includes for the first time
a section on intuitive decision making, largely in-
fluenced by research on the RPD model.
Schmitt and Klein (1999) have adapted the
RPD model to military planning guidance. Their
strategy reduces planning time without sacrific-
ing plan quality (Ross, Klein, Thunholm, Schmitt,
& Baxter, 2004) and has become the basis for tac-
tical decision making in the Swedish armed forces
(Thunholm, 2006).
The field of NDM has also provided guidance
for training decision making and related cogni-
tive skills. Cannon-Bowers and Salas (1998) have
described the range of lessons learned from the
TADMUS (Tactical Decision Making Under
Stress) project initiated by the Navy following the
USS Vincennes shoot-down decision. These in-
clude methods for providing stress inoculation
along with approaches for individual and team
decision training.
The NDM movement has seen a surprisingly
rapid adoption of its findings. Within 10 years of
the initial NDM meeting, experiential models
were accepted as the standard account of decision
making by most practitioners. NDM conferences
have been held every 2 to 3 years, alternating be-
tween the United States and Europe. In addition,
the Cognitive Engineering and Decision Making
Technical Group, formed to provide an outlet for
NDM research, has become one of the largest and
most active in the Human Factors and Ergonom-
ics Society.
Where is NDM heading? Because cognitive
field research methods have proven so effective for
generating insights about decision making, they
are being used to study other “macrocognitive”
functions, such as situation awareness, sensemak-
ing, planning and replanning, and the ways they
are linked (Klein, et al., 2003). Macrocognition, the
study of cognitive adaptations to complexity, may
reflect the next step in the evolution of NDM. Mac-
rocognitive functions are performed at the level of
individuals. These functions are also performed by
teams, as emphasized by Letsky, Warner, Fiore,
Rosen, and Salas (2007), who build on NDM re-
search on shared mental models and team knowl-
edge (e.g., Cooke, Salas, Kiekel, & Bell, 2004). The
growth of interest in macrocognition suggests that
the premises of NDM are stimulating research and
applications that cover a broader and interrelated
NATURALISTIC DECISION MAKING 459
set of cognitive functions at the team, organiza-
tional, and individual levels.
ACKNOWLEDGMENTS
I thank Robert Hoffman, Danny Kahneman,
Beth Veinott, and Raanan Lipshitz for their very
helpful comments and criticisms in reviewing an
earlier draft of this manuscript. I also appreciate
the helpful suggestions of three anonymous re-
viewers.
REFERENCES
Cannon-Bowers, J. A., & Salas, E. (1998). Making decisions under
stress: Implications for individual and team training. Washington,
DC: American Psychological Association.
Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2004). Advances in mea-
suring team cognition. In E. Salas & S. M. Fiore (Eds.), Team cogni-
tion: Understanding the factors that drive process and performance
(pp. 83–106). Washington, DC: American Psychological Association.
Crandall, B., Klein, G., & Hoffman, R. R. (2006). Working minds: A
practitioner’s guide to cognitive task analysis. Cambridge, MA:
MIT Press.
Elstein, A. S., Shulman, L. S., & Sprafka, S. A. (1978). Medical problem
solving: An analysis of clinical reasoning. Cambridge, MA: Harvard
University Press.
Epstein, S. (1994). Integration of the cognitive and psychodynamic
unconscious. American Psychologist, 49, 709–724.
Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judg-
ment and social cognition. Annual Review of Psychology, 59,
255–278.
Hammond, K. R., Hamm, R. M., Grassia, J., & Pearson, T. (1987).
Direct comparison of the efficacy of intuitive and analytical cog-
nition in expert judgment. Proceedings of IEEE Transactions on
Systems, Man, and Cybernetics, SMC-17, 753–770.
Isenberg, D. J. (1984, November–December). How senior managers
think. Harvard Business Review, pp. 80–90.
Johnson, J. G., & Raab, M. (2003). Take the first: Option generation and
resulting choices. Organizational Behavior and Human Decision
Processes, 91(2), 215–229.
Experience the Situation in a Changing Context
Is the Situation
Familiar?
Recognition has four aspects
Actions
1…n
Plausible
Goals
Relevant
Cues
Expectancies
Yes
Seek More
Information
Reassess
Situation
No
Mental Simulation
of Action (n)
Will it Work?
Implement
Yes
Yes, but
Modify
Are
Expectancies
Violated?
Yes
No
No
Figure 1. Model of recognition-primed decision making. (Decision making in action: Models and methods. G. A. Klein,
J. Orasanu, R. Calderwood, C. E. Zsambok, Editors. Copyright © 1993 by Ablex Publishing Corporation. Norwood,
NJ. Reproduced with permission of Greenwood Publishing Group, Inc., Westport, CT.)
460 June 2008 –
Human Factors
Kahneman, D. (2003). Maps of bounded rationality: Psychology for
behavioral economics. American Economic Review, American
Economic Association, 93(5), 1449–1475.
Kahneman, D., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under
uncertainty: Heuristics and biases. Cambridge, MA: Cambridge
University Press.
Klein, G. A. (1989). Recognition-primed decisions. In W. B. Rouse (Ed.),
Advances in man-machine systems research (Vol. 5, pp. 47–92).
Greenwich, CT: JAI Press.
Klein, G. A. (1998). Sources of power: How people make decisions.
Cambridge, MA: MIT Press.
Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid deci-
sion making on the fireground. Proceedings of the Human Factors
and Ergonomics Society 30th Annual Meeting, 1, 576–580.
Klein, G. A., Orasanu, J., Calderwood, R., & Zsambok, C. E. (Eds.).
(1993). Decision making in action: Models and methods.Norwood,
NJ: Ablex Publishing Corporation.
Klein, G. A., Ross, K. G., Moon, B. M., Klein, D. E., Hoffman, R. R.,
& Hollnagel, E. (2003). Macrocognition. IEEE Intelligent Systems,
18(3), 81–85.
Klein, G. A., Wolf, S., Militello, L., & Zsambok, C. (1995). Character-
istics of skilled option generation in chess. Organizational Behavior
and Human Decision Processes, 62(1), 63–69.
Letsky, M., Warner, N., Fiore, S. M., Rosen, M., & Salas, E. (2007).
Macrocognition in complex team problem solving. In 12th Inter-
national Command and Control Research and Technology Sym-
posium (ICCRTS) (CD-ROM). Washington, DC: Command and
Control Research Program (CCRP). Available from http://www.
dodccrp.org/events/12th_ICCRTS/CD/iccrts_main.html
Lipshitz, R. (1993). Converging themes in the study of decision making
in realistic settings. In G. A. Klein, J. Orasanu, R. Calderwood, &
C. E. Zsambok (Eds.), Decision making in action: Models and
methods (pp. 103–137). Norwood, NJ: Ablex.
Lipshitz, R., Klein, G., Orasanu, J., & Salas, E. (2001). Focus article:
Taking stock of naturalistic decision making. Journal of Behavioral
Decision Making, 14, 331–352.
Orasanu, J., & Connolly, T. (1993). The reinvention of decision making.
In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.),
Decision making in action: Models and methods (pp. 3–20). Nor-
wood, NJ: Ablex.
Rasmussen, J. (1983). Skill, rules and knowledge: Signals, signs, and
symbols, and other distinctions in human performance models.
IEEE Transactions on Systems, Man and Cybernetics, SMC-13(3),
257–266.
Ross, K. G., Klein, G., Thunholm, P., Schmitt, J. F., & Baxter, H. C.
(2004). The recognition-primed decision model. Military Review,
LXXIV(4), 6–10.
Schmitt, J. F., & Klein, G. (1999). How we plan. Marine Corps Gazette,
83(10), 18–26.
Simon, H. (1957). Models of man: Social and rational. New York: Wiley.
Thunholm, P. (2006). A new model for tactical mission planning for the
Swedish Armed Forces. In Proceedings of the 2006 Command and
Control Research and Technology Symposium: The state of the art
and the state of the practice (CD-ROM). Washington, DC: Com-
mand and Control Research Program (CCRP). Available from http://
www.dodccrp.org/html4/events_past.html#2006
Yates, J. F., Veinott, E. S., & Patalano, A. L. (2003). Hard decisions, bad
decisions: On decision quality and decision aiding. In S. L. Schneider
& J. Shanteau (Eds.), Emerging perspectives on judgment and deci-
sion research(pp. 13–63). New York: Cambridge University Press.
Gary Klein is the chief scientist of Klein Associates, a
group he formed in 1978. The Klein Associates Division
is now part of Applied Research Associates. Dr. Klein re-
ceived his Ph.D. in experimental psychology from the
University of Pittsburgh in 1969. He was an assistant pro-
fessor of psychology at Oakland University (1970–1974)
and worked as a research psychologist for the U.S. Air
Force (1974–1978).
Date received: October 1, 2007
Date accepted: February 6, 2008
