ArticlePDF Available

Why Expertise Matters: A Response to the Challenges


Abstract and Figures

Five different scientific communities are challenging the abilities of experts and even the very concept of expertise: the decision research community, the sociology community, the heuristics and biases community, the evidence-based practices community, and the computer science community (including the fields of artificial intelligence, automation, and big data). Although each of these communities has made important contributions, the challenges they pose are misguided. This essay describes the problems with each challenge and encourages researchers in the five communities to explore ways of moving forward to improve the capabilities of experts.
Content may be subject to copyright.
NOVEMB ER/DECEMBER 2017 1541-1672/17/$33.00 © 2017 IEEE 67
Published by the IEEE Computer Societ y
Editor: Robert R. Ho ffman, Institute for Human and Machine Cognition,
Why Expertise Matters:
A Response to the
Gary Klein, MacroCognition LLC
Ben Shneiderman, University of Maryland
Robert R. Hoffman and Kenneth M. Ford, Institute for Human and Machine Cognition
We dedicate this article to our colleague Robe rt Wears,
who tragically died in July 2017 just before we started
to work on this article.
Overwhelming scientic evidence demon-
strates that experts’ judgments can be highly
accurate and reliable. As dened in the scientic
literature,1 experts
employ more effective strategies than others,
and do so with less effort;
perceive meaning in patterns that others do not
form rich mental models of situations to support
sensemaking and anticipatory thinking;
have extensive and highly organized domain
knowledge; and
are intrinsically motivated to work on hard
problems that stretch their capabilities.
Our society depends on experts for mission-critical,
complex technical guidance for high-stakes decision
making because they can make decisions despite in-
complete, incorrect, and contradictory information
when established routines no longer apply.2 Experts
are the people the team turns to when faced with
difcult tasks.
Despite this empirical base, we witness a num-
ber of challenges to the concept of expertise. Tom
Nichols’ The Death of Expertise presents a strong
defense of expertise,3 a defense to which we are
adding in this article. We address the attempts
made by ve communities to diminish the credibil-
ity and value of experts (see Figure 1). These chal-
lenges come from
decision researchers who show that simple lin-
ear models can outperform expert judgment;
heuristics and biases researchers who have
claimed that experts are as biased as anyone else;
sociologists who see expertise as just a social
practice-oriented researchers seeking to replace
professional judgments with data-based pre-
scriptions and checklists; and
technophiles who believe that it is only a matter
of time before articial intelligence (AI) surpasses
Each of these communities has questioned the
value of expertise, using different arguments, per-
spectives, and paradigms.
Society needs experts, even though they are fal-
lible. Although we are expert-advocates, eager to
highlight the strengths of experts, we acknowledge
that experts are not perfect and never will be. Our
purpose is to correct the misleading claims and im-
pressions being spread by the expertise-deniers.
Then we hope to engage productively with each of
these communities to improve human performance
through better training, workows, and technology.
We begin with the challenge from the decision
research community because this body of work
can be traced back the furthest, to the mid-1960s,
and echoes to this day.
The Challenge from
the Decision Research
Research by some experimental
psychologists shows that in judg-
ment tasks, simple linear mod-
els will be more consistent in their
performance than human judges.
Examples are faculty ratings of
graduate students versus a model
based on grades and test scores, or
physicians’ ratings of cancer biopsy
results versus a model based on sur-
vival statistics:4,5
There are some, but only a few, truly re-
markable judges, whereas there are many
so-called experts who are no better than
complete novicesthe picture of the ex-
pert painted in broad brush strokes by this
research is relatively un atteringwhen-
ever possible, human judges should be re-
placed by linear models.6
A few aspects of this literature are
The linear models are derived in
the rst place from the advice of
experts about what the key vari-
ables are—the variables that ex-
perts themselves use in making
The decision research tends to re-
duce expertise to single measures,
such as judgment hit rate, ignoring
more qualitative contributions to
For many of the studies, it is not
obvious that the particular judg-
ment task that is presented to the
participants is actually the same
task that the experts routinely
perform, and therefore might
not be the task at which they are
For many of the studies, there is
scant evidence that the participants
who are called experts actually
qualify for that designation, apart
from their having had a certain
numbers of years of experience.
Advocates of this view go beyond
their empirical base by generalizing
from studies using college students
as judges to argue for the fallibility
of all judges.
Although linear models are con-
sistent, when they fail, they fail
miserably. One problem involves
“broken leg cues.” A linear model
might do a decent job of predicting
whether a given person is likely to
go to the movies this weekend, but
will fail because it is blind to the
fact that the person in question just
broke a leg.7 Experts will perform
better than the model if they have
information to which the model is
Who are the experts? In some
studies, the linear models were com-
pared to college students, sometimes
called “judges.” And even in studies
in which the judges were profession-
als, perhaps the linear models should
be compared to the best of the profes-
sionals rather than to the average.
Even when the linear models out-
performed the experts, it is a mistake
to infer that the linear models got it
right and the experts failed miserably.
The linear models had their greatest
edge in domains6 and prediction tasks
involving human activity (that is,
clinical psychologists, psychiatrists,
counselors, admissions ofcers, pa-
role ofcers, bank loan ofcers, and
so on). However, the linear models
weren’t very accurate—it was just
that the experts were even worse. In
one often-cited study of cancer di-
agnosis, the linear model performed
better than oncologists at predicting
patient longevity, but a closer look
shows that the model only accounted
for 18 percent of the variance in the
judgment data.9 The clearest conclu-
sion from this and other studies on
linear modeling is that some things
are of intrinsic low predictability.
Next is the challenge from the
heuristics and biases (HB) commu-
nity, which can be traced to the early
The Challenge from the
Heuristics and Biases
Led by Daniel Kahneman and Amos
Tver s k y,10 the HB community has
called into question assumptions
about rationality by demonstrating
that people fall prey to a wide variety
of biases in judgment and probabilis-
tic reckoning, and that even experts
sometimes show these biases. This
nding helped create a mindset that
experts are not to be trusted. The
proliferation of HB research in aca-
demic psychology departments has
strengthened the impression that ex-
pert judgments are not accurate.
The HB paradigm typically uses
participants who are not experts (col-
lege students) and gives them arti-
cial laboratory tasks that require little
training and have little or no ecologi-
cal validity. The tasks, conveniently
enough, can be performed in a col-
lege class period. Bias effects found
using this “paradigm of convenience”
Challenges to expertise
and biases
Sociology Decision
Figure 1. The ve communities that have
challenged the concept of expertise.
diminish or disappear when research-
ers add context11 or when research-
ers have genuine experts engage in
their familiar environments rather
than work on articial puzzles
and probability-juggling tasks. Vari-
ations in the materials, instructions,
procedures, or experimental design
can cause bias effects to diminish or
Although some studies have shown
that bias can occur in expert reason-
ing,13 several studies show that bias
effects are much smaller than those of
the college students.14 There is mixed
evidence for the claim that experts
tend to be overcondent, and what
evidence there is stems from narrow
methods for measuring condence.
Experts such as weather forecasters
and reghters are careful to keep
their judgments within their core spe-
cialty and to use experience and ac-
curate feedback to attain reasonable
levels of condence in their judg-
ments. Weather forecasters search for
evidence that conrms a hypothesis;
it would be irrational not to. On the
other hand, weather forecasters de-
liberately and deliberatively look for
evidence that their hypotheses might
be wrong.
The HB researchers’ antipathy to-
ward experts opened the way for
the emergence of the naturalistic
decision-making movement,15 which
regards heuristics as strengths ac-
quired through experience, rather
th an we akne sse s. Tver sky a nd
Kahneman were careful to state that,
“In general these heuristics are quite
useful, but sometimes they lead to se-
vere and systematic errors.”16 How-
ever, the HB eld usually ignores this
caveat and emphasizes the downside
of heuristics.
We now come to the challenge
from sociology, which began in the
1970s and emerged most forcefully in
the 1980s.
The Challenge from
Sociological analysis of the conse-
quences of occupational special-
ization has considered the value of
professions to society.17 Given its
close association to the concept of
professions, the concept of expertise
was also assessed from the sociologi-
cal perspective, referred to as “science
and technology studies.” Ethnogra-
phers, sociologists, and philosophers
of science researched expertise in do-
mains including astronomy, physics,
and endocrinology.18–20 Their reso-
nant paradigms have been referred
to as “situated cognition,” “distrib-
uted cognition,” and the “sociology
of scientic knowledge.”21–24 Some
individuals have dened their para-
digm, in part, as a reaction to cogni-
tive psychology:
If one relegates all of cognition to in-
ternal mental processes, then one is re-
quired to pack all the explanatory ma-
chinery of cognition into the individual
mind as well, leading to misidentica-
tion of the boundaries of the cognitive
syste m, and the over-at tribution to the
individual mind alone all of the process-
es that give rise to intelligent behavior.25
Proponents of the situated cogni-
tion approach offer many good ex-
amples of why one should dene “the
cognitive system” as persons acting
in coordination with a social group
to conduct activities using tools and
practices that have evolved within a
culture.25 The core claim is that ex-
pertise and cognition reside in the in-
teraction among the individual and
the team, community, or organiza-
tion. The strongest view is that ex-
pertise is a social attribution or role,
a matter of prestige and authority. A
moderate view is that individual cog-
nition is an enabling condition for ex-
pertise, which just happens to be a
condition that is not of particular in-
terest in a sociological analysis.
One of the valuable aspects of this
perspective is to sensitize us to the
importance of external resources and
community relationships for the ac-
quisition, expression, and valuation
of expertise. Thus, we respect these
researchers and their contributions.
The importance of context has been
recognized in cognitive psychology
for decades,26 and in computer sci-
ence as well.27 We agree completely
that resources for cognition are in
the world. We agree that teamwork
and organizational issues are an im-
portant part of naturalistic decision
making. Indeed, the notion of “mac-
rocognition”28 refers to coordinating
and maintaining common ground as
primary functions. However, cog-
nitive scientists are disappointed by
any approach that takes the strong
stance that expertise is merely a so-
cial attribution, a stance that dis-
counts the importance and value of
individual cognition, knowledge, and
There is overwhelming empirical
evidence that individual knowledge is
crucial in expert reasoning and prob-
lem solving. If you plug experts and
nonexperts into the same work set-
tings you will nd huge differences
in the quality of the outputs of the
groups/teams. The claims derived from
a dismissive reaction to cognitive-
individualist views move the pendu-
lum too far. Sociologists including
Harry Collins29 and Harald Mieg30
have taken the balanced view, which
we advocate, describing the impor-
tance of individual expertise along
with social and contextual factors
that can be essential for developing
and maintaining expertise. We re-
main hopeful that over time, this bal-
anced view will predominate.
We now come to challenges that
have emerged most recently.
The Challenge from the
Evidence-Based Practices
The evidence-based practices com-
munity argues that professionals need
to nd the best scientic evidence, de-
rive prescriptive procedures for deci-
sions, and adhere to these procedures
rather than rely on their own judg-
ments.31 This approach has been ad-
vocated in healthcare, where it is
referred to as evidence-based medi-
cine. This community argues against
trusting experts because they rely
on anecdotal practices and out-of-
date and ineffective remedies. This
takeaway message seeks to replace
reliance on experts with faith in
dened scientic studies. Clearly, em-
pirical evaluation studies have great
value, but we do not believe that such
studies deserve uncritical acceptance.
Witness how the evidence seems to
change every two years. Witness also
the difculty of sorting out the evi-
dence base for a patient with multiple
medical problems. Clinicians must
consider the individual patient, who
may differ from the criteria on which
the evidence is based. We seek a bal-
ance between scientic evidence and
broad experience.32
One way that evidence is compiled
is through checklists. These are valu-
able safety tools to prevent decision
makers from omitting important
steps in a process, but they are not
decision-support tools. We believe
that reducing complex judgments
to simple checklists often misses es-
sential aspects of decision making.33
Checklists work for stable, well-
dened tasks, and have to be carefully
crafted with a manageable number of
steps. If the checklist is sequential,
each step must lead to a clear out-
come that serves as the trigger for the
next step. However, in complex and
ambiguous situations, the anteced-
ent conditions for each step are likely
to be murky; expert decision makers
must determine when to initiate the
next step or whether to initiate it at
all. Although checklists can be help-
ful, it is risky to have individuals use
checklists for complex tasks that de-
pend on considerable tacit knowledge
to judge when a step is appropriate,
how to modify a step, and how to de-
cide whether the checklist is work-
ing.34 Experts must decide what to
do when the best practices conict
with their own judgments. They must
revise plans that do not seem to be
working. It is one thing to hold physi-
cians to task for relying on ineffective
remedies and ignoring scientic evi-
dence that the procedures that they
were once taught that have since been
shown ineffective, but it is another
thing to compel physicians to rely
on scientic evidence by procedural-
izing clinical judgment in a checklist
and penalize them for not following
the steps.
Guidelines, rules, and checklists
raise the oor by preventing silly
errors—mistakes that even a rst-
year medical student might recog-
nize as an error. But they also lower
the ceiling, making it easy to shift to
an unthinking, uncritical mode that
misses subtle warning signs and does
not serve the needs of patients.
Finally, we come to the challenge
from within computer science itself.
The Challenge from
Computer Science
This challenge has been presented
on three fronts: AI, big data, and
automation. It is claimed that these
technologies are smarter and more re-
liable than any human. Since experts
are the gold standard of performance,
demonstrations of smart technology
win big when they beat out an expert.
AI successes have been widely
publicized. IBM’s Deep Blue beat
Garry Kasparov, the reigning chess
champion at the time. IBM’s Watson
beat a panel of experts at the game of
Jeopardy. AlphaGo trounced one of
the most highly regarded Go masters.
These achievements have been inter-
preted as showing that AI can out-
perform humans at any cognitively
challenging task. But the successes in-
volve games that are well-structured,
with unambiguous referents and de-
nitive correct answers. In contrast,
most decision makers face wicked
problems with unclear goals in am-
biguous and dynamic situations.
Roger Schank, an AI pioneer, stated
atly that “Watson is a fraud.”35 He
objected to IBM’s claims that Watson
could outthink humans and nd in-
sights within large datasets. Although
Watson excels at keyword searches,
it does not consider the context of
the passages it is searching, and as
a result is insensitive to underlying
messages in the material. Schank’s
position is that counting words is not
the same as inferring insightful con-
clusions. Our experience is that AI
developers have much greater appre-
ciation for human expertise than the
AI popularizers.
A good example of the challenge
to expertise comes from the weather
forecasting domain. Articles with
titles such as “All Hail the Com-
puter!”36 promulgate the myth that
if more memory and faster process-
ing speeds could be thrown at the
problem, the need for humans would
evaporate. Starting in the late 1980s,
as more computer models were in-
troduced into operational forecast-
ing, prognostications were made
that computer models would out-
perform humans within the next 10
years—for example, “[The] human’s
advantage over the computer may
eventually be swamped by the vastly
increased number crunching ability
of the computer ... as the computer
driven models will simply get bigger
and better.37 Articles in the scientic
literature as well as the popular press
continue to present the stance of hu-
man versus machine, asking whether
“machines are taking over.36 This
stance conveys a counterproductive
attitude of competition in which the
experts cannot beat the computers.
A more productive approach would
be to design technologies that enhance
human performance. The evidence
clearly shows that the expert weather
forecaster adds value to the outputs of
the computer models. Furthermore,
“numerical prediction models do not
produce a weather forecast. They pro-
duce a form of guidance that can help
a human being decide upon a forecast
of the weather.”38,39
Next, we turn to the denigration
of expertise that has been expressed
by advocates of big data analytics.
Despite their widely publicized suc-
cesses, a closer look often tells a dif-
ferent story. For instance, Google’s
FluTrends project initially seemed
successful at predicting u outbreaks,
but over time it misled public health
planners.40 Advocates of big data
claim that the algorithms can detect
trends, spot problems, and generate
inferences and insights; no human,
no matter how expert, could pos-
sibly sift through all of the avail-
able sensor data; and no human can
hope to interpret even a fraction of
these data sources. These statements
are all true. But the big data com-
munity wants to reduce our trust in
domain experts so decision makers
become comfortable using automated
big data analyses. Here is a typical
and dangerous claim: “The big tar-
get here isn’t advertising, though. It’s
science … faced with massive data,
this approach to science—hypothesize,
model, test—is becoming obsolete
Petabytes allow us to say: Correlation
is enough. We can stop looking for
A balanced view recognizes that
big data analytics can identify pat-
terns where none exist. Big data al-
gorithms can follow historical trends
but might miss departures from these
trends, as in the broken leg cues, cues
that have implications that are clear
to experts but aren’t part of the algo-
rithms. Further, experts can use ex-
pectancies to spot missing events that
may be highly signicant. In contrast,
big data approaches, which crunch
the signals received from a variety of
sources, are unaware of the absence
of data and events.
Finally, we consider the challenge
offered by proponents of automation.
Some researchers in the automation
community have promulgated the
myth that more automation can ob-
viate the need for humans, including
experts. The enthusiasm for technol-
ogies is often extreme.42 Too many
technologists believe that automa-
tion can compensate for human lim-
itations and substitute for humans.
They also believe the myth that tasks
can be cleanly allocated to either the
human or the machine. These mis-
leading beliefs have been questioned
by cognitive systems engineers for
more than 35 years, yet the debunk-
ing has to be periodically refreshed
in the minds of researchers and pro-
gram managers.43 The misleading be-
liefs persist because of the promissory
note that more automation means
fewer people, fewer people means
fewer errors, and (especially) fewer
people means reduced costs.44
Nearly every funding program that
calls for more automation is premised
with the claim that the introduction
of automation will entail a need for
fewer expert operators at potentially
lower cost to the organization. But
the facts are in plain view: The intro-
duction often requires more experts.
Case studies45 show that automation
creates new kinds of cognitive work
for the operator, often at the wrong
times. Automation often requires
people to do more, to do it faster, or
to do it in more complex ways. The
explosion of features, options, and
modes often creates new demands,
new types of errors, and new paths
toward failure. Ironically, as these
facts became apparent, decision mak-
ers seek additional automation to
compensate for the problems trig-
gered by the automation.44
We see technology—AI, big data,
and automation—continuing to im-
prove, which will make computers
ever more valuable tools. But in the
spirit of human-centered comput-
ing, we dene intelligent systems as
human-machine systems that am-
plify and extend human abilities.46
The technology in such work systems
is designed to improve human per-
formance and accelerate the achieve-
ment of expertise. We hope that
expertise-deniers can get past the
mindset of trying to build systems to
replace the experts and instead seek
to build useful technologies that em-
power experts.
If the challenges to expertise hold
sway, the result might be degrada-
tion of the decision making and resil-
ience of organizations and agencies.
Once such organizations accept the
expertise-deniers’ arguments, they
may sideline domain experts in favor
of statistical analysts and ever more
automation. They are likely to di-
vert funding from training programs
that might produce experts into tech-
nology that makes decisions without
human intervention or responsible
action. Shifting cognitive work over
to automation may deskill workers,
erode the expertise that is crucial for
adaptability, and lead to a downward
spiral of diminishing expertise.
Experts are certainly not perfect,
so the challenges can be useful for
increasing our understanding of the
boundary conditions of expertise. We
do not want to return to an era where
medicine was governed by anecdote
rather than data—we think it essen-
tial to draw from evidence and from
expertise. We appreciate the dis-
coveries of the heuristics and biases
researchers—the heuristics they have
uncovered can have great value for fos-
tering speculative thinking. We respect
the judgment and decision research
community—we want to take advan-
tage of their efforts to improve the way
we handle evidence and deploy our
intuitions. We want to productively
move forward with improved informa-
tion technology—we want these tools
to be designed to help us gain and en-
hance expertise. We value the social
aspects of work settings—we want
to design work settings and team ar-
rangements that magnify expertise.
Our hope is to encourage a balance
that respects expertise while designing
new ways to strengthen it.
We regard the design of cognitive
work systems as the design of human-
machine interdependencies, guided by
the desire to make the machines com-
prehensible, predictable, and control-
lable. This course of action seems best
suited to promote human welfare and
enable greater achievements.47,48
We thank Bon ni e Dorr, Ha l Daume, Jon-
athan Laza r, Jim Hendler, Mark Smith,
and Jenny Preece for their comments on
a draft of this article; and Jan Maarten
Sch raagen and Paul Ward for their helpful
comments and suggestions and for their
patience and encouragement. Th is essay
was adapted from a longer and more in-
dept h account, “T he War on E xperts,”
that will appear in the Oxford H andbook
of Expertise.49
1. K.A. Ericsson et al., Cambridge
Handbook of Expertise and Expert
Performan ce, 2nd ed., Cambridge Univ.
Press, 2017.
2. B. Shneiderman and G. Klein, “Tools
That Aid Expert Decision Making:
Supporting Frontier Thinking, Social
Engagement and Responsibility,” blog,
Psycholog y Today, Mar. 2017; www /blog /seeing-what
3. T. Nichols, The Death of Expertise,
Oxford Univ. Press, 2017.
4. R. Dawes, “The Robust Beauty of
Improper Linear Models,” American
Psychologist, vol. 34, no. 7, 1979,
pp. 571–582.
5. P.E. Meehl, “Seer Over Sign: The First
Good Example,” J. Experimental
Research in Personality, vol. 1, no. 1,
1965, pp. 27–32.
6. R. Hastie and R. Dawes, Rational
Choice in an Uncertain World, Sage
Publications, 2001.
7. K. Salzinger, “Clinical, Statistical, and
Broken-Leg Predictions,” Behavior and
Philosophy, vol. 33, 2005, pp. 91–99.
8. R. Johnston, Analytic Culture in the
U.S . Intelligence Community: An Eth-
nographic Study, Center for the Study
of Intelligence, Washington, DC, 2005.
9. H.J. Einhorn and R.M. Hogarth,
“Judging Probable Cause,” Psyc hologi-
cal Bull., vol. 99, no. 1, 1978, pp. 3 –19.
10. D. Kahneman and A . Tversky, “Pros-
pect Theory: An Analysis of Decision
under Risk, Econometrica, vol. 47,
no. 2, 1979, pp. 263 –291.
11. D.W. Cheng et al., “Pragmatic Versus
Syntactic Approaches to Training De-
ductive Reasoning,” Cognitive Psychol-
ogy, vol. 18, no. 3, 1986, pp. 293–328.
12. R. Hertwig and G. Gigerenzer, “The
‘Conjunction Fallacy’ Revisited: How In-
telligent Inferences Look like Reasoning
Errors,J. Behavioral Decision Making,
vol. 12, no. 2, 1999, pp. 27–305.
13. B. Fischhoff, “Eliciting Knowledge
for Analytical Representation,” IEEE
Trans. Systems, M an, and Cybernetics,
vol. 19, no. 3, 1989, pp. 448– 461.
14. M.D. Shields, I. Solomon, and W.S.
Waller, “Effects of Alternative Sample
Space Representations on the Accuracy
of Auditors’ Uncertainty Judgments,”
Accounting, Organizations, and Soci-
ety, vol. 12, no. 4, 1987, pp. 375–385.
15. G. Klein, R. Calderwood, and A. Clinton-
Cirocco, “Rapid Decision Making on
the Fire Ground,Proc. Human Factors
and Ergonomics Soc. Ann. Meeting,
vol. 30, no. 6, 1986, pp. 576 –580.
16. A. Tversky and D. Kahneman, “Judg-
ment under Uncertainty: Heu ristics and
Biases, Science, vol. 185, Sept. 1974,
pp. 1124–1131.
17. J. Evetts, “Professionalism: Value and
Ideology,Current Sociology, vol. 61,
no. 5– 6, 2013, pp. 778 –779.
18. H.M. Collins, Changing Order.
Replication and Induction in Scientic
Practice, 2nd ed., Univ. of Chicago
Press, 1992.
19. B. Latour and S. Woolgar, Laborato ry
Life. The Social Construction of Sci-
entic Facts, Sage Publications, 1979.
20. M. Lynch, Scientic Practice and Or-
dinary Action, Cambridge Univ. Press,
21. K.D. Knorr-Cetina, The Manufacture
of Knowledge, Pergamon Press, 1981.
22. J. Lave, “Situating Learning in Com-
munities of Practice,” Perspectives on
Socially Shared Cognition, L.B. Resnick,
J.M. Levine, and S.D. Teasley, eds.,
American Psychological Assoc., 1993,
pp. 63–82.
23. L. Suchman, Plans and Situated Ac-
tions: The Problem of Human-Machine
Communication, Cambridge Univ.
Press, 1987.
24. E. Wenger, Communities of Practice:
Lear ning, Meaning, & Identity, Cam-
bridge Univ. Press, 1998.
25. M.S. Weldon, “Remembering as a
Social Process,” The Psychology of
Lear ning and Motivation, vol. 40,
no. 1, 2000, pp. 67–120.
26. G.A. Miller, “Dismembering Cog ni-
tion,” One Hundred Years of
Psychological Research in America,
Johns Hopkins Univ. Press, 1986,
pp. 277–298.
27. N.M. Agnew, K.M. Ford, and P.J.
Hayes, “Expertise in Context: Person-
ally Construc ted , Socially Selected
and Reality-Relevant?” Int’l J. E xpe rt
Systems, vol. 7, no. 1, 19 94, pp.
28. G. Klein et al., “Macrocognition,”
IEEE Intelligent Systems, vol. 18, no. 3,
2003, pp. 81–85.
29. H.M. Collins,A Sociological/
Philosophical Perspective on Expertise:
The Acquisition of Expertise Through
Socialization,Cambridge Handbook
of Expertise and Expert Performance,
2nd ed., K.A. Ericsson et al., Cambridge
Univ. Press, 2017.
30. H.A. Mieg, “Social and Sociological
Factors in the Development of Exper-
tise,” Cambridge Handbook of Ex-
pertise and Expert Performance, K.A.
Ericsson et al., Cambridge Univ. Press,
2006, pp. 743–760.
31. A.R. Roberts and K.R. Yeager, eds.,
Evidence -Based Practice Manual:
Research and Outcome Measures in
Health and Human Ser vices, Oxford
Univ. Press, 2004.
32. E. Barends, D.M. Rousseau, and R.B.
Briner, Evidence- Based Manage-
ment: The Basic Principles, Center for
Evidence-Based Management,
Amsterdam, 2014.
33. R.L. Wears and G. Klein, “The Rush
from Judgment,” Annals of Emergency
Medicine, forthcoming, 2017.
34. D.E. Klein et al., “Can We Trust Best
Practices? Six Cognitive Challenges of
Evidence-Based Approaches,” J. Cogni-
tive Eng. and Decision Making, vol. 10,
no. 3, 2016, pp. 244 –254.
35. R. Schank, “The Fraudulent Claims
Made by IBM about Watson and AI,”
36. R.A. Kerr, “Weather Forecasts Slowly
Clearing Up,” Science, vol. 38, no. 388,
2012, pp. 734–737.
37. P.S. Targett, “Predicting the Future
of the Meteorologist: A Forecaster’s
View,” Bull. Australian Meteorological
and Oceanographic Soc., vol. 7, no. 1,
1994, pp. 46 –52.
38. H.E. Brooks, C. A. Doswell, and R.A.
Maddox, “On the Use of Mesoscale
and Cloud-Scale Models in Operational
Forecasting,” Weather and Forec asting,
vol. 7, Mar. 1992, pp. 120 –132.
39. R.R. Hoffman et al., Minding the
Weather: How Expert Forecasters
Think, MIT Press, 2017.
40. D. Lazer et al., “The Parable of Google
Flu: Traps in the Big Data Analysis,”
Science, vol. 343, 14 Mar. 2014,
pp. 1203–1205.
41. C. Anderson, “The End of Theory: The
Big Data Deluge Makes the Scientic
Method Obsolete,” Wired, 23 June
2008; w
42. E. Brynjolfsson and A. McAfee, The Sec-
ond Machine Age: Work, Progress, and
Prosperity in a Time of Brilliant Tech-
nologies, W.W. Nor ton & Co., 2014.
43. J.M. Bradshaw et al., “The Seven
Deadly Myths of ‘Autonomous Sys-
tems’,” IEEE Intelligent Systems,
vol. 28, no. 3, 2013, pp. 54– 61.
44. “Technical Assessment: Autonomy,”
report from the Ofce of the Assistant
Secretary of Defense for Research and
Engineering, Ofce of Technical Intelli-
gence, US Department of Defense, 2015.
45. R.R. Hoffman, T.M. Cullen, and J.K.
Hawley, “Rhetoric and Reality of Auton-
omous Weapons: Getting a Grip on the
Myths and Costs of Automation,” Bull.
Atomic Scientists, vol. 72, no. 4, 2016;
doi:10.1080/00963402. 2016.1194619.
46. J.M. Johnson et al., “Beyond Coop-
erative Robotics: The Central Role of
Interdependence in Coactive Design,”
IEEE Intelligent Systems, vol. 26, no. 3,
2011, pp. 81–88.
47. M. Johnson et al., “Seven Cardinal Vir-
tues of Human-Machine Teamwork,”
IEEE Intelligent Systems, vol. 29, no. 6,
2014, pp. 74–79.
48. G. Klein et al., “Ten Challenges for
Making Automation a ‘Team Player’ in
Joint Human-Agent Activity,” IEEE In-
telligent Systems, vol. 19, no. 6, 2004,
pp. 91–95.
49. P. Ward et al., eds., Oxford Handbook
of Expertise, Oxford Univ. Press, forth-
Gary Klein is senior scientist at M acroCog-
nition LLC. His research interests include
naturalistic decision making. Klein received
his PhD in experimental psychology from
the University of Pittsburgh. Contact him at
Ben Shneiderman is distinguished univer-
sity professor in the Department of Com-
puter Science at the University of Maryland.
His research interests include human-com-
puter interaction, user experience design, and
information visualization. Shneiderman has
a PhD in computer science from SUNY-Stony
Brook. Contact him at
Robert R. Hoffman is a senior research
scientist at the Institute for Human and
Machine Cognition. His research interests
include macrocognition and complex cog-
nitive systems. Hoffman has PhD in exper-
imental psychology from the University of
Cincinnati. He is a fellow of the Association
for Psychological Science and the Human
Factors and Ergonomics Society and a senior
member of IEEE. Contact him at rhoffman@
Kenneth M. Ford is director of the Insti-
tute for Human and Machine Cognition.
His research interests include articial in-
telligence and human-centered computing.
Ford has a PhD in computer science from
Tulane University. Contact him at kford@
Read your subscriptions
through the myCS
publications portal at
... However, building knowledge in this way only allows for the audience to learn from the explanation agent. In fields such as fraud detection, there are also likely to be instances where experts have more knowledge than the explainer, resulting in them outperforming the system-generated explanation [116]. ...
... Whilst we look to XAI to communicate unknown patterns and influences extracted from the prescribed data, the expert audience adds breadth, supplementing the explanation with their own peripheral knowledge and undocumented experiences. Hence in expert systems, designing explanation with an interactive dialogue in mind allows for the development of a "learning loop", which ultimately enhances the performance of both the XAI agent and the audience [116]. ...
... We suggest focusing on this type of knowledge because, once again, the managerial field is diverse in terms of context, theory, methods, and contributions. If we restricted the reviews to cumulative evidence, practitioners could reject the approach for lack of storytelling (Rynes et al., 2018), contextualization (Rousseau and Gunia, 2016) and decision support (Klein et al., 2017). ...
... Another important criticism is the manual nature of some parts of the process. This criticism seems particularly relevant at a time where practice appears to privilege computational approaches such as big data and analytics (Klein et al., 2017). Even though we already embrace automation in the construction of the ontology, a question that arises is whether conceptual frameworks and decision tables can also be automated or semi-automated. ...
Full-text available
Evidence-based management (EBMgt) concerns making complex managerial decisions by combining the decision maker’s knowledge and experience with scientific knowledge. A major obstacle to this practice is the lack of support. In this paper, we employ design science to develop an ensemble of conceptual, information and IT artefacts that support EBMgt. The approach is instantiated and validated in a case study using a combination of methods, including proof of concept, laboratory experiment, satisfaction survey, and focus groups. The proposed approach turns EBMgt into a repeatable practice, supported by reusable tools, and brings some degree of automation to the process. To our knowledge, EBMgt services have not yet been realised in a concrete manner, and not with concrete empirical results. Therefore, this study provides a first-of-a-kind demonstration and assessment of the viability of EBMgt services.
... Thus, unlike jurors or laypeople, who rarely need to rely on meta-information, judges can be expected to have considerable expertise in weighing information in the context of their profession. Experts' judgments are often thought of as superior, as experts seem able to resort to more efficient task-specific strategies compared to other people (Klein, Shneiderman, Hoffman & Ford, 2017). By extension, judges may be especially able to make judgments and decisions while remaining unaffected by erroneous evidence. ...
Full-text available
Previous studies have shown that people are truth-biased in that they tend to believe the information they receive, even if it is clearly flagged as false. The truth bias has been recently proposed to be an instance of meta-cognitive myopia, that is, of a generalized human insensitivity towards the quality and correctness of the information available in the environment. In two studies we tested whether meta-cognitive myopia and the ensuing truth bias may operate in a courtroom setting. Based on a well-established paradigm in the truth-bias literature, we asked mock jurors (Study 1) and professional judges (Study 2) to read two crime reports containing aggravating or mitigating information that was explicitly flagged as false. Our findings suggest that jurors and judges are truth-biased, as their decisions and memory about the cases were affected by the false information. We discuss the implications of the potential operation of the truth bias in the courtroom, in the light of the literature on inadmissible and discredible evidence, and make some policy suggestions.
... Second, assessing cognitive load across officer experience levels provides insight into the differences by which less-experienced and experienced officers mentally process situations involving UoF. Compared to less-experienced individuals, experienced individuals tend to make decisions that are more efficient, effective and accurate (Klein et al., 2017). However, research directly comparing UoF decision-making among less-experienced and experienced officers has been limited. ...
Purpose Few studies have tested the efficacy of instruction based on cognitive load theory in police use-of-force (UoF) training due to limitations of existing cognitive load measures. Although linguistic measures of cognitive load address these limitations, they have yet to be applied to police UoF training. This study aims to discuss the aforementioned issue. Design/methodology/approach Officers’ verbal behavioral data from two UoF de-escalation projects were used to calculate cognitive load and assess how it varied with officer experience level (less-experienced, experienced). The verbal data were further analyzed to examine specific thinking patterns that contributed to heightened cognitive load across officer experience levels. Findings Across both studies, responses from less-experienced officers contained greater usage of cognitive language than responses from experienced officers. Specific cognitive processes that contribute to cognitive load in specific situations were also identified. Originality/value This paper enables police trainers to facilitate the development of adaptive training strategies to improve police UoF training via the reduction of cognitive load, and also contributes to the collective understanding of how less-experienced and experienced officers differ in their UoF decision-making.
... For the solution of this kind of sequential optimization decisionmaking problem with delayed return, this paper adopts the Q-learning algorithm. Iteratively computes the state-action value function Qðs, aÞ of the Markov decision process [19]. ...
Full-text available
This paper takes the intelligent engineering as an example and uses the Work Breakdown Structure (WBS) tool to decompose the intelligent engineering into specific subsystems and describe each subsystem. In the field of intelligent engineering construction, it is impossible to ensure that the construction project will not be changed. To reduce scope changes, efforts can be made in three aspects: first, to accurately locate project requirements and find enough work to be done; second, to scientifically define the scope of the project and delete unnecessary work; and third, to effectively control scope changes. Ensure that all work is done to achieve the project goals. This paper starts from the actual project management experience, conducts a comprehensive summary and refinement, proposes key strategies and methods for project scope management, and puts forward the following points of view. Intelligent design should be completed by professional design agencies, and intelligent construction should be completed by professional construction teams. There should be standards for intelligent acceptance to follow. Hope this article will be helpful for project managers.
... Expertise and expert performance have become important components in research assessing police use-of-force decision-making. The literature across a wide variety of domains-including law enforcement-indicates that judgment and decision-making from more experienced individuals tends to be more effective, efficient, and accurate than less experienced individuals in a given domain (Ericsson et al. 1993;Klein et al. 2017;Larrick and Feiler 2015). For instance, police officers with more experience select more appropriate courses of action than less experienced officers, evaluate scenarios differently, and attend to different information (Andersen and Gustafsberg 2016;Di Nota and Huhta 2019;Johnson et al. 2014;Suss and Ward 2018;Vickers and Lewinski 2012;Ward et al. 2011). ...
Given the vast amount of evidence showing the substantial influence of emotion on decision-making, we examined emotionality—a person’s emotional reactivity to a stimulus—in police use-of-force decision-making between a sample of expert (n = 42) and novice (n = 36) officers. Officers observed body-worn camera footage and described the course of action they would take, the kinds of information they paid attention to, and their assessment of the situation. Natural language processing techniques were used to detect measures of valence, arousal, and dominance from officer responses. Linear mixed-effects models indicated that responses from experts were more positively valenced, more dominant, and less arousing compared with responses from novices. In other words, the ability to react and assess situations calmly, with a greater sense of control, and less negatively seems to be linked with the production of accurate, effective, and efficient behaviors that mark expert policing. These results shed light on the ways in which expert and novice officers understand and experience stimuli involved in use-of-force decision-making. Practical implications for use-of-force training and future directions are discussed.
Full-text available
Providing decision makers with more information is often expected to result in more informed and superior decisions. This is especially true when leveraging artificial intelligence (AI) to explore and find complex patterns in vast amounts of data. Although AI can enable an “information advantage,” truly intelligent systems should buffer scarce human cognitive resources from information overload and be well adapted to the environment in which they are deployed. Paradoxically, some practitioners have conflated AI's information processing superiority with a contradictory decision-support goal: to provide human decision makers with more , higher quality, or more novel courses of action, regardless of context, than they could generate without AI. In this article, I review the evidence examining the costs and benefits of providing decision makers with more or less choice and identify the factors that moderate the relationship between the amount of choice and decision effectiveness. Although providing more information and choice increases confidence and certainty in one's decision, it can make decision making more difficult, decrease satisfaction, and result in poorer decision outcomes. The research indicates that such negative effects are influenced by the level of entropy and variety provided and can be reduced with increased familiarity but are further compounded when decisions are increasingly effortful, difficult, or complex. The review concludes with guidance on how designers might leverage knowledge of choice overload and associated moderator effects to create more adaptive and effective decision support systems.
Full-text available
An article published in 2018 found that statistical analysis of a single data set by 29 teams of expert analysts led to considerable variation in results, due to differences in analytic choices, but that the variation could not be explained by differences in statistical expertise. The main idea investigated was that statistical software expertise might constitute a functional predictor of software choice, and consequently, result variation, due to algorithmic differences among different statistical software packages. Unfortunately, even though 82.35% of survey respondents indicated that software expertise was indeed the primary reason for their software choice, this does not constitute sufficient evidence for the stated assumption. Subsequent, more rigorous research is required. FRENCH: Un article publié en 2018 a révélé que l'analyse statistique d'un seul jeu de données par 29 équipes d'analystes experts entraînait une variation considérable des résultats obtenus, due à des différences entre leurs choix analytiques, mais que cette variation ne pouvait être expliquée par des différences entre leur niveau d'expertise statistique. L'idée principale investiguée était que le niveau d'expertise d'utilisation de logiciels statistiques pourrait constituer un prédicteur fonctionnel du choix du logiciel et, par conséquent, de la variation des résultats, en raison de différences algorithmiques entre différents logiciels statistiques. Malheureusement, même si 82,35% des répondants au sondage effectué ont indiqué que leur niveau d'expertise logicielle était en effet la raison principale de leur choix, cela ne constitue pas une preuve suffisante pour la supposition énoncée. Des recherches subséquentes, plus rigoureuses sont nécessaires.
Professional judgment is derived from a person’s intuition, training, and level of expertise. When exploring the influence that expertise has on the process of tree risk assessment, it is helpful to approach the topic in relation to its impact across various related fields and disciplines. This paper reviews the effects of arboricultural and tree risk assessment training on the assessor and overall tree risk assessment methodology through the lens of professional judgment and decision making. Additionally, the topic of risk perception is explored based on how it can affect decision making. Concepts and theories related to risk perception are applied to arboriculture and tree risk assessment to provide additional insight into how subjectivity and personal bias may affect recommendations, mitigation, and the overall management our urban forests. The review finds that an individual’s perception of a risk can be equally as influential as the reality of the risk on the decision-making process, recommendations, and subsequent outcomes of an assessment. Furthermore, experts, similar to novices, are susceptible to the influence of perceived risk. Much of the available research has suggested that the acquisition of professional expertise (i.e., previous experience, training, and accreditation) can result in decision making that is more closely tied to the reality of a risk. Ultimately, a great deal remains unknown regarding our understanding of professional expertise and its influence on the tree risk assessment process.
This chapter provides the reader with a framing of the book’s content. First, a brief overview of expertise is offered using definitions and scholarship that are continually referenced throughout the book. It also situates those constructs within the context of work and organizations. Next, each chapter is introduced to provide readers with an overview of how the authors of each chapter frame expertise within a particular domain or how they address a particular question related to expertise that is relevant to organizations.
Full-text available
There is a growing popularity of data-driven best practices in a variety of fields. Although we applaud the impulse to replace anecdotes with evidence, it is important to appreciate some of the cognitive constraints on promulgating best practices to be used by practitioners. We use the evidence-based medicine (EBM) framework that has become popular in health care to raise questions about whether the approach is consistent with how people actually make decisions to manage patient safety. We examine six potential disconnects and suggest ways to strengthen best practices strategies.
Full-text available
This article plays counterpoint to our previous discussions of the “seven deadly myths” of autonomous systems. The seven deadly myths are common design misconceptions to be acknowledged and avoided for the ills they breed. Here, we present seven design principles to be understood and embraced for the virtues they engender. The cardinal virtues of classical antiquity that were adopted in Christian tradition included justice, prudence, temperance, and fortitude (courage). As we’ll show in this essay, in effective human-machine teamwork we can also see virtues at play—namely clarity, humility, resilience, beneficence (helpfulness), cohesiveness, integrity, and thrift. As we unfold the principles that enable these virtues to emerge, it will become clear that fully integrating them into the design of intelligent systems requires the participation of a broad range of stakeholders who aren’t always included in such discussions, including workers, engineers, operators, and strategic visionaries developing research roadmaps. The principles aren’t merely for the consumption of specialists in human factors or ergonomics. We illustrate these principles and their resultant virtues by drawing on lessons learned in the US Defense Advanced Research Projects Agency (DARPA) Robotics Challenge (DRC).
This book argues that the human cognition system is the least understood, yet probably most important, component of forecasting accuracy. Minding the Weather investigates how people acquire massive and highly organized knowledge and develop the reasoning skills and strategies that enable them to achieve the highest levels of performance. The authors consider such topics as the forecasting workplace; atmospheric scientists’ descriptions of their reasoning strategies; the nature of expertise; forecaster knowledge, perceptual skills, and reasoning; and expert systems designed to imitate forecaster reasoning. Drawing on research in cognitive science, meteorology, and computer science, the authors argue that forecasting involves an interdependence of humans and technologies. Human expertise will always be necessary.
A primary motivation for developing autonomous weapons is the assumption that such systems require fewer combatants with lower levels of expertise, thus cutting costs. For more than two decades, the Defense Department has employed highly automated weapons, including the Patriot air defense system and the Predator aircraft system. For the myths of automation to hold, the human costs of the systems should have fallen as the Army and Air Force became more accustomed to them. But automated weapon systems have actually required more highly trained and experienced experts, and the failure to train personnel to operate the systems adequately has led to fratricide and the loss of innocent life. The Defense Department needs to improve its policies on the procurement and development of automated systems if the weapons are to be responsive and effective in future conflicts.
Accurate prediction of behavior is a critical task for the psychologist, particularly for the practitioner. Outstanding among those who have successfully wrestled with this complicated task is Paul Meehl. Yet, small has been the influence of his work on the everyday practice of prediction. The object of this paper is to review Meehl's work in this area and, using behavior analysis, to seek an understanding of practitioners' continued opposition to his findings.
Many decisions are based on beliefs concerning the likelihood of uncertain events such as the outcome of an election, the guilt of a defendant, or the future value of the dollar. Occasionally, beliefs concerning uncertain events are expressed in numerical form as odds or subjective probabilities. In general, the heuristics are quite useful, but sometimes they lead to severe and systematic errors. The subjective assessment of probability resembles the subjective assessment of physical quantities such as distance or size. These judgments are all based on data of limited validity, which are processed according to heuristic rules. However, the reliance on this rule leads to systematic errors in the estimation of distance. This chapter describes three heuristics that are employed in making judgments under uncertainty. The first is representativeness, which is usually employed when people are asked to judge the probability that an object or event belongs to a class or event. The second is the availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development, and the third is adjustment from an anchor, which is usually employed in numerical prediction when a relevant value is available.