Gary Klein

ShadowBox LLC & MacroCognition LLC

Mental models are characterized as a person’s mental representation of how something works, guiding how they process information, anticipate future events, and interact with devices and tools in complex sociotechnical systems. This paper introduces the Mental Model Matrix (MMM), a novel framework for conceptualizing mental models in the context of...

Introduction The purpose of the Stakeholder Playbook is to enable the developers of explainable AI systems to take into account the different ways in which different stakeholders or role-holders need to “look inside” the AI/XAI systems. Method We conducted structured cognitive interviews with senior and mid-career professionals who had direct expe...

The development of AI systems represents a significant investment of funds and time. Assessment is necessary in order to determine whether that investment has paid off. Empirical evaluation of systems in which humans and AI systems act interdependently to accomplish tasks must provide convincing empirical evidence that the work system is learnable...

This paper summarizes the psychological insights and related design challenges that have emerged in the field of Explainable AI (XAI). This summary is organized as a set of principles, some of which have recently been instantiated in XAI research. The primary aspects of implementation to which the principles refer are the design and evaluation stag...

When people make plausibility judgments about an assertion, an event, or a piece of evidence, they are gauging whether it makes sense that the event could transpire as it did. Therefore, we can treat plausibility judgments as a part of sensemaking. In this paper, we review the research literature, presenting the different ways that plausibility has...

If a user is presented an AI system that portends to explain how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? This question entails some key concepts of measurement such as explanation goodness and trust. We present methods for enabling developers and researchers to: (1) Asses...

In recent years, there has been growing recognition of the active role of patients in healthcare, highlighting the need to support patient work by designing systems, processes, and communications that are patient-centered. Patients are called upon to make consequential decisions based on complex cognitive and perceptual cues. Some patients may even...

The development of AI systems represents a significant investment. But to realize the promise of that investment, performance assessment is necessary. Empirical evaluation of Human-AI work systems must adduce convincing empirical evidence that the work method and its AI technology are learnable, usable, and useful. The theme to this Report is the n...

This Report is a companion to the Report titled "Requirements for the Evaluation of Human-AI Work Systems." Whereas that Report focused on the minimum necessary empirical requirements for the assessment of AI systems, this Report provides additional recommendations and technical details to assist the developers of AI systems. Recommendations are pr...

The purpose of the Stakeholder Playbook is to enable system developers to take into account the different ways in which stakeholders need to "look inside" of the AI/XAI systems. Recent work on Explainable AI has mapped stakeholder categories onto explanation requirements. While most of these mappings seem reasonable, they have been largely speculat...

This report describes a Self-Explaining Scorecard for appraising the self-explanatory support capabilities of XAI systems. The Scorecard might be useful in conceptualizing the various ways in which XAI system developers are supporting users, and might also help in comparing and contrasting the various approaches.

When people make plausibility judgments about an assertion, an event, or a piece of evidence, they are gauging whether it makes sense. Therefore, we can treat plausibility judgments as sensemaking activities. In this paper, we review the research literature, presenting the different ways that plausibility has been defined and measured. Then we desc...

This material is approved for public release. Distribution is unlimited. This material is based on research sponsored by the Air Force Research Lab (AFRL) under agreement number FA8650-17-2-7711. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views...

The Cognitive Tutorial concept is based on the view that the genuine cognitive challenges to forming functional and accurate mental models of AI systems can be formalized, documented, and "trained in." Its purpose is to serve as a means of global explanation of an AI or machine learning system. A Cognitive Tutorial is created specifically to accele...

The process of explaining something to another person is more than offering a statement. Explaining means taking the perspective and knowledge of the Learner into account and determining whether the Learner is satisfied. While the nature of explanation—conceived of as a set of statements—has been explored philosophically and empirically, the proces...

The field of Explainable AI (XAI) has focused primarily on algorithms that can help explain decisions and classification and help understand whether a particular action of an AI system is justified. These \emph{XAI algorithms} provide a variety of means for answering a number of questions human users might have about an AI. However, explanation is...

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea abo...

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea abo...

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea abo...

The process of explaining something to another person is more than offering a statement. Explaining means taking the perspective and knowledge of the Learner into account, and determining whether the Learner is satisfied. While the nature of explanationconceived of as a set of statements has been explored philosophically, empirically, and experim...

This is an integrative review that address the question, "What makes for a good explanation?" with reference to AI systems. Pertinent literatures are vast. Thus, this review is necessarily selective. That said, most of the key concepts and issues are expressed in this Report. The Report encapsulates the history of computer science efforts to create...

The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We di...

This paper describes lessons we have learned about presenting cognitive skills training. We have used ShadowBox as our training approach (Klein and Borders in J Cogn Eng Decis Mak 10:268–280, 2016), but the lessons apply regardless of specific techniques employed. We analyze key takeaways and lessons learned throughout the course of multiple Shadow...

Five communities actively disparage the benefits of expertise. This chapter explains why their criticisms are misguided. Experimental psychologists have shown that linear models can outperform experts, but the factors driving these models are drawn from experts’ judgments. The Heuristics and Biases community asserts that experts are prone to flawed...

In 2016, we examined the connection between naturalistic decision making and the trend toward best practice compliance; we used evidence-based medicine (EBM) in health care as an exemplar. Paul Falzer’s lead paper in this issue describes the historical underpinnings of how and why EBM came into vogue in health care. Falzer also highlights the epist...

What makes for an explanation of “black box” AI systems such as Deep Nets? We reviewed the pertinent literatures on explanation and derived key ideas. This set the stage for our empirical inquiries, which include conceptual cognitive modeling, the analysis of a corpus of cases of "naturalistic explanation" of computational systems, computational co...

This is the fourth in a series of essays about explainable AI. Previous essays laid out the theoretical and empirical foundations. This essay focuses on Deep Nets, and con-siders methods for allowing system users to generate self-explanations. This is accomplished by exploring how the Deep Net systems perform when they are operating at their bounda...

This is the third in a series of essays about explanation. After laying out the core theoretical concepts in the first article, including aspects of causation and abduction, the second article presented some empirical research to reveal the great variety of purposes and types of causal reasoning, as well as a number of different causal explanation...

Five different scientific communities are challenging the abilities of experts and even the very concept of expertise: the decision research community, the sociology community, the heuristics and biases community, the evidence-based practices community, and the computer science community (including the fields of artificial intelligence, automation,...

As seen in Malcolm Gladwell's Blink: the modern classic on how people make decisions by drawing on prior experience and using a combination of intuition and analysis. We have all seen images of firefighters rescuing people from burning buildings and paramedics treating bombing victims. How do these individuals make the split-second decisions that s...

This is the first in a series of essays that addresses the manifest programmatic interest in developing intelligent systems that help people make good decisions in messy, complex, and uncertain circumstances by exploring several questions: What is an explanation? How do people explain things? How might intelligent systems explain their workings? Ho...

Sepsis is clinically defined as the presence of the systemic inflammatory response syndrome with a presumed source of infection. For every one-hour delay in antibiotic treatment of severe sepsis or severe shock, patient survival decreases incrementally. Little consensus on how sepsis alerts and warnings should be generated and displayed. The Shadow...

The core of the workshop would have to be an example application that called for cognitive systems engineering. We considered several different possibilities. One option was to design a kitchen. Another option was to redesign a global positioning system device for helping rental car customers navigate in unfamiliar cities. We initially selected the...

There is a growing popularity of data-driven best practices in a variety of fields. Although we applaud the impulse to replace anecdotes with evidence, it is important to appreciate some of the cognitive constraints on promulgating best practices to be used by practitioners. We use the evidence-based medicine (EBM) framework that has become popular...

Unlike behavioral skills training, cognitive skills training attempts to impart concepts that typically depend on tacit knowledge. Subject-matter experts (SMEs) often deliver cognitive training, but SMEs are expensive and in short supply, causing a training bottleneck. Recently, Hintze developed the ShadowBox method to overcome this limitation. As...

Training hour reductions for resident physicians have resulted in fewer opportunities for novices to manage critically ill patients. Our goals were (a) to understand differences in how novices and experts notice and interpret clinical cues using sepsis as an exemplar and (b) to develop simulations that replicate clinical cues to facilitate acquisit...

We trace several trajectories—the evolution of field-based decision making models in the mid-1980s to the formation of the Naturalistic Decision Making movement in 1989, then the further broadening of NDM into Macrocognition in 2003, and finally the transition from macrocognitive models into a set of methods and tools to boost cognitive performance...

When experts leave organizations due to retirement or turnover, their valuable experience, skills and knowledge often depart with them. A major challenge facing organizations across various domains is their ability to capture and transfer this expertise to the younger workforce. This becomes increasingly problematic as organizations grow and hire l...

Intelligence analysis (IA) has long been a topic of interest for the human factors community, leading to an increased understanding of the analytical process and many new insights. Yet, the scope and breadth of IA is growing and evolving for many different reasons. IA is more dynamic and multi-faceted than it has ever been. The process of IA is cha...

This paper describes using a modified version of the ShadowBox training method for capturing cognitive characteristics of expertise in the Child Protective Services (CPS) field. Researchers developed realistic CPS scenarios and performed a content analysis on participants’ qualitative responses to decisions, leading to the identification of the cha...

The Naturalistic Decision Making (NDM) community defines intuition as based on large numbers of patterns gained through experience, resulting in different forms of tacit knowledge. This view contrasts with Fast and Frugal Heuristics (FFH) researchers, who view intuition in terms of general purpose heuristics. The NDM view also differs from the Heur...

This article is a commentary, reflecting on the special section ‘Applications for naturalistic decision-making’ (Gore, Flin, Stanton, Wong, 2015, J. Occup. Organ. Psychol., doi:10.1111/joop.12121).

The objective of this project was to understand why and how some police officers and military personnel are more effective than others at managing civilian encounters without creating hostility – ‘Good Strangers’ ( GS s). We conducted cognitive task analysis ( CTA ) interviews with 17 US police officers and 24 US warfighters ( M arines and A rmy so...

We examined preferences for different forms of causal explanations for indeterminate situations. Background: Klein and Hoffman distinguished several forms of causal explanations for indeterminate, complex situations: single-cause explanations, lists of causes, and explanations that interrelate several causes. What governs our preferences for single...

Cognitive Task Analysis (CTA) has become part of the standard tool set of cognitive engineering. CTAs are routinely used to understand the cognitive and collaborative demands that contribute to performance problems, the basis of expertise, as well as the opportunities to improve performance through new forms of training, user interfaces, or decisio...

We sought to understand how some police officers and military personnel are more effective than others at increasing civilian good will following encounters. Such officers can be termed “Good Strangers” (GSs). We conducted Cognitive Task Analysis (CTA) interviews with 17 U.S. police officers and 24 warfighters (Marines and Army soldiers). The CTA i...

The purpose of this response is to clarify several of the claims made by Endsley in her focus paper. Endsley criticizes the Data/Frame model of sensemaking for misrepresenting her views on situation awareness and for failing to provide as complete an explanation as her theory. However, she has contributed to confusion about her theory of situation...

Cognitive engineering is an interdisciplinary approach to the analysis, modeling, and design of engineered systems or workplaces in which humans and technologies jointly operate to achieve system goals. As individuals, teams, and organizations become increasingly reliant on information technology and automation, it is more important than ever for s...

Cognitive engineering and decision making has become central to current human factors and ergonomics research and practice. This trend parallels exciting developments in decision making research in general. Experimental economics and judgment and decision making research boom academically and gain much public attention, while theories and economic...

One way to help trainees develop expertise is to let them see the world through the eyes of experts. However, the tasks of gaining access to the expert's cognition and then of making experts available for training are daunting and impractical. Recently, however, Hintze (2008) developed a technique to allow trainees to shadow the thinking of experts...

This meta-analysis assessed how successfully Diabetes Self-Management Education (DSME) interventions help people with type 2 diabetes achieve and maintain healthy blood glucose levels. We included 52 DSME programs with 9,631 participants that reported post-intervention A1c levels in randomized controlled trials. The training conditions resulted in...

This handbook collects and organizes contemporary cognitive engineering research, drawing on the original research of more than 60 contributing experts. Coverage of human factors, human-computer interaction, and the conceptual foundations of cognitive engineering is extensive, addressing not only cognitive engineering in broader organizations and c...

This Panel discusses decision and learning theory and application in the design of serious games to train people for bias-free critical thinking, particularly in analytical domains. The theoretical understanding of cognitive bias will determine the shape of the cognitive work that is entrained by the games. Panelists will describe an approach to bi...

Although insight is often invoked as a phenomenon of problem solving and innovation, it has rarely been studied in a naturalistic fashion. The purpose of the study reported here was to learn more about insights as they occur in field settings as opposed to controlled laboratory conditions. The authors collected a set of 120 examples of insight take...

Intelligent software algorithm are increasingly becoming a tool in consumers daily lives. Users understand the basic mechanics of the intelligent software systems they rely on, but often novices have no direct knowledge of their intelligent devices algorithm, data requirements, limitations, and representations. Problems can go beyond those caused b...

The naturalistic decision making (NDM) approach seeks to understand human cognitive performance by studying how individuals and teams choose among alternatives in real-world settings. Unlike laboratory studies, NDM research captures human interactions with actual work settings that are typically uncertain, complex, fluid, and time-pressured. The ND...

Anticipatory thinking is a critical macrocognitive function of individuals and teams. It is the ability to prepare in time for problems and opportunities. We distinguish it from prediction because anticipatory thinking is functional—people are preparing themselves for future events, not simply predicting what might happen. And it is aimed at potent...

This chapter focuses on reasoning about the causal explanation of events and human activities that are indeterminate and complex. We first consider some classical ideas about physical causation from David Hume and John Stuart Mill, who had significant impact on the psychology of reasoning, and we find in their writings some notions that carry over...

[This is an edited version of the original, unpublished 1985 study that identified recognition-primed decision making, with a new commentary added.] The objective of this study was to examine the way in which decisions are made by highly proficient personnel, under conditions of extreme time pressure, and in environments where the consequences of t...

Implicit learning involves the largely unconscious learning of dynamic statistical patterns and features, which leads to the development of tacit knowledge. This kind of learning is a ubiquitous, robust phenomenon that likely occurs in most, if not all, tasks in which individuals engage throughout their lives. In this paper, we argue that implicit...

Deliberate practice—meaning drill-like practice under the direction of a coach—is key to developing expertise in sports and music. But working professionals and businesspeople typically have no time for practice. We propose deliberate performance as a type of practice that professionals and businesspeople can pursue while they work as a way to acce...

An effort is under way to roadmap for investigations aimed at developing robust and broadly-applicable methods for “accelerated learning” (Hoffman, et al., 2009). This includes methods for: (1) Facilitating the acquisition of expertise in mission or organization-critical specializations and (2) Retaining that expertise in the form of both personnel...

Team sensemaking, in many ways, is more critical than individual sensemaking. It poses additional coordination requirements and it offers additional ways for sensemaking to break down. It is more difficult to accomplish, and it may be a larger contributor to accidents than failures at the individual level. In this article, we describe team sensemak...

For occupations such as intelligence analysis a great deal of effort has gone into developing critical thinking training and methods. These programs have a great deal of value, but they may also have drawbacks. This article discusses several problems with critical thinking programs and suggests that their emphasis on reducing mistakes needs to be b...

To develop effective programs for training people to handle commonly encountered problems, it is necessary to recognize that such problems are typically. ill defined and require additional goal specification. Most current programs have developed from information processing or from Deweyan theories of problem solving. However, neither theory has met...

This article reports on an effort to explore the differences between two approaches to intuition and expertise that are often viewed as conflicting: heuristics and biases (HB) and naturalistic decision making (NDM). Starting from the obvious fact that professional intuition is sometimes marvelous and sometimes flawed, the authors attempt to map the...

Motivation – This paper describes the initial results of a naturalistic inquiry into the way people derive causal inferences. Research approach – We examined media accounts of economic, political, military, and sports incidents to determine the types of causal explanations that are commonly invoked. Findings – We found two interacting processes at...

Motivation – Intelligent software tools face substantial obstacles to adoption, because of their complexity and lack of transparency. Research Approach – We describe the Experiential User Guide for intelligent software systems, which aims to compress critical challenges users discover along the pathway to expertise. Design– The EUG uses vignettes t...

We wish to pose accelerated learning as a challenge for intelligent systems technology. Research on intelligent tutoring systems has proved that accelerated learning is possible. The Sherlock tutor for electronics troubleshooting, for example, condensed four years of on-the-job training to approximately 25 hours, compressing the duration of the exp...

The article is intended to aid students and consumers of Cognitive Systems Engineering (CSE) in learning about CSE. A proliferation of terms describing CSE and related constructs makes it difficult to sort out differences and similarities across approaches. To aid the reader in exploring CSE, we examine areas of confusion surrounding CSE, define CS...

The article is intended to aid students and consumers of Cognitive Systems Engineering (CSE) in learning about CSE. A proliferation of terms describing CSE and related constructs makes it difficult to sort out differences and similarities across approaches. To aid the reader in exploring CSE, we examine areas of confusion surrounding CSE, define CS...

In making decisions, when should we go with our gut and when should we try to analyze every option? When should we use our intuition and when should we rely on logic and statistics? Most of us would probably agree that for important decisions, we should follow certain guidelines—gather as much information as possible, compare the options, pin down...

This article describes the origins and contributions of the naturalistic decision making (NDM) research approach. NDM research emerged in the 1980s to study how people make decisions in real-world settings. Method: The findings and methods used by NDM researchers are presented along with their implications. The NDM framework emphasizes the role of...

This publication contains reprint articles for which IEEE does not hold copyright. You may purchase this article from the Ask*IEEE Document Delivery Service at http://www.ieee.org/services/askieee/

Objective: This article describes the origins and contributions of the naturalistic decision making (NDM) research approach. Background: NDM research emerged in the 1980s to study how people make decisions in real-world settings. Method: The findings and methods used by NDM researchers are presented along with their implications. Results: The NDM f...

Cognitive systems engineering is a value-added technology offering many benefits that outweigh its costs.

The Second Forum on the Future Role of the Human in the Forecast Process occurred on 2-3 August 2005 at the American Meteorological Society's Weather Analysis and Forecasting Conference in Washington, D.C. The forum consisted of three sessions. This paper discusses the second session, featuring three presentations on the cognitive and psychological...