Conference PaperPDF Available

Naturalistic Psychological Model of Explanatory Reasoning: How people explain things to others and to themselves

  • ShadowBox LLC & MacroCognition LLC

Abstract and Figures

The process of explaining something to another person is more than offering a statement. Explaining means taking the perspective and knowledge of the Learner into account, and determining whether the Learner is satisfied. While the nature of explanationconceived of as a set of statements has been explored philosophically, empirically, and experimentally, the process of explaining, as an activity, has received less attention. We conducted a naturalistic study, looking at 74 cases of explaining. The results imply three models: local explaining about why a device (such an intelligent system) acted in a surprising way, global explaining about how a device works, and self-explaining in which people craft their own understanding. The examination of the process of explaining as it occurs in natural settings helped us to identify a number of mistaken beliefs about how explaining works.
Content may be subject to copyright.
International Conference on Naturalistic Decision Making 2019, San Francisco, California
Naturalistic Psychological Model of Explanatory
Reasoning: How People Explain Things to Others and to
Gary KLEINa, Robert HOFFMANb, and Shane MUELLERc
aMacroCognition LLC
bInstitute for Human Machine Cognition
cMichigan Technological University
The process of explaining something to another person is more than offering a statement. Explaining
means taking the perspective and knowledge of the Learner into account, and determining whether
the Learner is satisfied. While the nature of explanationconceived of as a set of statements has
been explored philosophically, empirically, and experimentally, the process of explaining, as an
activity, has received less attention.
We conducted a naturalistic study, looking at 74 cases of explaining. The results imply three
models: local explaining about why a device (such an intelligent system) acted in a surprising way,
global explaining about how a device works, and self-explaining in which people craft their own
understanding. The examination of the process of explaining as it occurs in natural settings helped
us to identify a number of mistaken beliefs about how explaining works.
Sensemaking; electronics; engineering
From time to time, we all need to help another person understand why something happened, why a machine
behaved in an unexpected manner, or even how a complex device works. When we explain, we don’t want to
fashion an explanation that is too detailed, or one that lacks the necessary detail. Our objective in the work reported
here was to explore the nature of explaining in various contexts. One of those contexts is the field of Artificial
Intelligence (AI).
The challenge of explaining how AI systems think has been a long-standing one, as seen in the pioneering
works of William Swartout (McKeown & Swartout, 1983; Swartout & Moore, 1993; William Clancey, 1983,
1987). They argued that the AI system needs to have within it an explainable model of the task, and also, a model
of the user. In contrast to current Deep Net and Machine Learning systems, the early explainable systems had easy
access to symbolic notation of knowledge, possibly making it easier to create human-meaningful accounts of the
workings of the system. Yet many of those explanation systems failed to achieve what they had promised. Today’s
AI systems use calculational mechanisms that are much more opaque and may take substantial inferencing to be
made sense of, and so the challenge is even greater.
To some extent, the same argument could be made about how humans think about the reasoning of other
humans. Nevertheless, Pearl (2018) notes that even though we have such a meager understanding of how our
minds work, and how other people think, we can still communicate with each other, learn from each other, guide
each other and motivate each other. We can dialog in a language of cause and effect. In contrast, we cannot dialog
with intelligent machines and one reason is that they do not “speak” meaningfully about cause and effect. People
communicate via a language of reasons, which is different from the AI language of variables and weights and
We believe that today’s AI explanation systems are still in the formative stage and are just starting to
tackle the challenges of explanation that proved the undoing of earlier AI systems.
In their review of the literature, Hoffman et al. (2018) found that while there was ample material on the
process of generating explanations, the process of explaining has not been much studied. Explanations are
Gary K. et al Explanatory Reasoning
typically taken to be statements and can be evaluated in terms of factors such as clarity, comprehensiveness, and
accuracy. In contrast, “explaining” is an interactive dialog that involves the Explainer and at least one Learner.
To be effective, the process of explaining needs to take the Learner's perspective into account.
As part of the DARPA Explainable Artificial Intelligence (XAI) program we conducted a naturalistic
study of what happens when a person tries to explain the reasons for a decision or action or the workings of a
device to another person. Our goal for this naturalistic study was to model how explanations are formed. Such a
model might help the researchers seeking to enhance the explainability of intelligent systems.
We identified and examined 74 examples, some as complex as the Air France 447 disaster, others much simpler,
attempting to learn about the process of explaining from these examples. A number of the examples came from
Degani's (2004) descriptions of automation failures. Other cases came from news media, and from interviews we
had conducted for other projects. A number of examples came from the Reddit web site “Explain Like I’m Five,”
which attempts to explain complex phenomena for unsophisticated audiences. [[From this set of 74, we selected
26 examples for further examination based on their richness and their judged potential for constructing a model
of explaining.]] The cases included intelligent systems (e.g., IBM's Watson playing Jeopardy, AlphaGo playing
Go), minimally intelligent systems (e.g., autopilots of commercial airlines and passenger ships, cruise controls for
automobiles), mechanical systems (e.g., ceiling fans, motel alarm clocks, blood pressure monitors), and some
decision making events that did not involve machines (e.g., Magnus Carlsen’s winning move in a chess
Types of Explaining Activity
We divided the examples into two types: local and global. Local explaining seeks to justify why specific actions
were taken or decisions were made. In contrast, global explaining involves confusion or uncertainty, usually about
how a system works. The explaining in these instances are not tethered to any particular incident.
In addition to local and global examples, we investigated a third type of explaining activity, self-explaining, where
there is no Explainer. Instead, Learners have to gather available evidence and sort it out by themselves. We make
no claim for the exhaustiveness of these categories the local/global distinction formed a part of the initial
rationale of the DARPA XAI program, and the addition of an orthogonal self-explaining category emerged
through observation of efforts of the eleven teams in the DARPA program.
People request local explanations when they want to know why something happened. Events did not go as
expected, or a machine acted up for some reason. The process is fairly straightforward a surprising event
engenders a need for something to be explained, leading to a diagnosis of this request, followed by a process of
building and then packaging the explanation in the context of the Learner’s background. We identified 28
examples of local explaining, out of our set of 74 cases, and we used them to create a generic script for the process
of explaining, as shown in Figure 1. In all of the examples we studied, the Explainer began by trying to diagnose
the reason for the inquiry. Since an assumption was violated, the Explainer tried to determine which assumption
was wrong, in order to correct it. This process is very focused and can be very brief, as opposed to presenting a
lengthy account of a system or a situation and then trying to extract the gist.
Gary K. et al Explanatory Reasoning
Figure 1 : Model of the Process of local Explaining.
One example was the Airbus A330 tragedy, in which there were 228 deaths. The airplane took off from
Rio de Janeiro, Brazil on 1 June 2009, on its way to Paris, France. However, after lifting off and climbing, the
airplane crashed into the Atlantic Ocean. Some wreckage and two bodies were recovered in a few days, but the
black boxes, the remaining bodies, and the bulk of the wreckage were not located for another two years. Why did
the airplane crash? The explanation that the aircraft stalled begs the question of why it stalled. One obvious
explanation is pilot error, but a more careful examination of the incident shows that the pilot error was itself caused
by misrepresentations about the intelligent technology that supposedly made the airplanes stall-proof. The
mechanisms that were designed to prevent stall were valid as long as the airplane sensors were working correctly,
but when they iced up (as in the incident), the airplane became vulnerable to stalling. A more detailed attempt to
explain the accident identified seven distinct causes that resulted in confusions that chained and intersected. An
event that first seemed to make no sense came into focus.
Reviewing the 28 incidents, we identified several criteria for what would count as a plausible cause for
an event.
In parallel with identifying the causes to go into a story is the process of building the story around the
causes. In building the story, plausibility comes into play. Each state transition in the chain has to plausibly follow
from the previous state. The Learner needs to imagine how he/she would make the transitions. If plausibility is
breaking down, the explanation is seen as problematic.
With more complex cases, the Explainer may shift from story building to a Causal Landscape (Klein et
al., 2018) or some other representation of a larger number of causes that operate in parallel and also intersect.
Sometimes, the Explainer will draw a diagram to show the new belief/assumption. Explaining can take other
forms, such as using a contrast to illustrate how the current situation is different from one that seems similar, or
offering an analog to get the point across.
Next, the Explainer will give some thought to packaging the explanation. The context, along with the
Learner’s characteristics, will affect tradeoffs of effort, cost and time. Stories should not be too complicated
perhaps invoking three causes or less, and no more than six transitions.
In story building there is a cognitive strain to provide appropriate complexity without being overwhelming. There
are several ways to reduce the number of causes so as to increase the Learner’s comprehension. One way to
maintain the constraint on three causes is to be selective about which causes to include, dropping the ones that are
less relevant. Another approach to keep things manageable is to lump several causes to abstract them into a
more general cause.
When is the explaining process finished? The stopping point is when the Learner arrives as a perspective
shift, as a result of modifying or replacing a belief/assumption. This perspective shift enables the Learner to
appreciate that he/she would have made the decision or recommendation in question given what was known at the
Gary K. et al Explanatory Reasoning
Successful explaining also leads to performance outcomes The Learner can now do a better job in
carrying out a task. The Learner can generate more accurate expectancies or predictions about a system or about
another person. The Learner will now be able to shift perspectives and see the tasks from the viewpoint of the
intelligent system or the other person. The Learner will do better at gauging trust especially trust in machines,
especially smart machines.
Researchers have acknowledged since at least as far back as Clancey (1983) that global understanding is an
important goal of explanation. In contrast to local explaining, which focuses on what happened during a specific
incident, global explaining is about how things work generally how a device works, how a strategy works, how
an organization works. Our model of global explaining is based on the 46 out of the 74 cases, but we focused on
19 of those cases because of the completeness of the accounts of "How does x work?" The cases were most often
expressed as questions, such as: Why do some modern elevators not stop at the next floor? Why is it so hard to
set a digital watch? Why are we often confused by ceiling fans, by airplane reading lights, by the mute button on
a TV remote, by motel telephones and clock alarms?
For example, how does a ceiling fan work (and why do we sometimes get confused about operating
them)? Ceiling fans use a simple interface that doesn’t require a screen or anything fancy just the cord and
the visual of the fan turning.
The source of confusion is that if the blades are rotating, it is easy to forget how many times you tugged
on the cord. And then you won’t know if the next tug will increase the rotation speed more or will turn it off. The
device does not display its history. All you know is whether or not it is rotating and how fast. Making it more
confusing, you don’t get instant feedback, as you would with a 3-way bulb. If you are already at the highest speed
the next tug will turn it off. But you wouldn’t know that because it continues to rotate. So you tug the cord again,
starting it up again. The delayed feedback can make it very difficult to control the fan. Therefore, the essence of
understanding the fan is to grasp the exception and the reason for it lack of displayed history and current state,
plus delayed feedback.
In many ways our account of global explaining is similar to that of local explaining. However, we found
two fundamental differences. One difference is that local explanations typically assume that the Learner is familiar
with the set-up or with the device, hence the surprise when expectations are violated. Global explanations do not
assume familiarity. Therefore, global explanations cannot focus on the violated expectation.
A second difference is that with local explaining, the Explainer seeks to diagnose the confusion, typically
zeroing in on a flaw in the Learner’s mental model. The Explainer then seeks to help the Learner revise his mental
model. For global explaining, the Explainer has no reason to believe that the Learner’s mental model is defective
and so the Explainer is not seeking to correct the Learner’s mental model — only to expand or enrich it. Figure 2
shows a script for global explaining, starting with the issue of how the device or computational system performs
a function of interest.
Gary K. et al Explanatory Reasoning
Figure 2: Model of the Process of Global Explaining.
All of the 46 cases of global explaining in our sample were triggered by ignorance and curiosity. The
Learner’s features are essentially the same as in local Explaining. The important factors are: sophistication of the
Learner’s mental model, the Learner’s goals, common ground issues, and time pressure.
Time pressure is usually less of an issue with global explaining than local explaining, which is triggered
by a need to understand and react to a surprise. In addition, the issue of situation understanding does not come up
for global Explaining because the explaining process is not tethered to a specific incident.
In our sample of 46 cases of global explaining, 37 appeared to involve some attempt to take the features
of the Learner into account and these addressed the first feature, the imagined sophistication of the Learner’s
mental model. The remaining 9 cases did not appear to imagine the Learner’s sophistication.
For global explaining, the process of diagnosis is different than for local Explaining. The Explainer is
not diagnosing the Learner’s confusion or flaws in the Learner’s beliefs. Instead, the Diagnosis process in Figure
2 is primarily about the Explainer’s speculations about what the Learner is missing. The Learner may be missing
a framework if the system is sufficiently strange. Or the Learner may be missing some of the components or some
of the links. Or the Learner may be missing the causal information that makes the story or the diagram plausible.
The Explainer’s assessment will guide the way he/she describes the working of the system.
Figure 2 introduces the concept of an Explanatory Script. The script consists of the topics most frequently
used in explaining how something works. In reviewing the 19 cases of global explaining in our corpus that we
studied in greater depth, we identified several recurring elements. These include: Components The components
of the device or computational system. Links The causal links connecting these components. Near neighbors
There often is a comparable device that can serve as an analog, and the explaining will also describe contrasts
with this near neighbor. Exceptions The situations that the device doesn’t handle well plus an account of why
they are so troublesome, such as the delayed feedback and lack of a history display with the ceiling fan. 13 of the
19 cases included an exception, often as the focus of the explanation. Tacit knowledge is often introduced here as
the types of knowledge needed to operate the device when it encounters these exceptions. We find that a preferred
format for global explaining is a diagram, rather than a story, to portray components and causal linkages. Further,
the diagram format is typically embellished with annotations in order to describe the challenges, the nearest
neighbor, the contrasts to that neighbor, and the exceptions.
Gary K. et al Explanatory Reasoning
The Learner’s features come into play to select the level of detail the Explainer uses for the elements in
the Explanatory Script. The Learner's mental model affects the level of detail most heavily, but the set of goals
that motivated the Learner to seek an explanation would also impact the level of detail provided.
What is the stopping point? The Explainer and the Learner are both seeking an outcome in which the
Learner can mentally simulate the operation of the device. Each mental simulation is essentially a story, moving
from one state to another, with plausible transitions. The Learner is trying to imagine how these transitions work.
Learners will be satisfied to stop if they feel confident that in most cases they will be able to imagine the system
outputs if they are given the system inputs.
Our third path of investigation was to examine what happens when a person engages in self-explaining.
Chi and VanLehn have conducted extensive studies of the process of self-explaining (e.g., VanLehn et al., 1990 ;
Chi & VanLehn, 1991; Chi & Wylie, 2014). One of their conclusions is that Learners retain more information and
achieve deeper understanding when they construct an explanation when they fill in details and critical steps
rather than passively receiving information. We have attempted to build on this work by applying it to self-
explanation of Machine Learning systems. For this path, we were only interested in attempts to understand the
workings of an intelligent system. We suggest that in self-explaining, the Learner begins with a frame of some
sort. It may be a story, which is a causal chain, or a visualization such as a diagram, or an analog.
With self-explaining, no one is around to diagnose the problem, the reason why Learner was surprised (for local
Explaining), or why the Learner was uncertain about how the system works (for global Explaining). Learners
have to do it themselves have to gain insight into the flaws in their mental model or the missing cues and the
missing connections, or the mistaken information, or the ways that context changes the situation.
The Learner is engaged in sensemaking. That’s what self-explaining how a device works is all about. Learners
are engaging in a sensemaking activity to build or modify a frame for understanding devices. The Learner will
typically start with some sort of frame. Often this frame will take the form of a story -- a causal chain (which
constitutes a mental model) that will let Learners achieve their goals. The causal chain is the most typical process
but there are other forms of self-explaining: creating a visualization such as a diagram, or invoking an analog plus
the contrast to describe how the analog, although similar, is still different in important ways and that
comparison/contrast serves to deepen the self-explaining.
With local Explaining, the Learner is questioning a frame that has up to that point been used to understand
a situation. The frame can take several forms, e.g., a story, a visualization, an analog. As Klein, Hoffman and
Moon (2006) suggested, one tactic is to preserve the frame by explaining away the discrepancy or modifying the
frame in minor ways. Another tactic is to re-frame, either to discard the frame in favor of a better one or else to
use the available data to construct a frame from scratch. This re-framing or frame construction process is
particularly relevant for global Explaining, as the Learner is trying to build a mental model (which can be
considered as a type of frame).
Learners employ several different types of sensemaking tactics. (a) They can manipulate the devices to
explore how they work by changing inputs and seeing how the outputs vary. Of particular value are interventions
to make the AI fail. If Learners know how the AI works then they should be able to break it. By trying to break it
they are learning how it works. This is the concept of "explanation as exploration." Learners can do this via
occlusions and masks, or by altering the training set to see the impact. Or, they can do this by entering erroneous
data to see what happens. (b) Learners can diagnose failures, either historical failures or failures they incur or
failures they deliberately manufacture. Learners can look for situations in which the AI device generated the
wrong answers, even stupid answers. Some of these are available in the literature, and sometimes the developers
provide these examples. These are the exceptions discussed earlier with regard to global Explaining. (c) Learners
can study examples and analogs for clues about how the device works; the analogs are a source of frames to try
out. Analogs include similar cases and near neighbors; Learners can draw inferences about similarities and also
search for contrasts. (d) Learners can engage in a limited dialog with a device. (e) Learners can examine the
options considered by the AI system and the evaluation method to select from the options. To support this type of
probe, AI systems can show a list of the options generated by the system and a pre-determined set of evaluation
features. Another tactic is for the AI system to show the Learner the dominant goal (from a pre-defined and limited
menu of goals) for each segment of a mission. (f) Attention. Learners may find it useful to determine what the
system is attending to. Some of the devices present heat maps (a rough analog to attention) and saliency maps. An
AI system can also provide a timeline to synchronize attention with events. Learners may get to see the
components of an image that generated a label. Or else the Learners are shown the visualization of high-
dimensional features. (g) Representations of internal structure. Many kinds of diagrams and visualizations come
into play here as support for self-explanation and for enabling the Learner to visualize the internal functionality
of the AI. These include diagrams of algorithms, of a feature matrix, of an AND/OR graph, a decision tree, a
Gary K. et al Explanatory Reasoning
causal network, modules in a reasoning chain, and tables listing features that justify a classification; (h) Verbal
labels for features. (i) Training History Learners can benefit from a description of how the system was trained.
We also offer the distinction between mindless self-explaining and meaningful self-explaining. Mindless
self-explaining might be thought of as a shotgun approach, scattering « clues » and assessing whether users find
these helpful. The tactic is to provide (mostly) true features of the AI system, leaving it to the user to make sense
of them. In contrast, meaningful self-explaining is guided by a concept of what the self-explaining should achieve.
It would be guided by the Chi and VanLehn findings about the importance of constructing an explanation as
opposed to passively receiving information. It would be guided by the work of Klein (2013) on various pathways
for making discoveries. It would be guided by the work of Klein, Borders & Besujen (2018) on mental models.
One primary goal of self-explaining is to form a better mental model of how a device works, but Klein, Borders
and Besuijen (2018) have argued that mental models have several facets not just a knowledge of how a system
works but also its limitations and the ways it can fail. Additionally, mental models of devices include tactics for
making a system work adapting it as necessary. And mental models can include a knowledge of the ways that
people can get confused and misled in operating a system. We suggest that attempts to use self-explaining to gain
better understanding of AI systems address all of these facets and not just the first one, the way the system works.
Of particularly importance would be material about how a system might fail, and the diagnoses for failures. We
suggest that attempts to foster self-explaining formulate a richer concept of what the user needs to understand,
going beyond the notion of more specifics about how the system works.
Surprise: The explaining process is triggered by a surprise, a violated expectation, as opposed to seeing
explanations as attempts to fill slots and gaps in knowledge.
Diagnosis: Diagnosis is critical on the part of the Explainer to pin down the violated expectation In contrast, other
accounts of explanations start with a complete explication of all the relevant causes and their connections, and
view the challenge as trimming details and simplifying gisting. We disagree. In our view, by diagnosing a
single flawed assumption, or a small set of flawed assumptions, the process of explaining is a very focused process
rather than trimming details from a comprehensive account.
Perspective Mismatch: How do we diagnose the reason why another person or a mechanical device took a
surprising decision? Our hypothesis is that there is a small set of possible reasons and these can help the earner
make a perspective shift. The reasons we identified are: That person/device might have different knowledge than
I do, or different goals, or might be operating under different constraints, or using different reasoning tactics
(which is especially important in dealing with AI), or is aware of different affordances than I am, or has a different
mindset, or has sized up the situation differently than I did, or has a different value system than I do.
Stopping Rule: The stopping rule for explaining is based on a perspective shift in which the Learner gains the
ability to see the situation from the vantage point of the other person or the device, so that the “surprising” event
is no longer a surprise.
Language of Reasons: Explaining relies on a language of reasons. These reasons can be causes, analogs,
contrasts, confusions, and stories. The language of reasons, of causality, is different from the language of
correlation and the strengthening/weakening of connections between layers in a neural net.
Contrasts: Stories often explain by presenting contrasts. Our literature review (Hoffman et al., 2018) turned up
papers asserting that the Learner is not simply wondering why a device recommended course of action x, but
rather, why did it recommend x as opposed to y? Our naturalistic study showed that there are other contrasts of
interest besides alternative courses of action. There can be contrasts in beliefs, in goals, in the way the situation is
Explanatory Script: We postulate an Explanatory Script which is a set of several items: components of the
system, the causal links between the components, the nearest neighbor along with contrasts to that analog, and the
Exceptions: The last component of the Explanatory Script, the exceptions, is often the richest one for explaining
how the system works or doesn’t work. These kinds of exceptions provide insight into the inner workings of a
program and serve an important function in reminding us that Machine Learning systems rely on very different
reasoning strategies than people do.
Diagrams: Global explaining typically depends on a diagram of internal structure, often with annotations, as
opposed to the story format for local Explaining.
Stopping Point: The stopping point for global Explaining is to get the Learner to be able to run a mental
simulation with standard starting conditions and be reasonably confident in the output of the mental simulation.
Gary K. et al Explanatory Reasoning
We view self-explaining as a type of sensemaking. We have also identified several sensemaking tactics that come
into play: manipulating the device, diagnosing the reasons for failures, studying examples, and studying analogs
(near neighbors). Another tactic for novice device users is to discard a mental model that assumes that the
advanced AI device thinks the way they do.
Acknowledgement and Disclaimer
This material is approved for public release. Distribution is unlimited. This material is based on research sponsored
by the Air Force Research Lab (AFRL) under agreement number FA8650-17-2-7711. The U.S. Government is
authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation
thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as
necessarily representing the official policies or endorsements, either expressed or implied, of AFRL or the U.S.
Degani, A. (2004). Taming HAL: Designing interfaces beyond 2001. New York: Palgrave/Macmillan.
Chi, M.T.H., & VanLehn, K. (1991). The content of physics self-explanations. The Journal of the Learning
Sciences, 1 (1), 69-105.
Chi, M.T.H., & Wylie, R. (2014). The ICAP framework : Linking cognitive engagement to active learning
outcomes. Educational Psychologist, 49(4), 219-243.
Clancey, W. J. (1983). The epistemology of a rule-based expert system—a framework for explanation. Artificial
Intelligence, 20(3), 215–251.
Clancey, W.J. (1987). Knowledge-based tutoring: The GUIDON program. Cambridge, MA: MIT press.
Hoffman, R.R., Klein, G., and Mueller, S.T. (2018, January). "Literature Review and Integration of Key Ideas for
Explainable AI." Report prepared by Task Area 2, DARPA XAI Program.
Klein, G. (2013). Seeing what others don’t : The remarkable ways we gain insights. New York : PublicAffairs.
Klein, G. (2018, March/April). Explaining explanation Part 3: The causal landscape. IEEE Intelligent Systems
(pp. 83-88).
Klein, G., Borders, J., & Besuijen, R. (2018). Mental models project report. Final report prepared for the Center
for Operator Performance, Dayton, Ohio.
McKeown, K.R., and Swartout, W. R. (1987). Language generation and explanation. Annual Review of Computer
Science, 2(1), 401–449.
Pearl, J. (2018). The book of why: The new science of cause and effect. New York: Basic Books.
Swartout, W. R., & Moore, J. D. (1993). Explanation in second generation expert systems. Second Generation
Expert Systems, 543, 585.
VanLehn, K., Ball, W., & Kowalski, B. (1990). Explanation-based learning of correctness : Towards a model of
the self-explanation effect. In M. Piatelli-Palmarini (Ed.) Proceedings of the Twelfth Annual Conference
of the Cognitive Science Society, Hillsdale, NJ : Erlbaum, pps. 717-724.
... In these previous studies, different levels of transparency have been determined and used, for example, global versus local explanations [24], and procedural transparency versus additional data [23]. These classifications roughly make the same distinction; there is a medium transparency level on the one hand, where only general procedures or rationales are given, and a high transparency level on the other hand, where additional specific, often numerical, information is provided. ...
... We decided to provide the regression equation as maximum transparency, as it both reveals a quantification of the effects in this particular case (through the coefficients) as well as something on the general procedure, cf. [24]. In the spirit of [44], our explanation comprises a how-explanation (opposed to for example a why-explanation), targeting competence-related trust issues of health coaches. ...
Conference Paper
Health professionals can potentially take advantage of self-tracking data of their clients. To make effective use of those data, professionals need meaningful modeling beyond data summaries. Yet, transparency of such models is an important prerequisite for trust. Paradoxically, prior research indicates that high transparency levels potentially erode trust, and we hypothesize that this effect is moderated by users' expertise levels. We conducted an online study with participants with various expertise levels in both health coaching and modeling methods. Domain experts showed higher congruency with the system when presented with high transparency over medium transparency, whereas novices showed higher levels of trust in medium transparency over high transparency. Modeling expertise did not influence trust across transparency levels. Our results emphasize the importance of tailoring the transparency level to the needs and beliefs of health professionals, to facilitate effective collaboration with health coaching support systems based on self-tracking data of their clients.
... The current interest in XAI is ostensibly and programmatically motivated by the need to make AI systems more transparent, understandable, and thus usable. However, in light of some empirically grounded findings, presented, among others, in the literature on naturalistic decision making [63,64], this interest appears to be more instrumental to the rising prevalence and diffusion of automated decision making (ADM) systems, especially when their use is anticipated in contexts for which the main legislative frameworks (e.g., the EU GDPR) require these systems to also provide reasons for their output whenever this latter can have legal effects (see, e.g., the debate around ADM and human-in-theloop decision making in the risk assessment domain [19,65,66], which uses ADM. This addresses a requirement for justification, rather than explanation, although these two concepts are often conflated (for a line of reasoning on the difference between explanation and justification, see [18]). ...
Full-text available
The emergence of black-box, subsymbolic, and statistical AI systems has motivated a rapid increase in the interest regarding explainable AI (XAI), which encompasses both inherently explainable techniques, as well as approaches to make black-box AI systems explainable to human decision makers. Rather than always making black boxes transparent, these approaches are at risk of painting the black boxes white, thus failing to provide a level of transparency that would increase the system’s usability and comprehensibility, or even at risk of generating new errors (i.e., white-box paradox). To address these usability-related issues, in this work we focus on the cognitive dimension of users’ perception of explanations and XAI systems. We investigated these perceptions in light of their relationship with users’ characteristics (e.g., expertise) through a questionnaire-based user study involved 44 cardiology residents and specialists in an AI-supported ECG reading task. Our results point to the relevance and correlation of the dimensions of trust, perceived quality of explanations, and tendency to defer the decision process to automation (i.e., technology dominance). This contribution calls for the evaluation of AI-based support systems from a human–AI interaction-oriented perspective, laying the ground for further investigation of XAI and its effects on decision making and user experience.
... The key point is that people who are involved in an interaction with an XAI system are abducting to understand what, how, and why the AI does what it does, and not just make sense of the world that is being observed or controlled (e.g., forecasting the weather is guided by the outputs of computational models). The process of sensemaking, or self-explaining a complex system, is deliberative and effortful (Chi, Roy, & Hausmann, 2008;Chi, Siler, Jewong, Yamauchi, & Hausmann, 2001;Klein et al., 2019;Klein, Hoffman, et al., 2021;Renkl & Eitel, 2019). The process of explaining often takes the form of a dialog in which an explainer and a learner collaborate, explore, and co-adapt (Clancey, 1987;Walton, 2011). ...
A challenge in building useful artificial intelligence (AI) systems is that people need to understand how they work in order to achieve appropriate trust and reliance. This has become a topic of considerable interest, manifested as a surge of research on Explainable AI (XAI). Much of the research assumes a model in which the AI automatically generates an explanation and presents it to the user, whose understanding of the explanation leads to better performance. Psychological research on explanatory reasoning shows that this is a limited model. The design of XAI systems must be fully informed by a model of cognition and a model of pedagogy, based on empirical evidence of what happens when people try to explain complex systems to other people and what happens as people try to reason out how a complex system works. In this article we discuss how and why C. S. Peirce's notion of abduction is a best model for XAI. Peirce's notion of abduction as an exploratory activity can be regarded as supported by virtue of its concordance with models of expert reasoning that have been developed by modern applied cognitive psychologists.
... The current interest in XAI is ostensibly and programmatically motivated by the need to make artificial intelligence (AI) systems more transparent, understandable, and thus usable. However, in light of some empirically-grounded findings and the literature on naturalistic decision making [38,39], this interest appears to be more instrumental to the rising prevalence and diffusion of automated decision making (ADM) systems, especially when their use is anticipated in contexts for which the main legislative frameworks (e.g., the EU GDPR) require these systems to also provide reasons of their output whenever this latter can have a legal effects. This addresses a requirement for justification, rather than explanation, although these two concepts are often conflated (for an argumentation about the difference between explanation and justification, see [14]). ...
Full-text available
The shift from symbolic AI systems to black-box, sub-symbolic, and statistical ones has motivated a rapid increase in the interest toward explainable AI (XAI), i.e. approaches to make black-box AI systems explainable to human decision makers with the aim of making these systems more acceptable and more usable tools and supports. However, we make the point that, rather than always making black boxes transparent, these approaches are at risk of \emph{painting the black boxes white}, thus failing to provide a level of transparency that would increase the system's usability and comprehensibility; or, even, at risk of generating new errors, in what we termed the \emph{white-box paradox}. To address these usability-related issues, in this work we focus on the cognitive dimension of users' perception of explanations and XAI systems. To this aim, we designed and conducted a questionnaire-based experiment by which we involved 44 cardiology residents and specialists in an AI-supported ECG reading task. In doing so, we investigated different research questions concerning the relationship between users' characteristics (e.g. expertise) and their perception of AI and XAI systems, including their trust, the perceived explanations' quality and their tendency to defer the decision process to automation (i.e. technology dominance), as well as the mutual relationships among these different dimensions. Our findings provide a contribution to the evaluation of AI-based support systems from a Human-AI interaction-oriented perspective and lay the ground for further investigation of XAI and its effects on decision making and user experience.
... To compare the simplicity of the CXAI explanations with other explanations, we examined a corpus of explanations collected from the internet, popular press, and other sources (G. Klein et al., 2019) about general topics. These explanations covered many kinds of complex systems outside of the AI domain. ...
Full-text available
An important subdomain in research on Human-Artificial Intelligence interaction is Explainable AI (XAI). XAI aims to improve human understanding and trust in machine intelligence and automation by providing users with visualizations and other information explaining the AI’s decisions, actions, or plans and thereby to establish justified trust and reliance. XAI systems have primarily used algorithmic approaches designed to generate explanations automatically that help understanding underlying information about decisions and establish justified trust and reliance, but an alternate that may augment these systems is to take advantage of the fact that user understanding of AI systems often develops through self-explanation (Mueller et al., 2021). Users attempt to piece together different sources of information and develop a clearer understanding, but these self-explanations are often lost if not shared with others. This thesis research demonstrated how this ‘Self-Explanation’ could be shared collaboratively via a system that is called collaborative XAI (CXAI). It is akin to a Social Q&A platform (Oh, 2018) such as StackExchange. A web-based system was built and evaluated formatively and via user studies. Formative evaluation will show how explanations in an XAI system, especially collaborative explanations, can be assessed based on ‘goodness criteria’ (Mueller et al., 2019). This thesis also investigated how the users performed with the explanations from this type of XAI system. Lastly, the research investigated whether the users of CXAI system are satisfied with the human-generated explanations generated in the system and check if the users can trust this type of explanation.
... Approche descriptive. L'approche descriptive de la théorie de la prise de décision observe qu'en situation naturaliste l'agent ne s'attend pas à recevoir toutes les explications ni une liste causale exhaustive [7]. Une bonne explication, pour pouvoir être le pilier d'une prise de décision, doit être sélectionnée de manière contextuelle pour son caractère contrastif, en regard d'autres éléments [12]. ...
Conference Paper
Full-text available
L’accroissement du niveau d’autonomie décision-nelle des systèmes de drones nécessite l'introduction d’algorithmes complexes qui permettent d’envisager un champ opérationnel plus large, mais dont le fonctionnement devient plus opaque pour l’opérateur en charge de la supervision. Cette étude s’appuie sur les mécanismes cognitifs sous-jacents à la prise décision humaine pour définir des outils d’amélioration de la transparence des propositions du système. Une expérimentation en micro-monde est menée pour étudier l’influence de ces différents types d’assistance sur le jugement de l’opérateur dans le cadre d’une replanification de mission de drone.
Full-text available
COGLE (COmmon Ground Learning and Explanation) is an explainable artificial intelligence (XAI) system where autonomous drones deliver supplies to field units in mountainous areas. The mission risks vary with topography, flight decisions, and mission goals. The missions engage a human plus AI team where users determine which of two AI-controlled drones is better for each mission. This paper reports on the technical approach and findings of the project and reflects on challenges that complex combinatorial problems present for users, machine learning, user studies, and the context of use for XAI systems. COGLE creates explanations in multiple modalities. Narrative “What” explanations compare what each drone does on a mission and “Why” based on drone competencies determined from experiments using counterfactuals. Visual “Where” explanations highlight risks on maps to help users to interpret flight plans. One branch of the research studied whether the explanations helped users to predict drone performance. In this branch, a model induction user study showed that post-decision explanations had only a small effect in teaching users to determine by themselves which drone is better for a mission. Subsequent reflection suggests that supporting human plus AI decision making with pre-decision explanations is a better context for benefiting from explanations on combinatorial tasks. This article is protected by copyright. All rights reserved.
Explainable AI represents an increasingly important category of systems that attempt to support human understanding and trust in machine intelligence and automation. Typical systems rely on algorithms to help understand underlying information about decisions and establish justified trust and reliance. Researchers have proposed using goodness criteria to measure the quality of explanations as a formative evaluation of an XAI system, but these criteria have not been systematically investigated in the literature. To explore this, we present a novel collaborative explanation system (CXAI) and propose several goodness criteria to evaluate the quality of its explanations. Results suggest that the explanations provided by this system are typically correct, informative, written in understandable ways, and focus on explanation of larger scale data patterns than are typically generated by algorithmic XAI systems. Implications for how these criteria may be applied to other XAI systems are discussed.
The field of Explainable AI (XAI) has focused primarily on algorithms that can help explain decisions and classification and help understand whether a particular action of an AI system is justified. These \emph{XAI algorithms} provide a variety of means for answering a number of questions human users might have about an AI. However, explanation is also supported by \emph{non-algorithms}: methods, tools, interfaces, and evaluations that might help develop or provide explanations for users, either on their own or in company with algorithmic explanations. In this article, we introduce and describe a small number of non-algorithms we have developed. These include several sets of guidelines for methodological guidance about evaluating systems, including both formative and summative evaluation (such as the self-explanation scorecard and stakeholder playbook) and several concepts for generating explanations that can augment or replace algorithmic XAI (such as the Discovery platform, Collaborative XAI, and the Cognitive Tutorial). We will introduce and review several of these example systems, and discuss how they might be useful in developing or improving algorithmic explanations, or even providing complete and useful non-algorithmic explanations of AI and ML systems.
Full-text available
Explainability is essential for users to effectively understand, trust, and manage powerful artificial intelligence applications.
Full-text available
This is the third in a series of essays about explanation. After laying out the core theoretical concepts in the first article, including aspects of causation and abduction, the second article presented some empirical research to reveal the great variety of purposes and types of causal reasoning, as well as a number of different causal explanation patterns. Taking the notion of reasoning patterns a step further, the author describes a method whereby a decision maker can go from a causal explanation to a viable course of action for making positive change in the future, not to mention aiding decision making in general.
Full-text available
Production rules are a popular representation for encoding heuristic knowledge in programs for scientific and medical problem solving. However, experience with one of these programs, mycin, indicates that the representation has serious limitations: people other than the original rule authors find it difficult to modify the rule set, and the rules are unsuitable for use in other settings, such as for application to teaching. These problems are rooted in fundamental limitations in mycin's original rule representation: the view that expert knowledge can be encoded as a uniform, weakly structured set of if/then associations is found to be wanting.To illustrate these problems, this paper examines mycin's rules from the perspective of a teacher trying to justify them and to convey a problem-solving approach. We discover that individual rules play different roles, have different kinds of justifications, and are constructed using different rationales for the ordering and choice of premise clauses. This design knowledge, consisting of structural and strategic concepts which lie outside the representation, is shown to be procedurally embedded in the rules. Moreover, because the data/hypothesis associations are themselves a proceduralized form of underlying disease models, they can only be supported by appealing to this deeper level of knowledge. Making explicit this structural, strategic and support knowledge enhances the ability to understand and modify the system.
This article describes the ICAP framework that defines cognitive engagement activities on the basis of students’ overt behaviors and proposes that engagement behaviors can be categorized and differentiated into one of four modes: Interactive, Constructive, Active, and Passive. The ICAP hypothesis predicts that as students become more engaged with the learning materials, from Passive to active to constructive to interactive, their learning will increase. We suggest possible knowledge-change processes that support the ICAP hypothesis and address the limitations and caveats of the hypothesis. In addition, empirical validation for the hypothesis is provided by examining laboratory and classroom studies that focus on three specific engagement activities: note taking, concept mapping and self-explaining. We also consider how ICAP can be used as a tool for explaining discrepant findings, dictate the proper choice of a control condition, and evaluate students’ outputs. Finally, we briefly compare ICAP to existing theories of learning.
Several earlier studies have found the amount learned while studying worked-out examples is proportional to the number of self-explanations generated while studying examples. A self-explanation is a comment about an example statement that contains domain-relevant information over and above what was stated in the example line itself. This article analyzes the specific content of self-explanations generated by students while studying physics examples. In particular, the content is analyzed into pieces of constituent knowledge that were used in the comments. These were further analyzed in order to trace the source of knowledge from which self-explanations. The first is deduction from knowledge acquired earlier while reading the text part of the chapter, usually by simply instantiating a general principle, concept, or procedure with information in the current example statements. Such construction of the content of the example statements yield new general knowledge that helps complete the students' otherwise incomplete understanding of the domain principles and concepts. The relevance of this research for instruction and models of explanation-based learning is discussed. Keywords: Learning from examples, Self explanations, Problem solving, Physics, Examples, Explanations. (JHD)
Literature Review and Integration of Key Ideas for Explainable AI
  • R R Hoffman
  • G Klein
  • S T Mueller
Hoffman, R.R., Klein, G., and Mueller, S.T. (2018, January). "Literature Review and Integration of Key Ideas for Explainable AI." Report prepared by Task Area 2, DARPA XAI Program.
Seeing what others don't : The remarkable ways we gain insights
  • G Klein
Klein, G. (2013). Seeing what others don't : The remarkable ways we gain insights. New York : PublicAffairs.
Mental models project report. Final report prepared for the Center for Operator Performance
  • G Klein
  • J Borders
  • R Besuijen
Klein, G., Borders, J., & Besuijen, R. (2018). Mental models project report. Final report prepared for the Center for Operator Performance, Dayton, Ohio.
The book of why: The new science of cause and effect
  • J Pearl
Pearl, J. (2018). The book of why: The new science of cause and effect. New York: Basic Books.