No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Principles of Explanation in Human-AI Systems
Abstract
Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea about explainability. These systems are often not tested to determine whether the algorithm helps users accomplish any goals, and so their explainability remains unproven. We propose an alternative: to start with human-focused principles for the design, testing, and implementation of XAI systems, and implement algorithms to serve that purpose. In this paper, we review some of the basic concepts that have been used for user-centered XAI systems over the past 40 years of research. Based on these, we describe the "Self-Explanation Scorecard", which can help developers understand how they can empower users by enabling self-explanation. Finally, we present a set of empirically-grounded, user-centered design principles that may guide developers to create successful explainable systems.
... Another milestone in the development of XAI is the turn toward evaluation metrics for explanations (Mueller et al. 2021). The XAI community now acknowledges more in depth that it is not enough to generate explanations, but that it is also crucial to evaluate how good they are with respect to some formalized measure. ...
... They present a taxonomy of XAI that links with computer science and HCI communities, with a structured and dense collection of many related reviews. Very recent work on XAI metrics is provided by Müeller et al. in their 2021 paper (Mueller et al. 2021). They collect concrete and practical design principles for XAI in human-machine-systems. ...
... Graphs as of e.g. Mueller et al. (2019Mueller et al. ( , 2021; Linardatos et al. (2021); ...
In the meantime, a wide variety of terminologies, motivations, approaches, and evaluation criteria have been developed within the research field of explainable artificial intelligence (XAI). With the amount of XAI methods vastly growing, a taxonomy of methods is needed by researchers as well as practitioners: To grasp the breadth of the topic, compare methods, and to select the right XAI method based on traits required by a specific use-case context. Many taxonomies for XAI methods of varying level of detail and depth can be found in the literature. While they often have a different focus, they also exhibit many points of overlap. This paper unifies these efforts and provides a complete taxonomy of XAI methods with respect to notions present in the current state of research. In a structured literature analysis and meta-study, we identified and reviewed more than 50 of the most cited and current surveys on XAI methods, metrics, and method traits. After summarizing them in a survey of surveys, we merge terminologies and concepts of the articles into a unified structured taxonomy. Single concepts therein are illustrated by more than 50 diverse selected example methods in total, which we categorize accordingly. The taxonomy may serve both beginners, researchers, and practitioners as a reference and wide-ranging overview of XAI method traits and aspects. Hence, it provides foundations for targeted, use-case-oriented, and context-sensitive future research.
... He emphasizes that explanations are contrastive, social, and selected in a biased manner and also that causal relations are more influential than probabilities. Mueller et al. [42] claim that there is a necessity for human-inspired XAI guidelines, as psychological principles often remain underestimated. Hoffman et al. [31] assert that explanations are not properties of statements, but result from interactions. ...
... For instance, Miller has done incipient work in establishing criteria to evaluate XAI, by deriving principles from social sciences [41]. Moreover, Mueller et al. [42] provided an exhaustive list of principles that emerged within XAI literature. Within the epistemological domain, Hempel [28,30] introduced the principle of factuality, namely that the "explanans" and the "explanandum" must be true. ...
Despite explainable AI (XAI) has recently become a hot topic and several different approaches have been developed, there is still a widespread belief that it lacks a convincing unifying foundation. On the other hand, over the past centuries, the very concept of explanation has been the subject of extensive philosophical analysis in an attempt to address the fundamental question of “why” in the context of scientific law. However, this discussion has rarely been connected with XAI. This paper tries to fill in this gap and aims to explore the concept of explanation in AI through an epistemological lens. By comparing the historical development of both the philosophy of science and AI, an intriguing picture emerges. Specifically, we show that a gradual progression has independently occurred in both domains from logical-deductive to statistical models of explanation, thereby experiencing in both cases a paradigm shift from deterministic to nondeterministic and probabilistic causality. Interestingly, we also notice that similar concepts have independently emerged in both realms such as, for example, the relation between explanation and understanding and the importance of pragmatic factors. Our study aims to be the first step towards understanding the philosophical underpinnings of the notion of explanation in AI, and we hope that our findings will shed some fresh light on the elusive nature of XAI.
... He emphasizes that explanations are contrastive, social, and selected in a biased manner and also that causal relations are more influential than probabilities. Mueller et al. [13] claim that there is a necessity for human-inspired XAI guidelines, as psychological principles often remain underestimated. Hoffman et al. [14] assert that explanations are not properties of statements, but result from interactions. ...
... For instance, Miller has done incipient work in establishing criteria to evaluate XAI, by deriving principles from social sciences [7]. Moreover, Mueller et al. [13] provided an exhaustive list of principles that emerged within XAI literature. Within the epistemological domain, Hempel [43], [35] introduced the principle of factuality, namely that the "explanans" and the "explanandum" must be true. ...
Despite explainable AI (XAI) has recently become a hot topic and several different approaches have been developed, there is still a widespread belief that it lacks a convincing unifying foundation. On the other hand, over the past centuries, the very concept of explanation has been the subject of extensive philosophical analysis in an attempt to address the fundamental question of "why" in the context of scientific law. However, this discussion has rarely been connected with XAI. This paper tries to fill in this gap and aims to explore the concept of explanation in AI through an epistemological lens. By comparing the historical development of both the philosophy of science and AI, an intriguing picture emerges. Specifically, we show that a gradual progression has independently occurred in both domains from logical-deductive to statistical models of explanation, thereby experiencing in both cases a paradigm shift from deterministic to nondeterministic and probabilistic causality. Interestingly, we also notice that similar concepts have independently emerged in both realms such as, for example, the relation between explanation and understanding and the importance of pragmatic factors. Our study aims to be the first step towards understanding the philosophical underpinnings of the notion of explanation in AI, and we hope that our findings will shed some fresh light on the elusive nature of XAI.
... The provision of a global explanation does not conflict with the provision of a local explanation. In contrast, local and global approaches can serve to reinforce one another [32]. The following design principle addresses Requirement 2 and 3. ...
The need to derive explanations from machine learning (ML)-based AI systems has been addressed in recent research due to the opaqueness of their processing.However, a significant amount of productive AI systems are not based on ML but are expert systems including strong opaqueness.A resulting lack of understanding causes massive inefficiencies in business processes that involve opaque expert systems. This work uses recent research interest in explainable AI (XAI) to generate knowledge for the design of explanations in constraint-based expert systems.Following the Design Science Research paradigm, we develop design requirements and design principles. Subsequently, we design an artifact and evaluate the artifact in two experiments. We observe the following phenomena. First, global explanations in a textual format were well-received. Second, abstract local explanations improved comprehensibility. Third, contrastive explanations successfully assisted in the resolution of contradictions. Finally, a local tree-based explanation was perceived as challenging to understand.
Researchers focusing on how artificial intelligence (AI) methods explain their decisions often discuss controversies and limitations. Some even assert that most publications offer little to no valuable contributions. In this article, we substantiate the claim that explainable AI (XAI) is in trouble by describing and illustrating four problems: the disagreements on the scope of XAI, the lack of definitional cohesion, precision, and adoption, the issues with motivations for XAI research, and limited and inconsistent evaluations. As we delve into their potential underlying sources, our analysis finds these problems seem to originate from AI researchers succumbing to the pitfalls of interdisciplinarity or from insufficient scientific rigor. Analyzing these potential factors, we discuss the literature at times coming across unexplored research questions. Hoping to alleviate existing problems, we make recommendations on precautions against the challenges of interdisciplinarity and propose directions in support of scientific rigor.
As Artificial Intelligence (AI) systems increase in capability, so there are growing concerns over the ways in which the recommendations they provide can affect people's everyday life and decisions. The field of Explainable AI (XAI) aims to address such concerns but there is often a neglect of the human in this process. We present a formal definition of human-centred XAI and illustrate the application of this formalism to the design of a user interface. The user interface supports users in indicating their preferences relevant to a situation and to compare their preferences with those of a computer recommendation system. A user trial is conducted to evaluate the resulting user interface. From the user trial, we believe that users are able to appreciate how their preferences can influence computer recommendations, and how these might contrast with the preferences used by the computer. We provide guidelines of implementing human-centred XAI.
The development of AI systems represents a significant investment of funds and time. Assessment is necessary in order to determine whether that investment has paid off. Empirical evaluation of systems in which humans and AI systems act interdependently to accomplish tasks must provide convincing empirical evidence that the work system is learnable and that the technology is usable and useful. We argue that the assessment of human–AI (HAI) systems must be effective but must also be efficient. Bench testing of a prototype of an HAI system cannot require extensive series of large‐scale experiments with complex designs. Some of the constraints that are imposed in traditional laboratory research just are not appropriate for the empirical evaluation of HAI systems. We present requirements for avoiding “unnecessary rigor.” They cover study design, research methods, statistical analyses, and online experimentation. These should be applicable to all research intended to evaluate the effectiveness of HAI systems.
In this article, we suggest that the study of social interactions and the development of a “sense of agency” in joint action can help determine the content of relevant explanations to be implemented in artificial systems to make them “explainable.” The introduction of automated systems, and more broadly of Artificial Intelligence (AI), into many domains has profoundly changed the nature of human activity, as well as the subjective experience that agents have of their own actions and their consequences – an experience that is commonly referred to as sense of agency. We propose to examine the empirical evidence supporting this impact of automation on individuals’ sense of agency, and hence on measures as diverse as operator performance, system explicability and acceptability. Because of some of its key characteristics, AI occupies a special status in the artificial systems landscape. We suggest that this status prompts us to reconsider human–AI interactions in the light of human–human relations. We approach the study of joint actions in human social interactions to deduce what key features are necessary for the development of a reliable sense of agency in a social context and suggest that such framework can help define what constitutes a good explanation. Finally, we propose possible directions to improve human–AI interactions and, in particular, to restore the sense of agency of human operators, improve their confidence in the decisions made by artificial agents, and increase the acceptability of such agents.
The success of deep image classification networks has been met with enthusiasm and investment from both the academic community and industry. We suggest that human users of these systems will expect AI image classifiers to understand visual concepts similarly to humans, and thus expect them to succeed and fail in ways humans do. To investigate this, tested performance of six commercial and open-source image classification systems on imagery from a set of 10 tool categories, examining how 17 visual transforms impacted both human and AI classification. Results showed that (1) none of the visual transforms we examined produced substantial impairment for human recognition; (2) human errors were limited to mostly to functional confusions; (3) almost all visual transforms impacted nearly every image classifier negatively and often catastrophically; (4) human expectations about performance of AI classifiers map more closely onto human error than AI performance; and (5) models trained with an enriched training set involving examples of the transformed imagery achieved improved performance but were not inoculated from error.
This is an excerpt (36 pages) of the book "Designing Agents for People," which explains how to use comprehensive simulations of people, technology, and environment to design and test agent systems that we trust and find useful. The excerpt provides an overview of the book's contents and approach, including the preface, reader's guide, and index.
-----------------------------
CASE STUDIES OF COMPLEX SYSTEMS — I present six case studies that describe a complicated activity that involves people interacting with technology in challenging environments, including cleaning up an oil spill in the ocean, orchestrating firefighting operations, and running errands with a self-driving car (or a field trip on the moon). Each example is in the form of a project proposal for how to design agents that interact with people during such activities, in such a manner that they view the automation as providing a helpful tool. In particular, the agents must properly relate to what people want to accomplish and how they think and carry out their activities, that is, their practices, without further complicating what they are doing.
-----------------------------
SCIENTIFIC R&D METHODOLOGY — Each project uses the same general method for designing software—simulating in a computer program how the agents and people would behave in practical situations, over a period of hours or days. The simulations are formulated in Brahms, a computer language and tool that is specialized for modeling and simulating complex interactions among people, automation, and the environment. The combination of a comprehensive system model, what-if simulation, and experimentation with prototypes constitutes a scientific design methodology, as explained in the book’s Introduction. A Brahms work practice simulation can be used to create software agents, illustrated by an agent system operating in NASA’s Mission Control Center that interacts with ground personnel to manage file systems onboard the International Space Station.
-----------------------------
FUNDING ADVANCED TECHNOLOGY PROJECTS — This book is based on over 25 years of experience applying Brahms in corporate and government projects. It is particularly relevant to scientists and engineers who seek funding to design technology using a model-based, scientific research and development methodology. Each project description includes excerpts of a sponsoring agency’s request for proposals, reviews we received, responses to program managers, and reflections about the quality of the proposals and reviews. The prevalent text-based practice of proposing and reviewing solutions to complicated socio-technical problems—forbidding mutual learning between proposers and reviewers—is absurdly inappropriate for presenting new R&D methods and kinds of technology. In the Discussion and Recommendations part, I survey and compare other methods of communicating and refining technical proposals that have proven more successful and make suggestions for using reviewing technology more appropriately.
-----------------------------
BOOK CONTENTS — The book has 446 pages, including a detailed table of contents, 57 figures, and 19 tables; references; and a tutorial-style index.
-----------------------------
The Kindle book is a Print Replica edition. It is viewable only on Kindle Fire devices and Kindle Reading Apps for Android and Macintosh devices. Tablets do not support the hyperlinked page and figure/table references, please use the Page function to jump to the desired page.
From healthcare to criminal justice, artificial intelligence (AI) is increasingly supporting high-consequence human decisions. This has spurred the field of explainable AI (XAI). This paper seeks to strengthen empirical application-specific investigations of XAI by exploring theoretical underpinnings of human decision making, drawing from the fields of philosophy and psychology. In this paper, we propose a conceptual framework for building human-centered, decision-theory-driven XAI based on an extensive review across these fields. Drawing on this framework, we identify pathways along which human cognitive patterns drives needs for building XAI and how XAI can mitigate common cognitive biases. We then put this framework into practice by designing and implementing an explainable clinical diagnostic tool for intensive care phenotyping and conducting a co-design exercise with clinicians. Thereafter, we draw insights into how this framework bridges algorithm-generated explanations and human decision-making theories. Finally, we discuss implications for XAI design and development.
This is the fourth in a series of essays about explainable AI. Previous essays laid out the theoretical and empirical foundations. This essay focuses on Deep Nets, and con-siders methods for allowing system users to generate self-explanations. This is accomplished by exploring how the Deep Net systems perform when they are operating at their boundary conditions. Inspired by recent research into adversarial examples that demonstrate the weakness-es of Deep Nets, we invert the purpose of these adversar-ial examples and argue that spoofing can be used as a tool to answer contrastive explanation questions via user-driven exploration.
There has been a recent resurgence in the area of explainable artificial intelligence as researchers and practitioners seek to provide more transparency to their algorithms. Much of this research is focused on explicitly explaining decisions or actions to a human observer, and it should not be controversial to say that, if these techniques are to succeed, the explanations they generate should have a structure that humans accept. However, it is fair to say that most work in explainable artificial intelligence uses only the researchers' intuition of what constitutes a `good' explanation. There exists vast and valuable bodies of research in philosophy, psychology, and cognitive science of how people define, generate, select, evaluate, and present explanations. This paper argues that the field of explainable artificial intelligence should build on this existing research, and reviews relevant papers from philosophy, cognitive psychology/science, and social psychology, which study these topics. It draws out some important findings, and discusses ways that these can be infused with work on explainable artificial intelligence.
Adaptive autonomous systems of interest to DoD have a great potential to complement human performance in a wide range of missions. This is particularly true for adaptive systems that learn - those whose behavior on a given set of inputs may change over time, even after a system is fielded. Such adaptation makes exhaustive testing, certification and licensure of the final system impossible. The challenge is to establish high confidence that the system will perform dependably and behave as intended while safely, securely, reliably and effectively carrying out the assigned missions. This paper adapts approaches from other disciplines, such as software assurance theory where safety as well as performance is paramount. The key features of the approach include taking and retaining performance data throughout the development, and recognizing that system testing will need to continue beyond a fielding decision. An additional complexity is that performance testing needs to address how the autonomous modules make decisions and provide a standard for doing that. That is test, evaluation, verification and validation of such systems must inform testers about how a system satisfies requirements during development and about its potential to make effective system changes after fielding.
How can end users efficiently influence the predictions that machine learning systems make on their behalf? This paper presents Explanatory Debugging, an approach in which the system explains to users how it made each of its predictions, and the user then explains any necessary corrections back to the learning system. We present the principles underlying this approach and a prototype instantiating it. An empirical evaluation shows that Explanatory Debugging increased participants' understanding of the learning system by 52% and allowed participants to correct its mistakes up to twice as efficiently as participants using a traditional learning system.
The profile of people who use software and hardware systems has changed, while systems and the culture in which they are developed have not made the necessary adjustments. In order to transform the culture in which information technology systems are designed, developed and manufactured and to meet growing needs of computer users, the author recently proposed a User's Bill of Rights, a set of goals to challenge the computer industry to respect and address needs of computer users. The fundamental principle of the bill is that the user is always right. The user's rights define computer industry goals. Technology solutions are composed of a variety of hardware and software components and need to be tested as complete systems. End users and customers need to have results of these tests available. Technology magazines and journals could improve product reviews by testing with real user scenarios. Tasks users want to complete, not features that are available, are key to the value proposition when considering a technology purchase. Consumers can be supplied with the summary user data on performance and satisfaction based on working with different systems to complete real work. INSET: The User's Bill of Rights.
Intelligent software algorithm are increasingly becoming a tool in consumers daily lives. Users understand the basic mechanics of the intelligent software systems they rely on, but often novices have no direct knowledge of their intelligent devices algorithm, data requirements, limitations, and representations. Problems can go beyond those caused by a poor user interface design and a user's ability to under stand a tool's simple components, which could be alleviated with proper instruction. This article describes the Experiential User Guide (EUG), a concept designed to address these challenges.
Computers have transformed every facet of our culture, most dramatically communication, transportation, finance, science, and the economy. Yet their impact has not been generally felt in education due to lack of hardware, teacher training, and sophisticated software. Another reason is that current instructional software is neither truly responsive to student needs nor flexible enough to emulate teaching. The more instructional software can reason about its own teaching process, know what it is teaching, and which method to use for teaching, the greater is its impact on education. Building Intelligent Interactive Tutors discusses educational systems that assess a student's knowledge and are adaptive to a student's learning needs. Dr. Woolf taps into 20 years of research on intelligent tutors to bring designers and developers a broad range of issues and methods that produce the best intelligent learning environments possible, whether for classroom or life-long learning. The book describes multidisciplinary approaches to using computers for teaching, reports on research, development, and real-world experiences, and discusses intelligent tutors, web-based learning systems, adaptive learning systems, intelligent agents and intelligent multimedia.
An explanation facility is an important component of an expert system, but current systems for the most part have neglected the importance of tailoring a system's explanations to the user. This paper explores the role of user modeling in generating expert system explanations, making the claim that individualized user models are essential to produce good explanations when the system users vary in their knowledge of the domain, or in their goals, plans, and preferences. To make this argument, a characterization of explanation, and good explanation is made, leading to a presentation of how knowledge about the user affects the various aspects of a good explanation. Individualized user models are not only important, it is practical to obtain them. A method for acquiring a model of the user's beliefs implicitly by "eavesdropping" on the interaction between user and system is presented, along with examples of how this information can be used to tailor an explanation.
This paper describes the English explanation facility of the OWL Digitalis Advisor, a program designed to advise physicians regarding digitalis therapy. The program is written in OWL, an English-based computer language being developed at MIT. The system can explain, in English, both the methods it uses and how those methods were applied during a particular session. In addition, the program can explain how it acquires information and tell the user how it deals with that information either in general or during a particular session.
We present an overview of different theories of explanation from the phi- losophy and cognitive science communities. Based on these theories, as well as models of explanation from the knowledge-based systems area, we present a framework for explanation in case-based reasoning (CBR) based on explanation goals. We propose ways that the goals of the user and system designer should be taken into account when deciding what is a good explanation for a given CBR system. Some general types of goals relevant to many CBR systems are identified, and used to survey existing meth- ods of explanation in CBR. Finally, we identify some future challenges.
In order to be considered useful and acceptable, expert systems must be able to explain their knowledge of the domain and the reasoning process they employ to produce results and recommendations. Despite the fact that the need for explanation is widely recognized, current expert systems have only limited explanatory capabilities. Early approaches to explanation in expert systems are reviewed and their limitations discussed. Improvements to the explanation capabilities based on enriched knowledge bases of expert systems are presented. Further improvements in explanation requiring better generation techniques is stressed. Related work in the field of natural language generation suggests techniques that are useful to the task of explanation in expert systems; however, even those techniques will not provide all of the capabilities required for the task of carrying on a dialog with the user. Finally, the approach to explanation is described, which provides the facilities necessary to carry on an interactive dialog with the user.
To place the arguments advanced in this paper in alternative points of view with regard to mental models are reviewed. Use of the construct in areas such as neural information processing, manual control, decision making, problem solving, and cognitive science are discussed. Also reviewed are several taxonomies of mental models. The available empirical evidence for answering questions concerning the nature and usage of mental models is then discussed. A variety of studies are reviewed where the type and form of humans' knowledge have been manipulated. Also considered are numerous transfer of training studies whose results provide indirect evidence of the nature of mental models. The alternative perspectives considered and the spectrum of empirical evidence are combined to suggest a framework within which research on mental models can be viewed. By considering interactions of dimensions of this framework, the most salient unanswered questions can be identified.
Two experiments are reported which examined operators' trust in and use of the automation in a simulated supervisory process control task. Tests of the integrated model of human trust in machines proposed by Muir (1994) showed that models of interpersonal trust capture some important aspects of the nature and dynamics of human-machine trust. Results showed that operators' subjective ratings of trust in the automation were based mainly upon their perception of its competence. Trust was significantly reduced by any sign of incompetence in the automation, even one which had no effect on overall system performance. Operators' trust changed very little with experience, with a few notable exceptions. Distrust in one function of an automatic component spread to reduce trust in another function of the same component, but did not generalize to another independent automatic component in the same system, or to other systems. There was high positive correlation between operators' trust in and use of the automation; operators used automation they trusted and rejected automation they distrusted, preferring to do the control task manually. There was an inverse relationship between trust and monitoring of the automation. These results suggest that operators' subjective ratings of trust and the properties of the automation which determine their trust, can be used to predict and optimize the dynamic allocation of functions in automated systems.
AI systems are increasingly being fielded to support diagnoses and healthcare advice for patients. One promise of AI application is that they might serve as the first point of contact for patients, replacing routine tasks, and allowing health care professionals to focus on more challenging and critical aspects of healthcare. For AI systems to succeed, they must be designed based on a good understanding of how physicians explain diagnoses to patients, and how prospective patients understand and trust the systems providing the diagnosis, as well as the explanations they expect. In this thesis, I examine this problem across three studies. In the first study, I interviewed physicians to explore their explanation strategies in re-diagnosis scenarios. I identified five broad categories of explanation strategies and I developed a generic diagnostic timeline of explanations from the interviews. For the second study, I tested an AI diagnosis scenario and found that explanation helps improve patient satisfaction measures for re-diagnosis. Finally, in a third study I implemented different forms of explanation in a similar diagnosis scenario and found that visual and example-based explanation integrated with rationales had a significantly better impact on patient satisfaction and trust than no explanations, or with text-based rationales alone. Based on these studies and the review of the literature, I provide some design recommendations for the explanations offered for AI systems in the healthcare domain.
As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.
We summarize the potential impact that the European Union's new General Data Protection Regulation will have on the routine use of machine learning algorithms. Slated to take effect as law across the EU in 2018, it will restrict automated individual decision-making (that is, algorithms that make decisions based on user-level predictors) which "significantly affect" users. The law will also create a "right to explanation," whereby a user can ask for an explanation of an algorithmic decision that was made about them. We argue that while this law will pose large challenges for industry, it highlights opportunities for machine learning researchers to take the lead in designing algorithms and evaluation frameworks which avoid discrimination.
People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion-an illusion of explanatory depth. The illusion is far stronger for explanatory knowledge than many other kinds of knowledge, such as that for facts, procedures or narratives. The illusion for explanatory knowledge is most robust where the environment supports real-time explanations with visible mechanisms. We demonstrate the illusion of depth with explanatory knowledge in Studies 1-6. Then we show differences in overconfidence about knowledge across different knowledge domains in Studies 7-10. Finally, we explore the mechanisms behind the initial confidence and behind overconfidence in Studies 11 and 12, and discuss the implications of our findings for the roles of intuitive theories in concepts and cognition. © 2002 Leonid Rozenblit. Published by Cognitive Science Society, Inc. All rights reserved.
Although “mental models” are of central importance to system dynamics research and practice, the field has yet to develop an unambiguous and agreed upon definition of them. To begin to address this problem, existing definitions and descriptions of mental models in system dynamics and several literatures related to cognitive science were reviewed and compared. Available definitions were found to be overly brief, general, and vague, and different authors were found to markedly disagree on the basic characteristics of mental models. Based on this review, we concluded that in order to reduce the amount of confusion in the literature, the mental models concept should be “unbundled” and the term “mental models” should be used more narrowly. To initiate a dialogue through which the system dynamics community might achieve a shared understanding of mental models, we propose a new definition of “mental models of dynamic systems” accompanied by an extended annotation that explains the definitional choices made and suggests terms for other cognitive structures left undefined by narrowing the mental model concept. Suggestions for future research that could improve the field's ability to further define mental models are discussed. © 1998 John Wiley & Sons, Ltd.
Large Convolutional Neural Network models have recently demonstrated
impressive classification performance on the ImageNet benchmark \cite{Kriz12}.
However there is no clear understanding of why they perform so well, or how
they might be improved. In this paper we address both issues. We introduce a
novel visualization technique that gives insight into the function of
intermediate feature layers and the operation of the classifier. We also
perform an ablation study to discover the performance contribution from
different model layers. This enables us to find model architectures that
outperform Krizhevsky \etal on the ImageNet classification benchmark. We show
our ImageNet model generalizes well to other datasets: when the softmax
classifier is retrained, it convincingly beats the current state-of-the-art
results on Caltech-101 and Caltech-256 datasets.
This book draws on classroom experiences and faculty suggestions in providing a practical guide to teaching strategies to encourage active learning in the college classroom. A wide range of teaching tools which ask students to apply what they are learning are considered, including problem-solving exercises, cooperative student projects, informal group work, simulations, case studies, and role playing. Additionally, the book discusses how various small-group exercises, simulations, and case studies can be blended with the technological and human resources available outside the classroom. The book is divided into three parts. Part 1 surveys the general subject of active learning and why it makes sense as a teaching strategy. Part 2 considers four major active-learning teaching strategies in more depth: informal small groups, cooperative student projects, simulations, and case studies. Part 3 explores how reading assignments, outside resource persons, and electronic media can be successfully integrated with active-learning strategies in the classroom. Contains approximately 150 references and an index. (GLR)
The key idea behind active learning is that a machine learning algorithm can achieve greater accuracy with fewer labeled training instances if it is allowed to choose the data from which is learns. An active learner may ask queries in the form of unlabeled instances to be labeled by an oracle (e.g., a human annotator). Active learning is well-motivated in many modern machine learning problems, where unlabeled data may be abundant but labels are difficult, time-consuming, or expensive to obtain. This report provides a general introduction to active learning and a survey of the literature. This includes a discussion of the scenarios in which queries can be formulated, and an overview of the query strategy frameworks proposed in the literature to date. An analysis of the empirical and theoretical evidence for active learning, a summary of several problem setting variants, and a discussion of related topics in machine learning research are also presented.
Recent studies have pointed out several limitations of expert systems regarding user needs, and have introduced the concepts of cooperation and joint cognitive systems. While research on explanation generation by expert systems has been widely developed, there has been little consideration of explanation in relation to cooperative systems. Our aim is to elaborate a conceptual framework for studying explanation in cooperation. This work relies heavily on the study of human-human cooperative dialogues. We present our results according to two dimensions; namely, the relation between explanation and problem solving, and the explanation process itself. Finally, we discuss the implications of these results for the design of cooperative systems.
The Practitioner's Cycles Part 1 illustrated ways in which actual world problems- the forces and constraints of procurement- are at odds with the goals of a human-centered approach. In Part 2, we presented an application of the Practitioner's Cycles model to envisioned world problems-the creation of intelligent technologies for new work systems. (See the "Synopsis of the Practitioner's Cycles" sidebar for a quick review.) This essay (Part 3) describes formative implementations of the Practitioner's Cycles. We propose the Practitioner's Cycles concept as a solution to the simultaneous problems of maturing new technologies and rapidly transitioning those technologies into new products.
The present paper analyzes the self-generated explanations (from talk-aloud protocols) that “Good” and “Poor” students produce while studying worked-out examples of mechanics problems, and their subsequent reliance on examples during problem solving. We find that “Good” students learn with understanding: They generate many explanations which refine and expand the conditions for the action parts of the example solutions, and relate these actions to principles in the text. These self-explanations are guided by accurate monitoring of their own understanding and misunderstanding. Such learning results in example-independent knowledge and in a better understanding of the principles presented in the text. “Poor” students do not generate sufficient self-explanations, monitor their learning inaccurately, and subsequently rely heavily on examples. We then discuss the role of self-explanations in facilitating problem solving, as well as the adequacy of current AI models of explanation-based learning to account for these psychological findings.
Previous Human-Centered Computing department articles have reflected on the mismatch that can occur between the promise of intelligent technology and the results of technological interventions. Part 1 on the Practitioner's Cycles illustrated ways in which actual world problems-the forces and constraints of procurement-are at odds with the goals of human centering. This article culminated in a practitioner's tale, in which individuals acted on their own initiative and at their own risk, short-circuiting the rules and constraints that limit success at procurement. This paper presents a model based on the tale and focuses on how the model applies to envisioned world problems-the creation of intelligent technologies for new work systems.
Existing explanation facilities are typically far more appropriate for knowledge engineers engaged in system maintenance than for end-users of the system. This is because the explanation is little more than a trace of the detailed problem-solving steps. An alternative approach recognizes that an effective explanation often needs to substantially reorganize the actual line of reasoning and bring to bear additional information to support the result. Explanation itself becomes a complex problem-solving process that depends not only on the actual line of reasoning, but also on additional knowledge of the domain. This paper presents a new computational model of explanation and argues that it results in significant improvements over traditional approaches.
Sumario: Today many systems are highly automated. The human operator's role in these systems is to supervise the automation and intervene to take manual control when necessary. The operator's choice of automatic or manual control has important consequences for system performance, and therefore it is important to understand and optimize this decision process. In this paper a model of human trust in machines is developed, taking models of trust between people as a starting point, and extending them to the human-machine relationship
Many abductive understanding systems generate explanations by a backwards chaining process that is neutral both to the explainer's previous experience in similar situations and to why the explainer is attempting to explain. This article examines the relationship of such models to an approach that uses case-based reasoning to generate explanations. In this case-based model, the generation of abductive explanations is focused by prior experience and by goal-based criteria reflecting current information needs. The article analyzes the commitments and contributions of this case-based model as applied to the task of building good explanations of anomalous events in everyday understanding. The article identifies six central issues for abductive explanation, compares how these issues are addressed in traditional and case-based explanation models, and discusses benefits of the case-based approach for facilitating generation of plausible and useful explanations in domains that are complex and imperfectly understood.
Trust in automated systems
- B D Adams
- L E Bruyn
- S Houde
- P Angelopoulos
- K Iwasa-Madge
- C Mccann
Adams, B. D., Bruyn, L. E., Houde, S., Angelopoulos, P., Iwasa-Madge, K., and McCann, C. 2003. Trust in automated systems.
Canada Ministry of National Defence.
Context needs in cooperative building of explanations
- P Brézillon
Brézillon, P. 1994. Context needs in cooperative building of explanations. In First European Conference on Cognitive Science in
Industry (pp. 443-450).
Explanation Ontology: A Model of Explanations for User-Centered AI
- S Chari
- O Seneviratne
- D M Gruen
- M A Foreman
- A K Das
- D L Mcguinness
Chari, S., Seneviratne, O., Gruen, D. M., Foreman, M. A., Das,
A. K., & McGuinness, D. L. (2020, November). Explanation Ontology: A Model of Explanations for User-Centered AI. In International Semantic Web Conference (pp. 228-243). Springer.
Improving Automation Transparency: Addressing Some of Machine Learning's Unique Challenges
- C K Fallon
- L Blaha
Fallon, C. K., and Blaha, L. M. 2018. Improving Automation
Transparency: Addressing Some of Machine Learning's Unique
Challenges. International Conference on Augmented Cognition,
245-254. Springer.
Using the critiquing approach to cope with brittle expert systems
- S Guerlain
Guerlain, S. 1995. Using the critiquing approach to cope with
brittle expert systems. In Proceedings of the Human Factors and
Ergonomics Society Annual Meeting (Vol. 39, pp. 233-237).
SAGE Publications Sage CA: Los Angeles, CA.
Generating visual explanations. European Conference on Computer Vision
- L A Hendricks
- Z Akata
- M Rohrbach
- J Donahue
- B Schiele
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele,
B., and Darrell, T. 2016. Generating visual explanations. European Conference on Computer Vision, 3-19. http://link.springer.-com/chapter/10.1007/978-3-319-46493-0_1
- R R Hoffman
Hoffman, R.R. (ed) 2012. Collected Essays on Human-Centered
Computing, 2001-2011. New York: IEEE Computer Soc. Press.
Explaining explanation for "Explainable AI
- R R Hoffman
- G Klein
- S T Mueller
Hoffman, R. R., Klein, G., and Mueller, S. T. 2018b. Explaining
explanation for "Explainable AI". Proceedings of the 2018 conference of the Human Factors and Ergonomic Society (HFES),
Philadelphia PA, October 2018.