ChapterPDF Available

Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges


Abstract and Figures

Deep learning has made significant contribution to the recent progress in artificial intelligence. In comparison to traditional machine learning methods such as decision trees and support vector machines, deep learning methods have achieved substantial improvement in various prediction tasks. However, deep neural networks (DNNs) are comparably weak in explaining their inference processes and final results, and they are typically treated as a black-box by both developers and users. Some people even consider DNNs (deep neural networks) in the current stage rather as alchemy, than as real science. In many real-world applications such as business decision, process optimization, medical diagnosis and investment recommendation, explainability and transparency of our AI systems become particularly essential for their users, for the people who are affected by AI decisions, and furthermore, for the researchers and developers who create the AI solutions. In recent years, the explainability and explainable AI have received increasing attention by both research community and industry. This paper first introduces the history of Explainable AI, starting from expert systems and traditional machine learning approaches to the latest progress in the context of modern deep learning, and then describes the major research areas and the state-of-art approaches in recent years. The paper ends with a discussion on the challenges and future directions.
Content may be subject to copyright.
Explainable AI: A Brief Survey on History,
Research Areas, Approaches and Challenges
Feiyu Xu1, Hans Uszkoreit2, Yangzhou Du1, Wei Fan1, Dongyan Zhao3, and
Jun Zhu4
1AI Lab, Lenovo Research, Lenovo Group, China
2DFKI GmbH, Germany and Giance Technologies
3Institute of Computer Science and Technology, Peking University, China
4Department of Computer Science and Technology, Tsinghua University, China
Abstract. Deep learning has made significant contribution to the recent
progress in artificial intelligence. In comparison to traditional machine
learning methods such as decision trees and support vector machines,
deep learning methods have achieved substantial improvement in various
prediction tasks. However, deep neural networks (DNNs) are comparably
weak in explaining their inference processes and final results, and they
are typically treated as a black-box by both developers and users. Some
people even consider DNNs (deep neural networks) in the current stage
rather as alchemy, than as real science. In many real-world applications
such as business decision, process optimization, medical diagnosis and
investment recommendation, explainability and transparency of our AI
systems become particularly essential for their users, for the people who
are affected by AI decisions, and furthermore, for the researchers and
developers who create the AI solutions. In recent years, the explainabil-
ity and explainable AI have received increasing attention by both re-
search community and industry. This paper first introduces the history
of Explainable AI, starting from expert systems and traditional machine
learning approaches to the latest progress in the context of modern deep
learning, and then describes the major research areas and the state-of-
art approaches in recent years. The paper ends with a discussion on the
challenges and future directions.
Keywords: Explainable artificial intelligence ·intelligible machine learn-
ing ·explainable interfaces ·XAI ·interpretability
1 A Brief History of Explainable AI
In wiktionary, the word “explain” means for humans “to make plain, manifest,
or intelligible; to clear of obscurity; to illustrate the meaning of” [23]. In scientific
research, a scientific explanation is supposed to cover at least two parts: 1) the
2 F. Xu et al.
object to be explained (the “Explanandum” in Latin), and 2) the content of
explanation (the “Explanans” in Latin).
Explainable AI is not a new topic. The earliest work on Explainable AI could
be found in the literature published forty years ago [16] [19], where some expert
systems explained their results via the applied rules. Since AI research began,
scientists have argued that intelligent systems should explain the AI results,
mostly when it comes to decisions. If a rule-based expert system rejects a credit
card payment, it should explain the reasons for the negative decision. Since the
rules and the knowledge in the expert systems are defined and formulated by
human experts, these rules and knowledge are easy for humans to understand and
interpret. Decision tree is a typical method designed with explainable structure.
As illustrated in Fig. 1, starting at the top and going down, the solution path
in the decision tree presents the reasoning of a final decision.
Fig. 1. An example of decision tree, used by starting at the top and going down, level
by level, according to the defined logic. (Image courtesy of J. Jordan [10])
However, Explainable AI has become a new research topic in the context of
modern deep learning. Without completely new explanatory mechanisms, the
output of today’s Deep Neural Networks (DNNs) cannot be explained, neither
by the neural network itself, nor by an external explanatory component, and
not even by the developer of the system. We know that there are different archi-
tectures of DNNs designed for different problem classes and input data, such as
CNN, RNN, LSTM, shown in Fig. 2. All of them have to be considered as black
boxes - whose internal inference processes are neither known to the observer nor
interpretable by humans [7].
Explainability of a machine learning model is usually inverse to its prediction
accuracy - the higher the prediction accuracy, the lower the model explainability.
The DARPA Explainable AI (XAI) program presents a nice chart to illustrate
this interesting phenomena, as shown in Fig. 3, where decision trees have an
excellent degree of explainability but exhibit worst prediction accuracy among
the listed learning techniques. In the other extreme, Deep Learning methods are
Title Suppressed Due to Excessive Length 3
Fig. 2. A chart of several typical Deep Neural Networks (DNNs). (Image courtesy of
Fjodor Van Veen [22])
better in predictive capacity than any other learning methods but they are least
likely to be explicable.
Fig. 3. Explainability of machine learning models appear inverse to their prediction
accuracy. (Image courtesy of DARPA [21])
In recent years, AI researchers aim to open the black-box of neural networks
and turn it into a transparent system. As shown in Fig. 4, there are two main
strands of work in Explainable AI - transparency design and post-hoc expla-
nation. The transparency design reveals how a model functions, in the view of
developers. It tries to (a) understand model structure, e.g., the construction of
a decision tree; (b) understand single components, e.g., a parameter in logistic
regression; (c) understand training algorithms, e.g., solution seeking in a con-
vex optimization. The post-hoc explanation explains why a result is inferred, in
the view of users. It tries to (d) give analytic statements, e.g. why a goods is
recommended in a shopping website; (e) give visualizations, e.g. saliency map
is used to show pixel importance in a result of object classification; (f ) give ex-
4 F. Xu et al.
planations by example, e.g. K-nearest-neighbors in historical dataset are used to
support current results. A thorough description of the categorization of expla-
nation methods is found in Lipton et al. [13]. A comprehensive survey on recent
development of Explainable AI is provided in [5].
Fig. 4. Two categories of Explainable AI work: transparency design and post-hoc ex-
2 Relevance of Explainable AI
Increasing attention has recently been paid to Explainable AI across the world
both in research and in industry. In April 2017, DARPA funded the “Explain-
able AI (XAI) program”, aimed at improving explainability of AI decision [21].
In July 2017, the Chinese government released “The Development Plan for New
Generation of Artificial Intelligence” to encourage high-explainability AI and
strong-extensibility AI [18]. In May 2018, the “General Data Protection Regula-
tion” (GDPR) was published, in which the European Union grants their citizens
a “right to explanation” if they are affected by algorithmic decision-making [6].
Explainable AI will become increasingly important to all groups of stakeholders,
including the users, the affected people, and the developers of AI systems.
Explainable AI is important to the users who utilize the AI system. When
the AI recommends a decision, the decision makers would need to understand
the underlying reason. For example, medical doctor needs to understand what
pathological features in the input data were were guiding the algorithm before
accepting auto-generated diagnosis reports. A maintenance engineer needs to
understand which abnormal phenomena were captured by the inference algo-
rithm before following the repair recommendations. A financial investor wants
to understand what influencing factors were regarded as the critical ones by the
system algorithm before making the final investment decision. We have to verify
that the AI inference works as expected, because wrong decisions can be costly
and dangerous. Caruana et al. [3] presented a famous example “Pneumonia -
Asthma” to illustrate this point. An AI system which had been trained to pre-
dict the pneumonia risk of a person arrived at totally wrong conclusions. From
real data the model had learned that asthmatic patients with heart problems
have a much lower risk of dying of pneumonia than healthy persons. This cannot
be true since asthma is a factor that negatively affects the recovery. The training
Title Suppressed Due to Excessive Length 5
data were systematically biased, because in contrast to healthy persons, the ma-
jority of these asthma patients were under strict medical supervision. Hence this
group had a significant lower risk of dying of pneumonia. It should be noted,
though, that both the learning and the inference algorithms probably worked
correctly and also that the training data represented real cases. The insight that
the selection of the training data was not appropriate for predictions affecting
other populations may remain undiscovered if we have a black-box AI system.
Explainable AI is important to the people who are affected by AI decision. If
the AI makes its own decisions, e.g., braking of the car, shutting down a plant,
selling shares, assessing a job, issuing a traffic punishment order, the affected
people must be able to understand the reason. There are already legal regula-
tions that codify this demand [6]. Houston schools were using an AI algorithm,
called Educational Value-Added Assessment System (EVAAS), to evaluate the
performance of teachers. However, this AI system was successfully contested by
teachers in court, because negative reviews of teachers could not be explained
by the AI system [2].
Explainable AI could help developers to improve AI algorithm, by detecting
data bias, discovering mistakes in the models, and remedying the weakness. La-
puschkin et al. [11] presented an impressive example. As shown in Fig. 5, they
observed that the Fisher Vector method usually shows lower accuracy than Deep
Neural Networks in the task of object recognition. However, two methods reach
almost equal accuracy of recognition rate in the category “horse”, which is unex-
pected. A saliency map method called “Layer-wise Relevance Propagation” [12]
was then employed to analyze which pixel areas exactly make the models arrive
at their predictions. The authors observed that the two models use different
strategies to classify images of that category. The Deep Neural Network looked
at the contour of the actual horse, whereas the Fisher Vector model mostly relied
on a certain copyright tag, that happens to be present on many horse images.
Removing the copyright tag in the test images would consequently significantly
decrease the accuracy of the Fisher Vector model.
Fig. 5. Upper: the prediction accuracy of Fisher Vector and Deep Neural Network in
tasks of object recognition; Lower: model diagnosis using saliency map method. (Image
courtesy of Lapuschkin et al. [11])
6 F. Xu et al.
3 Relevant Explainable AI Problems and Current
As shown in Fig. 6, there are three typical approaches to understand the behavior
of a Deep Neural Network: (a) making the parts of the network transparent - the
color of the neuron indicates its activation status; (b) learning semantics of the
network components - a neuron could have a meaning if it is often activated by
a certain part of the object; (c) generation of explanations - a human-readable
textual explanation tells the underlying reason to support current decision.
Fig. 6. Three approaches for understanding a neural network, indicated by red-boxes
(a), (b) and (c)
3.1 Making the parts in DNN transparency
This section introduces two popular techniques, namely sensitivity analysis (SA) [15]
[1] and layer-wise relevance propagation (LRP) [17], for explaining prediction of
deep learning models.
SA explains a prediction based on the model’s locally evaluated gradient.
It assumes that the most relevant input features are the most sensitive for the
output. SA doesn’t explain the function value f(x), but rather quantifies the
importance of each input variable xi.
In contrast to SA, LRP explains predictions relative to the state of maximum
uncertainty. It redistributes the prediction f(x) backwards using local redistri-
bution rules until it assigns a relevance score Rito each input variable. The
relevance score Riof each input variable determines the variable’s importance
Title Suppressed Due to Excessive Length 7
to the prediction.
Rj=... =X
Rk=... =f(x).(2)
The Relevance conservation is the key property of the redistribution process.
This property ensures that no relevance is artificially added or removed during
redistribution. Thus, LRP truly decomposes the function values f(x) in contrast
to SA.
Fig. 7. Explaining predictions of an AI system using SA and LRP. (Image courtesy of
W. Samek [15])
Fig. 7 summarizes the process of explanation. The AI system correctly clas-
sifies the input image as “rooster”. SA indicates yellow flowers which occlude
part of the rooster need to be changed to make the image look more like the
predicted. However, such result would not indicate which pixels are actually piv-
otal for the prediction “rooster”. In contrast to SA, the heatmap computed with
LRP identifies pixels which are pivotal for the prediction “rooster”.
Additionally, SA and LRP are evaluated on three different classification tasks,
namely the annotation of images, the classification of text documents and the
recognition of human actions in videos. Fig. 8(A) shows two images from the
ILSVRC2012 [4] dataset, which have been correctly classified as “volcano” and
“coffee cup”, respectively. From the figure, we can see that SA heatmaps are
much noisier than the ones computed with LRP. SA doesn’t indicate how much
every pixel contributes to the prediction. LRP produces better explanations than
SA. Fig. 8(B) shows SA and LRP heatmaps overlaid on top of a document from
the 20Newsgroup dataset. In contrast to LPR, SA methods don’t distinguish be-
tween positive and negative evidence. Similarly, Fig. 8(C) shows LRP heatmaps
8 F. Xu et al.
not only visualizes the relevant locations of the action within a video frame, but
also identifies the most relevant time points within a video sequence.
Fig. 8. Explaining prediction of three different problems using SA and LRP. (Image
courtesy of W. Samek [15])
3.2 Learning semantic graphs from existing DNNs
Zhang et al. [24] proposes a method that learns a graphical model, called “ex-
planatory graph”, which reveals the knowledge hierarchy hidden inside a pre-
trained Convolutional Neural Network (CNN), as shown in Fig. 9. The graph
consists of multiple layers, each of them corresponds to a convolutional layer
in the CNN. Each node in the graph represents a specific part of the detected
object, as shown in the right side of the figure. These nodes are derived from re-
sponses of CNN filters with a disentangle algorithm. The edge connecting nodes
indicates their co-activation relationship in filter response and the spatial rela-
tionship in parts location. The layer shows different granularity of the part of
objects - larger parts appear in higher layers while smaller parts appear in lower
layers. This work, however, adopts an explanatory graph as a bridge to under-
stand the ordinary CNN. In later work [25], the authors introduce additional
losses to force each convolutional filter in CNN to represent a specific object
part directly, and produce an interpretable CNN.
Title Suppressed Due to Excessive Length 9
Fig. 9. An explanatory graph represents the knowledge hierarchy hidden in convolu-
tional layers of a CNN. (Image courtesy of Zhang et al. [24])
3.3 Generation of Explanations
This section introduces a novel framework which provides visual explanations
of a visual classifier [8]. Visual explanations are both image relevant and class
relevant. From Fig. 10 we can find image descriptions provides a sentence based
on visual information but not necessarily class relevant, while class definitions are
class relevant but not necessarily image relevant. In contrast, Visual explanation
such as “This is a western grebe because this bird has a long white neck, pointy
yellow beak, and a red eye.” includes the “red eye” property which is important
to distinguish between “western grebe” and “laysan albatross”. Therefore, Visual
explanations are both image relevant and class relevant. It explains why the
predicted category is the most appropriate for the image.
Fig. 10. Visual explanations are both image relevant and class relevant. (Image cour-
tesy of L. A. Hendricks [9])
Fig. 11 shows the generation of explanatory text on both an image and a
predicted class label. The input is run through a deep fine-grained recognition
pipeline to pick out nuanced details of the image and classifying it. The features
10 F. Xu et al.
and the label are then forwarded to the LSTM stack to produce a sequence of
Fig. 11. Generation of explanatory text with joint classification and language model.
(Image courtesy of L. A. Hendricks [9])
4 Challenges and Future Directions
The development of Explainable AI is facing both scientific and social demands.
We expect AI systems could help humans make decisions in mission-critical
tasks. Therefore, We need a more trustworthy and transparent AI, instead of
alchemy AI [20]. Ali Rahimi, the winner of the test-of-time award in NeurIPS
2017, expressed his expectations concerning AI solutions as follows: “We are
building systems that govern healthcare and mediate our civic dialogue. We
would influence elections. I would like to live in a society whose systems are built
on top of verifiable, rigorous, thorough knowledge, and not on alchemy. Let’s take
machine learning from alchemy to electricity” [14]. The term “electricity” in his
speech could be replaced by “chemistry” from our perspective, meaning that AI
Deep Learning AI should become part of science.
DARPA has invested 50 Million USD and launched a 5-year research pro-
gram on Explainable AI (XAI) [21], aiming to produce “glass-box” models that
are explainable to a “human-in-the-loop”, without greatly sacrificing AI perfor-
mance, as shown in Fig. 12. Human users should be able to understand the AI’s
cognition both in real-time and after the results achieved, and furthermore might
be able to determine when to trust the AI and when the AI should be distrusted.
In Phase 1, it is planned to achieve initial implementations of their explainable
learning systems. In Phase 2, it is to build a toolkit library consisting of machine
learning and human-computer interface software modules that could be utilized
for developing future explainable AI systems.
It is known that humans can acquire and use both explicit knowledge and
implicit knowledge. Moreover, humans can combine the two forms of knowledge
to a certain degree. For humans, understanding and explaining require explicit
knowledge. However, DNNs acquire and use implicit knowledge in the form of
probabilistic models. As they stand, they cannot understand anything. Other
AI methods model explicit knowledge, such as Knowledge Graphs. Today the
two worlds in AI technology are still largely separated. Researchers are now
Title Suppressed Due to Excessive Length 11
Fig. 12. Explainable AI (XAI) Concept presented by DARPA. (Image courtesy of
DARPA XAI Program [21])
strengthening their efforts to bring the two worlds together. The need-driven
research on Explainable AI is a source and a catalyst for the work dedicated to
this grand challenge.
1. Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., M¨uller,
K.: How to explain individual classification decisions. J. Mach. Learn. Res. 11,
1803–1831 (2010),
2. Cameron, L.: Houston Schools Must Face Teacher Evaluation Lawsuit (2017),
3. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible
models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.
In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining. pp. 1721–1730. ACM (2015)
4. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Li, F.: Imagenet: A large-scale hierar-
chical image database. In: 2009 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida,
USA. pp. 248–255 (2009).,
5. Doilovi, F.K., Bri, M., Hlupi, N.: Explainable artificial intelligence: A survey. In:
2018 41st International Convention on Information and Communication Tech-
nology, Electronics and Microelectronics (MIPRO). pp. 0210–0215 (May 2018).
6. Goodman, B., Flaxman, S.: European union regulations on algorithmic decision-
making and a ”right to explanation”. AI magazine 38(3), 50–57 (2017)
7. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.:
A survey of methods for explaining black box models. ACM Computing Surveys
(CSUR) 51 (2019)
12 F. Xu et al.
8. Hendricks, L.A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., Darrell, T.:
Generating visual explanations. In: European Conference on Computer Vision. pp.
3–19. Springer (2016)
9. Hendricks, L.A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., Darrell, T.:
Generating visual explanations. In: European Conference on Computer Vision. pp.
3–19. Springer (2016)
10. Jeremy, J.: Decision trees (2017),
11. Lapuschkin, S., Binder, A., Montavon, G., Muller, K.R., Samek, W.: Analyzing
classifiers: Fisher vectors and deep neural networks. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition. pp. 2912–2920 (2016)
12. Lapuschkin, S., Binder, A., Montavon, G., M¨uller, K.R., Samek, W.: The lrp tool-
box for artificial neural networks. Journal of Machine Learning Research 17(114),
1–5 (2016),
13. Lipton, Z.C.: The mythos of model interpretability. ACM Queue - Machine Learn-
ing 16 (2018)
14. Rahimi, A.: NIPS 2017 Test-of-Time Award presentation (2017),
15. Samek, W., Wiegand, T., M¨uller, K.R.: Explainable artificial intelligence: Un-
derstanding, visualizing and interpreting deep learning models. arXiv preprint
arXiv:1708.08296 (2017)
16. Scott, A.C., Clancey, W.J., Davis, R., Shortliffe, E.H.: Explanation capabilities of
production-based consultation systems. American Journal of Computational Lin-
guistics 62 (1977)
17. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks:
Visualising image classification models and saliency maps. In: 2nd International
Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April
14-16, 2014, Workshop Track Proceedings (2014),
18. State Council Chinese Government: Development Plan for New Genera-
tion Artificial Intelligence (2017),
07/20/content 5211996.htm
19. Swartout, W.R.: Explaining and justifying expert consulting programs. In: Pro-
ceedings of the 7th International Joint Conference on Artificial Intelligence (1981)
20. Tony, P.: LeCun vs Rahimi: Has Machine Learning Become Alchemy? (2017),
21. Turek, M.: DARPA - Explainable Artificial Intelligence (XAI) Program (2017),
22. Van Veen, F.: The Neural Network Zoo (2016),
23. Wiktionary: Explain (2019),
24. Zhang, Q., Cao, R., Shi, F., Wu, Y.N., Zhu, S.C.: Interpreting cnn knowledge via an
explanatory graph. In: Thirty-Second AAAI Conference on Artificial Intelligence
25. Zhang, Q., Wu, Y.N., Zhu, S.C.: Interpretable convolutional neural networks. In:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
pp. 8827–8836 (2018)
... Especially regulated industries, such as finance or the pharmaceutical industry, are not yet able to make full use of AI (Königstorfer and Thalmann 2021;Polzer et al. 2022). When AI decisions affect human lives, sensitive data, or may cause serious risks, AI needs to be trustworthy and reliable (Xu et al. 2019). ...
... Thus, the mechanisms generating an output based on input data become hidden inside a black-box (Adadi and Berrada 2018). For such opaque AI, users are unable to understand how the decisions are made by the AI (Xu et al. 2019). However, explanations of how the AI works and how individual decisions were made, are important factors to foster users' trust in the AI (Barredo Arrieta et al. 2020; Ribeiro et al. 2016). ...
... In addition, (b), IS used in industry need to be made reliable and accountable (Ryan 2020). This is especially important in sensitive areas (Xu et al. 2019), even though there often are requirements to have a human in the loop making the final decision (Stuurman and Lachaud 2022). Thus, in such cases, decision-making by experts should be supplemented, but not replaced by it (Sutton et al. 2018). ...
Conference Paper
Full-text available
Artificial Intelligence (AI) becomes increasingly common, but adoption in sensitive use cases lacks due to the black-box character of AI hindering auditing and trust-building. Explainable AI (XAI) promises to make AI transparent, allowing for auditing and increasing user trust. However, in sensitive use cases maximizing trust is not the goal, rather to balance caution and trust to find the level of appropriate trust. Studies on user perception of XAI in professional contexts and especially for sensitive use cases are scarce. We present the results of a case study involving domain-experts as users of a prototype XAI-based IS for decision support in the quality assurance in pharmaceutical manufacturing. We find that for this sensitive use case, simply delivering an explanation falls short if it does not match the beliefs of experts on what information is critical for a certain decision to be reached. Unsuitable explanations override all other quality criteria. Suitable explanations can, together with other quality criteria, lead to a suitable balance of trust and caution in the system. Based on our case study we discuss design options in this regard.
... As a consequence, with the advent of DL, there are increasing discussions in the field about the relationship between ML model complexity and interpretability and the tendency to use models that are too complicated for prediction tasks at hand (Rudin, 2019). Furthermore, increasing attention is paid to explainable ML (Belle and Papantonis, 2021;Rodríguez-Pérez and Bajorath, 2021a) and the overarching area of explainable AI (XAI) (Gunning et al., 2019;Gunning et al., 2021;Jiménez-Luna et al., 2020;Xu et al., 2019). XAI refers to different categories of computational approaches for rationalizing ML models and their decisions in different areas of basic and applied research (Gunning et al., 2019;Jiménez-Luna et al., 2020;Xu et al., 2019) as well as in scientific teaching (Clancey and Hoffman, 2021). ...
... Furthermore, increasing attention is paid to explainable ML (Belle and Papantonis, 2021;Rodríguez-Pérez and Bajorath, 2021a) and the overarching area of explainable AI (XAI) (Gunning et al., 2019;Gunning et al., 2021;Jiménez-Luna et al., 2020;Xu et al., 2019). XAI refers to different categories of computational approaches for rationalizing ML models and their decisions in different areas of basic and applied research (Gunning et al., 2019;Jiménez-Luna et al., 2020;Xu et al., 2019) as well as in scientific teaching (Clancey and Hoffman, 2021). Explanation methods are equally relevant for classification and regression models (Letzgus et al., 2021;Rodríguez-Pérez and Bajorath, 2020a). ...
Full-text available
Graph neural networks (GNNs) recursively propagate signals along the edges of an input graph, integrate node feature information with graph structure, and learn object representations. Like other deep neural network models, GNNs have notorious black box character. For GNNs, only few approaches are available to rationalize model decisions. We introduce EdgeSHAPer, a generally applicable method for explaining GNN-based models. The approach is devised to assess edge importance for predictions. Therefore, EdgeSHAPer makes use of the Shapley value concept from game theory. For proof-of-concept, EdgeSHAPer is applied to compound activity prediction, a central task in drug discovery. EdgeSHAPer’s edge centricity is relevant for molecular graphs where edges represent chemical bonds. Combined with feature mapping, EdgeSHAPer produces intuitive explanations for compound activity predictions. Compared to a popular node-centric and another edge-centric GNN explanation method, EdgeSHAPer reveals higher resolution in differentiating features determining predictions and identifies minimal pertinent positive feature sets.
... Obtaining valid uncertainty for molecular property prediction has been widely discussed in the cheminformatic field, but nevertheless, with only the uncertainty that tells the potential failure of molecular property prediction, the reason why the given molecular structure leads to failure in prediction often needs to be indirectly analyzed through human intuition. Since predictions from black box models such as deep learning methods are challenging to interpret and analyze due to their nontransparency [35], Explainable Artificial Intelligence (XAI) has recently received much attention [36][37][38]. In chemistry and drug discovery, different perspectives based on molecular graphs, compounds, atoms, or feature representations have been proposed and discussed for a better understanding of model characteristics and model behaviors [39,40]. ...
Full-text available
Quantifying uncertainty in machine learning is important in new research areas with scarce high-quality data. In this work, we develop an explainable uncertainty quantification method for deep learning-based molecular property prediction. This method can capture aleatoric and epistemic uncertainties separately and attribute the uncertainties to atoms present in the molecule. The atom-based uncertainty method provides an extra layer of explainability to the estimated uncertainties, i.e., one can analyze individual atomic uncertainty values to diagnose the chemical component that introduces uncertainty to the prediction. Our experiments demonstrate that atomic uncertainty can detect unseen chemical structures and identify chemical species whose data are potentially associated with significant noise. Furthermore, we propose a post-hoc calibration method to refine the uncertainty quantified by ensemble models for better confidence interval estimates. This work improves uncertainty calibration and provides a framework for assessing whether and why a prediction should be considered unreliable.
... XAI is a set of algorithms and plots that help humans understand and interpret the output of ML models [84]. XAI reveals the information that the actions are based on, thereby improving the performance of smart models. ...
Full-text available
An assessment of site suitability for wind and solar plants is a strategic step toward ensuring a low-cost, high-performing, and sustainable project. However, these issues are often handled on a local scale using traditional decision-making approaches that involve biased and non-generalizable weightings. This study presents a global wind and solar mapping approach based on eXplainable Artificial Intelligence (XAI). To the best of the author’s knowledge, the current study is the first attempt to create global maps for siting onshore wind and solar power systems and formulate novel weights for decision criteria. A total of 13 conditioning factors (independent variables) defined through a comprehensive literature review and multicollinearity analysis were assessed. Real-world renewable energy experiences (more than 55,000 on-site wind and solar plants worldwide) are exploited to train three machine learning (ML) algorithms, namely Random Forest (RF), Support Vector Machine (SVM), and Multi-layer Perceptron (MLP). Then, the output of ML models was explained using SHapley Additive exPlanations (SHAP). RF outperformed SVM and MLP in both wind and solar modeling with an overall accuracy of 90% and 89%, kappa coefficient of 0.79 and 0.78, and area under the curve of 0.96 and 0.95, respectively. The high and very high suitability categories accounted for 23.2% (~26.84 million km2) of the site suitability map for wind power plants. In addition, they covered more encouraging areas (24.0% and 19.4%, respectively, equivalent to ~50.31 million km2) on the global map for hosting solar energy farms. SHAP interpretations were consistent with the Gini index indicating the dominance of the weights of technical and economic factors over the spatial assessment under consideration. This study provides support to decision-makers toward sustainable power planning worldwide.
Prior work has introduced a form of explainable artificial intelligence that is able to precisely explain, in a human-understandable form, why it makes decisions. It is also able to learn to make better decisions without potentially learning illegal or invalid considerations. This defensible system is based on fractional value rule-fact expert systems and the use of gradient descent training to optimize rule weightings. This software system has demonstrated efficacy for many applications; however, it utilizes iterative processing and thus does not have a deterministic completion time. It also requires comparatively expensive general-purpose computing hardware to run on. This paper builds on prior work in the development of hardware-based expert systems and presents and assesses the efficacy of a hardware implementation of this system. It characterizes its performance and discusses its utility and trade-offs for several application domains.
Full-text available
European law now requires AI to be explainable in the context of adverse decisions affecting the European Union (EU) citizens. At the same time, we expect increasing instances of AI failure as it operates on imperfect data. This paper puts forward a neurally inspired theoretical framework called “decision stacks” that can provide a way forward in research to develop Explainable Artificial Intelligence (X-AI). By leveraging findings from the finest memory systems in biological brains, the decision stack framework operationalizes the definition of explainability. It then proposes a test that can potentially reveal how a given AI decision was made.
E-learning, remote learning systems, and hybrid models were necessary in the pandemics age and when cities are rapidly moving towards smart city status. Adaptive and individualized education is now a must to reduce the drawbacks of distance learning while ensuring high levels of accomplishment. Technology such as machine learning, artificial intelligence (XAI), and data mining are all helping to transform education in a smart city by allowing for personalized learning and the ability to tailor content based on individual preferences. Based on the combination of XAI and IoB technologies, this study proposes a new paradigm for smart educational systems. Using data on students' actions, researchers try to establish whether or not contemporary educational systems meet students' needs. Education systems have evolved, but not to the point where they can be personalized to meet the cognitive demands of students and assist them when face-to-face instruction isn't available. Using evaluation methods, the study found that there has been a shift in students' academic progress when monitoring using IoT/IoB to enable a relative response in support of their progression. The system response to learners was 37%, once we involve the IoB in the processes and let students and the system be aware of the collected and used data about students' behaviours, the student's response improved to 76 % . This indicates the major influence of IoB on learner assistance and system adaptations to their real demands for higher achievement.
Prior work introduced a gradient descent trained expert system that conceptually combines the learning capabilities of neural networks with the understandability and defensible logic of an expert system. This system was shown to be able to learn patterns from data and to perform decision-making at levels rivaling those reported by neural network systems. The principal limitation of the approach, though, was the necessity for the manual development of a rule-fact network (which is then trained using backpropagation). This paper proposes a technique for overcoming this significant limitation, as compared to neural networks. Specifically, this paper proposes the use of larger and denser-than-application need rule-fact networks which are trained, pruned, manually reviewed and then re-trained for use. Multiple types of networks are evaluated under multiple operating conditions and these results are presented and assessed. Based on these individual experimental condition assessments, the proposed technique is evaluated. The data presented shows that error rates as low as 3.9% (mean, 1.2% median) can be obtained, demonstrating the efficacy of this technique for many applications.
Conference Paper
Full-text available
In the last decade, with availability of large datasets and more computing power, machine learning systems have achieved (super)human performance in a wide variety of tasks. Examples of this rapid development can be seen in image recognition, speech analysis, strategic game planning and many more. The problem with many state-of-the-art models is a lack of transparency and interpretability. The lack of thereof is a major drawback in many applications, e.g. healthcare and finance, where rationale for model's decision is a requirement for trust. In the light of these issues, explainable artificial intelligence (XAI) has become an area of interest in research community. This paper summarizes recent developments in XAI in supervised learning, starts a discussion on its connection with artificial general intelligence, and gives proposals for further research directions.
Full-text available
In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
Full-text available
This paper learns a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside a pre-trained CNN. Considering that each filter in a conv-layer of a pre-trained CNN usually represents a mixture of object parts, we propose a simple yet efficient method to automatically disentangles different part patterns from each filter, and construct an explanatory graph. Each graph node represents a part pattern, and graph edges encode co-activation relationships and spatial relationships between patterns. We learn the explanatory graph for a pre-trained CNN in an unsupervised manner, i.e., without a need of annotating object parts. Experiments show that each graph node consistently represents the same object part among different images. We transfer part patterns in the explanatory graph to part localization, and our method greatly outperforms other approaches.
Conference Paper
Full-text available
Fisher vector (FV) classifiers and Deep Neural Networks (DNNs) are popular and successful algorithms for solving image classification problems. However, both are generally considered 'black box' predictors as the non-linear transformations involved have so far prevented transparent and interpretable reasoning. Recently, a principled technique, Layer-wise Relevance Propagation (LRP), has been developed in order to better comprehend the inherent structured reasoning of complex nonlinear classification models such as Bag of Feature models or DNNs. In this paper we (1) extend the LRP framework also for Fisher vector classifiers and then use it as analysis tool to (2) quantify the importance of context for classification, (3) qualitatively compare DNNs against FV classifiers in terms of important image regions and (4) detect potential flaws and biases in data. All experiments are performed on the PASCAL VOC 2007 and ILSVRC 2012 data sets.
Full-text available
The Layer-wise Relevance Propagation (LRP) algorithm explains a classifier's prediction specific to a given data point by attributing relevance scores to important components of the input by using the topology of the learned model itself. With the LRP Toolbox we provide platform-agnostic implementations for explaining the predictions of pre-trained state of the art Caffe networks and stand-alone implementations for fully connected Neural Network models. The implementations for Matlab and python shall serve as a playing field to familiarize oneself with the LRP algorithm and are implemented with readability and transparency in mind. Models and data can be imported and exported using raw text formats, Matlab's .mat files and the .npy format for numpy or plain text. ©2016 Sebastian Lapuschkin and Alexander Binder and Grégoire Montavon and Klaus-Robert Müller and Wojciech Samek.
Full-text available
Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but inter-pretable. And yet the task of interpretation appears underspecified. Papers provide diverse and sometimes non-overlapping motivations for in-terpretability, and offer myriad notions of what attributes render models interpretable. Despite this ambiguity, many papers proclaim inter-pretability axiomatically, absent further explanation. In this paper, we seek to refine the discourse on interpretability. First, we examine the motivations underlying interest in interpretabil-ity, finding them to be diverse and occasionally discordant. Then, we address model properties and techniques thought to confer interpretability, identifying transparency to humans and post-hoc explanations as competing notions. Throughout, we discuss the feasibility and desirability of different notions, and question the oft-made assertions that linear models are interpretable and that deep neural networks are not.
Conference Paper
Full-text available
Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself. Existing approaches for deep visual recognition are generally opaque and do not output any justification text; contemporary vision-language models can describe image content but fail to take into account class-discriminative image aspects which justify visual predictions. We propose a new model that focuses on the discriminating properties of the visible object, jointly predicts a class label, and explains why the predicted label is appropriate for the image. We propose a novel loss function based on sampling and reinforcement learning that learns to generate sentences that realize a global sentence property, such as class specificity. Our results on a fine-grained bird species classification dataset show that our model is able to generate explanations which are not only consistent with an image but also more discriminative than descriptions produced by existing captioning methods.
We summarize the potential impact that the European Union's new General Data Protection Regulation will have on the routine use of machine learning algorithms. Slated to take effect as law across the EU in 2018, it will restrict automated individual decision-making (that is, algorithms that make decisions based on user-level predictors) which "significantly affect" users. The law will also create a "right to explanation," whereby a user can ask for an explanation of an algorithmic decision that was made about them. We argue that while this law will pose large challenges for industry, it highlights opportunities for machine learning researchers to take the lead in designing algorithms and evaluation frameworks which avoid discrimination.
Conference Paper
In machine learning often a tradeoff must be made between accuracy and intelligibility. More accurate models such as boosted trees, random forests, and neural nets usually are not intelligible, but more intelligible models such as logistic regression, naive-Bayes, and single decision trees often have significantly worse accuracy. This tradeoff sometimes limits the accuracy of models that can be applied in mission-critical applications such as healthcare where being able to understand, validate, edit, and trust a learned model is important. We present two case studies where high-performance generalized additive models with pairwise interactions (GA2Ms) are applied to real healthcare problems yielding intelligible models with state-of-the-art accuracy. In the pneumonia risk prediction case study, the intelligible model uncovers surprising patterns in the data that previously had prevented complex learned models from being fielded in this domain, but because it is intelligible and modular allows these patterns to be recognized and removed. In the 30-day hospital readmission case study, we show that the same methods scale to large datasets containing hundreds of thousands of patients and thousands of attributes while remaining intelligible and providing accuracy comparable to the best (unintelligible) machine learning methods.