Figure 9 - available via license: CC BY
Content may be subject to copyright.
Comparison of the network attention in terms of relevance responses of three different model architectures to important game elements (ball, paddle and tunnel) as a function of training time. An overview of the compared model architectures is given in Table 2. All three architectures show a shift in strategy towards tunnel building. For the NIPS architecture the shift appears in two stages resulting in a larger shift in total, whereas for the Small architecture the tunnel does not become a relevant objective during the training period.

Comparison of the network attention in terms of relevance responses of three different model architectures to important game elements (ball, paddle and tunnel) as a function of training time. An overview of the compared model architectures is given in Table 2. All three architectures show a shift in strategy towards tunnel building. For the NIPS architecture the shift appears in two stages resulting in a larger shift in total, whereas for the Small architecture the tunnel does not become a relevant objective during the training period.

Source publication
Article
Full-text available
Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-sol...

Contexts in source publication

Context 1
... F2 (512)→(4) describes the second and in this case last fully connected layer with 512 input nodes and 4 outputs. For the NIPS architecture, the focus on ball and paddle is delayed, emerging approximately 5 training epochs later when compared to the Nature network (see Figure 9). The relevance on the tunnel region shifts in two stages and finally rises to an amount higher than that recorded for the Nature architecture. ...
Context 2
... 18 for example shows boats in water, partially below the horizon line, yielding negative relevance values, since their presence occludes the water. Figure 19 demonstrates, that the DNN classifier does not show this effect. ...
Context 3
... speculating, we consider the results for the neural network agent playing Atari games as shown in Figure 11 suggestive for the hypothesis that tight groupings In this scene, masts crossing the horizon and the water itself count towards the concept of "boat", while some vessels below the horizon -both boats to the middle left aligned with the camera taking the image and both boats to the right -are yielding disruptive visual features. See also Figure 19 for complementary heatmaps computed for class "boat" based on the DNN model. of relevance scores on intuitively important image elements indicate a well-performing predictor even beyond typical classification tasks. ...
Context 4
... hypothesize, that the model has learned to expect uniformly colored image borders for samples belonging to class "aeroplane". We verify that hypothesis in Figure 29 (left), which shows that any processing resulting in a square image besides the copying of border pixels will result in a decrease airplaneness of the sample. Mirror padding can e.g. ...
Context 5
... provide those sample images as input to the DNN and observe the effect on the model output corresponding to the prediction of airplanes. The results are shown in Figure 29 (right) verify again that introducing structure (which is with a certain likelihood much stronger for non-"aeroplane" images) to the border regions of the image (mirror and crop, random noise) reduce the airplaneness of the inputs further, while padding with (a fitting) constant color results in a comparatively high increase in the prediction score. Figure 31: Non-"aeroplane" samples and predicted scores for class "aeroplane", as affected by different preprocessing strategies to obtain square images. ...

Citations

... Bach et al. (2015) studied the variables that can be considered in the relevance score for pixel-wise explanations using LRP. Furthermore, and Lapuschkin et al. (2019) modified the calculation method of the relevance score to improve the interpretation of the results. In Sect. ...
Article
Full-text available
The deep learning (DL) model has performed successfully in various fields, including manufacturing. DL models for defect image data analysis in the manufacturing field have been applied to multiple domains such as defect detection, classification, and localization. However, DL models require trade-offs in accuracy and interpretability. We use explainable artificial intelligence techniques to analyze the predicted results of the defect image classification model, which is considered as a “black-box” model, to produce human-understandable results. We visualize defects using layer-wise relevance propagation-based methods, fit the model into a decision tree, and convert prediction results into human-interpretable text. Our research complements the interpretation of prediction results for the classification model. The domain expert can obtain the reliability and explanatory ability for the defect classification of TFT–LCD panel data of the DL model through the results of the proposed analysis.
... By observing the machine's explanations, users have the opportunity of building a better understanding of the machine's overall logic, which not only facilitates trust calibration [10], but also supports and amplifies our natural capacity of providing appropriate feedback [92]. Machine explanations are also key for identifying imperfections and bugs affecting ML models, such as reliance on confounded features that are not causally related with the desired outcome [97,60,137]. At the same time, human explanations are a very rich source of supervision for models [31] and are also very natural for us to provide. ...
... Explanations are also instrumental for identifying models that rely on confounds in the training data, such as watermarks in images, that happen to correlate with the desired outcome but that are not causal for it [97,60]. Despite achieving high accuracy during training, these models generalize poorly to real-world data where the confound is absent. ...
Preprint
Explanations have gained an increasing level of interest in the AI and Machine Learning (ML) communities in order to improve model transparency and allow users to form a mental model of a trained ML model. However, explanations can go beyond this one way communication as a mechanism to elicit user control, because once users understand, they can then provide feedback. The goal of this paper is to present an overview of research where explanations are combined with interactive capabilities as a mean to learn new models from scratch and to edit and debug existing ones. To this end, we draw a conceptual map of the state-of-the-art, grouping relevant approaches based on their intended purpose and on how they structure the interaction, highlighting similarities and differences between them. We also discuss open research issues and outline possible directions forward, with the hope of spurring further research on this blooming research topic.
... Adopting these technologies in contexts such as justice, transport, and healthcare has harmed society (Marcus and Davis 2019). The decision mechanisms of high-performance ML models can be based on hidden spurious correlations (Lapuschkin et al. 2019), leading users to make mistakes. XAI would help users to judge the decisions made by an automated system, allowing them to understand and prevent misuses of their results in different contexts. ...
Article
Full-text available
Providing meaningful and actionable explanations for end-users is a situated problem requiring the intersection of multiple disciplines to address social, operational, and technical challenges. However, the explainable artificial intelligence community has not commonly adopted or created tangible design tools that allow interdisciplinary work to develop reliable AI-powered solutions. This paper proposes a formative architecture that defines the explanation space from a user-inspired perspective. The architecture comprises five intertwined components to outline explanation requirements for a task: (1) the end-users’ mental models, (2) the end-users’ cognitive process, (3) the user interface, (4) the Human-Explainer Agent, and (5) the agent process. We first define each component of the architecture. Then, we present the Abstracted Explanation Space, a modeling tool that aggregates the architecture’s components to support designers in systematically aligning explanations with end-users’ work practices, needs, and goals. It guides the specifications of what needs to be explained (content: end-users’ mental model), why this explanation is necessary (context: end-users’ cognitive process), to delimit how to explain it (format: Human-Explainer Agent and user interface), and when the explanations should be given. We then exemplify the tool’s use in an ongoing case study in the aircraft maintenance domain. Finally, we discuss possible contributions of the tool, known limitations or areas for improvement, and future work to be done.
... They used this as a justification to bridge global and local XAI (such as LRP, (Guided) Grad-CAM etc.). The previous work in [21,22,24,25] also belongs to this category. Here, we showed that this argument is not valid and global methods can be valid for bias discovery. ...
Article
Full-text available
Digital histopathology poses several challenges such as label noise, class imbalance, limited availability of labelled data, and several latent biases to deep learning, negatively influencing transparency, reproducibility, and classification performance. In particular, biases are well known to cause poor generalization. Proposed tools from explainable artificial intelligence (XAI), bias detection, and bias discovery suffer from technical challenges, complexity, unintuitive usage, inherent biases, or a semantic gap. A promising XAI method, not studied in the context of digital histopathology is automated concept-based explanation (ACE). It automatically extracts visual concepts from image data. Our objective is to evaluate ACE’s technical validity following design science principals and to compare it to Guided Gradient-weighted Class Activation Mapping (Grad-CAM), a conventional pixel-wise explanation method. To that extent, we created and studied five convolutional neural networks (CNNs) in four different skin cancer settings. Our results demonstrate that ACE is a valid tool for gaining insights into the decision process of histopathological CNNs that can go beyond explanations from the control method. ACE validly visualized a class sampling ratio bias, measurement bias, sampling bias, and class-correlated bias. Furthermore, the complementary use with Guided Grad-CAM offers several benefits. Finally, we propose practical solutions for several technical challenges. In contradiction to results from the literature, we noticed lower intuitiveness in some dermatopathology scenarios as compared to concept-based explanations on real-world images.
... Machine learning makes use of the whole transcriptome landscape and not only of most prominent differential features, however, in terms of a "black box" without explicit annotation of the feature patterns the machine really learned for classification. The so-called layer-wise relevance propagation (LRP) method visualizes the metagene areas of high relevance for classification in SOM space [84] (Figure A2, feature portraits). On a rough scale, the relevant features agree with the differential spots discussed above, e.g., spot B and F distinguish BL versus DLBCL, and spot H and K distinguish ABC versus GCB-DLBCL. ...
... Relevant features for classification patterns are shown in the yellow box. It applied the layer-wise relevance propagation (LRP) method to visualize in SOM space what the machine really learned [84]. Classification quality was estimated as confusion matrix (input versus learned class), and precision (correct percentage of positive calls) and recall (percentage of actual true calls) values. ...
Article
Full-text available
Classification of lymphoid neoplasms is based mainly on histologic, immunologic, and (rarer) genetic features. It has been supplemented by gene expression profiling (GEP) in the last decade. Despite the considerable success, particularly in associating lymphoma subtypes with specific transcriptional programs and classifier signatures of up- or downregulated genes, competing molecular classifiers were often proposed in the literature by different groups for the same clas-sification tasks to distinguish, e.g., BL versus DLBCL or different DLBCL subtypes. Moreover, rarer sub-entities such as MYC and BCL2 “double hit lymphomas” (DHL), IRF4-rearranged large cell lymphoma (IRF4-LCL), and Burkitt-like lymphomas with 11q aberration pattern (mnBLL-11q) attracted interest while their relatedness regarding the major classes is still unclear in many re-spects. We explored the transcriptional landscape of 873 lymphomas referring to a wide spectrum of subtypes by applying self-organizing maps (SOM) machine learning. The landscape reveals a continuum of transcriptional states activated in the different subtypes without clear-cut border-lines between them and preventing their unambiguous classification. These states show striking parallels with single cell gene expression of the active germinal center (GC), which is character-ized by the cyclic progression of B-cells. The expression patterns along the GC trajectory are discriminative for distinguishing different lymphoma subtypes. We show that the rare subtypes take intermediate positions between BL, DLBCL, and FL as considered by the 5th edition of the WHO classification of haemato-lymphoid tumors in 2022. Classifier gene signatures extracted from these states as modules of coregulated genes are competitive with literature classifiers. They provide functional-defined classifiers with the option of consenting redundant classifiers from the literature. We discuss alternative classification schemes of different granularity and functional impact as possible avenues toward personalization and improved diagnostics of GC-derived lymphomas.
... Therefore, the use of BCI technology in daily life based on machine/deep learning models that achieve high performance should consider designing simple, compressed models. Furthermore, these more advanced models and architectures need to be explainable (Sturm et al., 2016) and tested for artifacts and Clever Hans effects (Lapuschkin et al., 2019;Anders et al., 2022). Through this review, we hope that many investigators will be motivated to focus on these aspects in the future and that they utilize both the advantages of machine learning and deep learning to contribute to further BCI advances. ...
Article
Full-text available
The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.
... Moreover, even corrupted data may betray sensitive information like race, eluding awareness of clinical experts (34). And last not least, spurious data correlations may lead to mismatch (35), famous in MLMD as "Clever Hans" (36). Model predictions must be evaluated with confidence limits of uncertainty in mind (37). ...
... 28 test oracles in specialized environments as pytest. 36 Test coverage is an issue and object of research in complex deep neural nets. Combinatorial testing as efficient blackbox approach is recommended. ...
Article
Full-text available
Recent progress in digital health data recording, advances in computing power, and methodological approaches that extract information from data as artificial intelligence are expected to have a disruptive impact on technology in medicine. One of the potential benefits is the ability to extract new and essential insights from the vast amount of data generated during health care delivery every day. Cardiovascular imaging is boosted by new intelligent automatic methods to manage, process, segment, and analyze petabytes of image data exceeding historical manual capacities. Algorithms that learn from data raise new challenges for regulatory bodies. Partially autonomous behavior and adaptive modifications and a lack of transparency in deriving evidence from complex data pose considerable problems. Controlling new technologies requires new controlling techniques and ongoing regulatory research. All stakeholders must participate in the quest to find a fair balance between innovation and regulation. The regulatory approach to artificial intelligence must be risk-based and resilient. A focus on unknown emerging risks demands continuous surveillance and clinical evaluation during the total product life cycle. Since learning algorithms are data-driven, high-quality data is fundamental for good machine learning practice. Mining, processing, validation, governance, and data control must account for bias, error, inappropriate use, drifts, and shifts, particularly in real-world data. Regulators worldwide are tackling twenty-first century challenges raised by “learning” medical devices. Ethical concerns and regulatory approaches are presented. The paper concludes with a discussion on the future of responsible artificial intelligence.
... Though ML is highly efficient, it has some limitation which reduces its accuracy in predicting properties. Such limitations include, but are not limited to, measurement error 22 , lack of generality and precision, reliance on high-quality data 23 , inability to determine high level concept 24 , prone to artifact 25 , good in interpolation but poor in extrapolation 21,26 . Another critical drawback for ML methods is the lack of laws, understanding, and knowledge from their use because ML methods are treated as black box 6 . ...
Article
Full-text available
Despite the machine learning (ML) methods have been largely used recently, the predicted materials properties usually cannot exceed the range of original training data. We deployed a boundless objective-free exploration approach to combine traditional ML and density functional theory (DFT) in searching extreme material properties. This combination not only improves the efficiency for screening large-scale materials with minimal DFT inquiry, but also yields properties beyond original training range. We use Stein novelty to recommend outliers and then verify using DFT. Validated data are then added into the training dataset for next round iteration. We test the loop of training-recommendation-validation in mechanical property space. By screening 85,707 crystal structures, we identify 21 ultrahigh hardness structures and 11 negative Poisson’s ratio structures. The algorithm is very promising for future materials discovery that can push materials properties to the limit with minimal DFT calculations on only ~1% of the structures in the screening pool.
... Layer-Wise Relevance Propagation (LRP) is a typical backpropagation-based method to predict classification results by calculating the contributions of individual pixels. In general, it starts from the output layer of the network, moves to the opposite direction, and redistributes the relevant score until it reaches the input layer [22,23]. Simultaneously, it follows the global conservation property, which indicates that whatever a neuron receives must be redistributed in equal amounts to the lower layer. ...
Article
Full-text available
Deep learning has demonstrated remarkable performance in the medical domain, with accuracy that rivals or even exceeds that of human experts. However, it has a significant problem that these models are “black-box” structures, which means they are opaque, non-intuitive, and difficult for people to understand. This creates a barrier to the application of deep learning models in clinical practice due to lack of interpretability, trust, and transparency. To overcome this problem, several studies on interpretability have been proposed. Therefore, in this paper, we comprehensively review the interpretability of deep learning in medical diagnosis based on the current literature, including some common interpretability methods used in the medical domain, various applications with interpretability for disease diagnosis, prevalent evaluation metrics, and several disease datasets. In addition, the challenges of interpretability and future research directions are also discussed here. To the best of our knowledge, this is the first time that various applications of interpretability methods for disease diagnosis have been summarized.
... These kinds of neural networks seemingly perform well under laboratory conditions, making them difficult to spot before deployment. The problem of CHP arises if a network focuses on features that are logically irrelevant for inference [77]. These could be watermarks or other text on images or on background details instead of the objects of interest. ...
Article
Full-text available
Neural networks for deep-learning applications, also called artificial neural networks, are important tools in science and industry. While their widespread use was limited because of inadequate hardware in the past, their popularity increased dramatically starting in the early 2000s when it became possible to train increasingly large and complex networks. Today, deep learning is widely used in biomedicine from image analysis to diagnostics. This also includes special topics, such as forensics. In this review, we discuss the latest networks and how they work, with a focus on the analysis of biomedical data, particularly biomarkers in bioimage data. We provide a summary on numerous technical aspects, such as activation functions and frameworks. We also present a data analysis of publications about neural networks to provide a quantitative insight into the use of network types and the number of journals per year to determine the usage in different scientific field.